The lack of built-in HTTPS seems killer to me. On the client-facing side it needs a separate daemon for TLS termination and on the upstream side there's no TLS support at all unless you're using the paid version.
The reason why Varnish does not have built in TLS, is a matter of security design and flexibility.
First, as to flexibility:
If we added TLS support to Varnish, we would have to pick a TLS library, which means our users would automatically be locked into that library.
This would violate one of our core principles, which is "Tools, not policies".
And given the state of TLS implementations, I /really/ dont want to make that choice, I want to leave that to the person responsible for the security at the site running Varnish.
Second, I think it is sound security engineering to confine the server certificate in a process which does that one thing and does it well.
I know several sites that have two different TLS frontends in front of Varnish, using different TLS implementations, so that whenever a CVE hits one of them, they can just shut that half down until the issue is fixed, and still remain in production. That would not be possible if TLS was bolted into Varnish.
I agree. I think varnish remained popular for 2 reasons:
- It was, at one time, higher performing than an nginx proxy cache. That doesn't seem to be the case anymore.
- VCL is very rich and flexible with primitives for reading/writing cookie contents, cache metadata, cache flushing, client and server state, and so on.
So probably the only reason left if that you're doing something really complicated with your cache in VCL.
Otherwise, as you say, the built in HTTPS of nginx, haproxy, nuster, or something else is a big advantage.
https://trends.builtwith.com/Web-Server/Varnish
My personal estimate is that around 20% of all HTTP traffic pass through a Varnish instance somewhere, many of which are not public-facing.
As to why people use Vanish, I think the main reason is that VCL gives you full and instant control over your HTTP traffic, and people really like that, because it lets unify heterogenous sites and pamper legacy systems as necessary.
- Varnish can guarantee that a given resource will be fetched at most once every TTL expiration since you can set up one instance to do so (+ some HA solution like hot standby). This complements a large CDN that is highly scalable but at a cost that there is no hard "synchronization" between servers. You will see many requests for the same resource, even when the CDN uses cache hierarchy because they can't risk such bottleneck not knowing in advance your traffic pattern. You can intentionally do that with Varnish and benefit from that.
- Complex caching rules, even including what to do if application is not available/slow. That includes serving stale content, how long to serve stale after TTL expires, guarantee that fetching a new version is performed in background. Example: https://varnish-cache.org/docs/trunk/users-guide/vcl-grace.h...
We've used varnish + some other magic in addition to Cloudflare to withstand Black Friday when marketing insisted the promo has to start at specified time, anuonced much earlier to all customers. The landing page had up-to date info on available products updated almost instantly, thanks to serving stale (in practice < 1s) content and guaranteed prefresh in background. Cache key was properly set up to exclude tracking elements from URLs (from FB, adwords etc) and only include what is necessary.
By this people mean things like printing the wrong URL in a full-page sunday newspaper add or splitting traffic based on first digit of customer-number and stuff like that.
Some of the screwups Varnish-ops have fixed are truly the stuff of legends, but telling them belongs to their heros :-)