Researching a new form of HTTP caching optimization | Hacker News Reader

Researching a new form of HTTP caching optimization

blog.phusion.nl

57 pointsohcoder49 comments

> the Varnish HTTP cache has been used very successfully to speed up WordPress. But Varnish doesn’t help a lot with logged-in traffic > This is caching that Varnish and other “normal” HTTP caches (including CloudFlare) could not have done

Varnish supports the ESI (Edge-Side Includes) standard, which allows it to cache fragments of a page, and for the cache server to build them again. It also allows you to completely bypass the cache for certain fragments. This is also supported by a number of CDNs (Fastly, Akamai). I've used the ESI technique several times and have been able to achieve a >98% cache hit rate on Fastly for a site with dynamic per-user content. Even the cache misses are only responsible for rendering a small component of the page

Exactly.

I'm not really sure what situations this built-in cache would be more effective than the likes of a well-tuned Varnish.

Good to know. Using edge side includes may be easier than trying to change the app to a semi single page app. But that only solves half of the problem. The other half is varying the response based on the value of a specific cookie.

I've updated the blog post with information regarding edge side include.

Varnish also supports custom cache keys via the VCL function `vcl_hash` as documented in https://www.varnish-software.com/static/book/VCL_functions.h...

I couldn't (quickly) find documentation on how to get the value of a specific cookie, but the server could send a user ID in a header or something Varnish can easily access to be used in the above function.

You will probably have to write a VCL plugin if you want to parse the cookie, but that's not a huge hassle.

You can handle cookies in pure VCL – the code's not particularly elegant but it's manageable for light usage:

https://www.varnish-cache.org/trac/wiki/VCLExampleRemovingSo...

I'm not sure whether I would call "munging the cookie header with regexps" "cookie handling" ;).

Agreed – it's possible and for simple tasks such as stripping an analytics cookie it's workable but for anything more serious you'd want something like https://github.com/lkarsten/libvmod-cookie

Off the top of my head:

    sub vcl_recv {
      set req.http.X-Custom-Cookie-Value = regsub(req.http.cookie, ".*;|^)YOUR_COOKIE_NAME=([^;]+)(;.*|$)", "\2");
    }

Then you can just:

    sub vcl_hash {
      set req.hash += req.http.X-Custom-Cookie-Value;
    }

And now you're cache segmented on that cookie value.

I understand that you want to offer something that 'beats' varnish, and it shines through in the article.

But I don't think it matters if your cache is better than every other cache. Rather, as long as you offer a convenient, easily implemented cache, built into the webserver, that's great in itself. We're using Passenger on all our production servers and are most satisfied, because of its ease of use.

Perhaps you could just write "this could be accomplished with Varnish, which has a lot of benefits for advanced cases, but we think our cache will be useful for those that prefer not to manage a separate caching tier."

> I understand that you want to offer something that 'beats' varnish, and it shines through in the article.

I am the author of the article. No, the point is not to "beat" Varnish. It is an article describing various ideas and a call for help. See https://news.ycombinator.com/item?id=8844905

Perhaps the writing style gave a competitive impression, so I've updated the article to mention that we're not out to beat Varnish, but to research the possibilities.

Knowing that Varnish can accomplish some of the things is good, because that way we can draw from an existing pool of experience.

Everything described in the article is covered by Varnish. You can hash on Vary headers, on individual cookie values, on the sum of the digits in the user's IP address if you want. ESI lets you provide partial caching of pages as the article describes - it's actually a separate standard that's existed since 2001 (http://en.wikipedia.org/wiki/Edge_Side_Includes).

Varnish also gives us things like ACLs for managing access to various resources, on-demand content purges, multiple routable backends with different cache/grace rules, and more powerfully, request pre-processing - one thing we do is process the request and determine if the agent is capable of accepting WebP images, and if they are, we add that to the hash key with a corresponding header for the app to key on and determine whether to serve JPEG or WebP images. This lets us serve WebP images to modern agents for faster downloads, while gracefully falling back to JPEG for anything we're not sure of.

Varnish is way more than a "make Wordpress not destroy your server" cache.

Varnish also supports plugins for extreme flexibility. For example, I wrote a plugin for our Varnish install which performs HMAC validation of a specific signed cookie and then sets a header which is used downstream in the caching rules.

Varnish is mature, powerful, and fast as hell. It would take a lot of work to reach a point where I'd swap it out for something else.

I've left some comments on the Disqus thread on the blog, but I'll reiterate my concern about the security of the cookie being set.

The cookie being set is unsigned according to the documentation[0] on rails, so a user could modify it and send it back to get a different cached response. Say that they saw that the user level was being set in there (like in the blog post) and they change the value to the 'staff' value to get the staff cached response. Probably not a good idea!

With that said, I don't think that this technique is adequate right now when you have user access level concerns and you're relying on that piece of unsigned information to not be tampered with.

[0] http://api.rubyonrails.org/classes/ActionDispatch/Cookies.ht...

I think it's been said enough here but Varnish can certainly do almost everything that they are saying it can't do, including some other storage optimizations like storing the gzipped response and serving a non-gzipped when requested (as of Varnish 3.0).

You can use Vary for tons of caching optimizations via varnish, such as caching mobile web-pages vs non-mobile web pages or just a particular header. It's all about just flexing a little bit of VCL (which I'll admit sometimes can throw people off).

They had me until this part:

> This is an HTTP cache built directly in Passenger so that it can achieve much higher performance than external HTTP caches like Varnish.

And since they have no benchmarks to really back up these claims, I'm skeptical they did much research against Varnish to tune or set it up. I'd love to see the numbers on varnish vs their turbocache. Without numbers, I have to take a lot of it with a grain of salt.

Either way, seems like it could be an extra handy thing to have in your toolbox, as long as it fits your stack.

You're reading too much hostility in it. The article does not claim to be better than Varnish. Performance is achieved by not implementing many features. For example, Varnish is a full-blown programming language and has an infinite size. The Passenger turbocache has almost no configuration options, does not support any sort of custom programming and only supports 8 entries at most. The fact that the max size is so small allows it to be implemented in very compact data structures, but its usefulness is also severely limited. It's merely designed with a different set of tradeoffs than Varnish.

It isn't that difficult to make a web server that's faster all the production servers out there. For example H2O is faster than Nginx... because it has infinitely less features. Ditto for Passenger's turbocache.

The rest of the article describes some ideas, some of which cannot be implemented using pure Varnish and require cooperation from the app. In my opinion you're missing out on a lot of interesting ideas if you stop reading only because you think Varnish is being slammed.

Fair enough! I can definitely see how I might have taken it too hard (or it came off in my writing of the comment)

I was just more or less intrigued by the actual claim and would have loved to just see some numbers on "higher performance than varnish" and the scenarios where it could reach those performance pieces. It just piqued my interest for future architecture considerations.

Thanks for enlightening me more on turbocache. It's on my list of things to go through tonight.

Edit: I did want to point out my bad choice of words. "They had me until here.", did not mean I didn't read the article in it's entirety. Just that I continued with a bit more skepticism.

I think the complaint is just that the approach is in no way as novel as you make it seem.

It can and has been implemented with Varnish and a few lines of VCL (with support of the app as well, of course).

What is "pure varnish" anyways? Proper varnish use always needs a tuned VCL, that's the whole core of the product.

Also, if I might say, I find your edits very handwavey. Please prove why that is impossible in VCL.

Fair enough. But even claiming that it's truly novel is not the point of the article. The point of the article is to research HTTP caching and to converse with the community about possibilities. It is not to present a new product.

In hindsight I see that the title may have been badly chosen. Before publication, none of the test readers have made any fuss about this.

> Please prove why that is impossible in VCL.

Where do you get impression that I claimed it's impossible in the VCL? It's pretty clear that the single page app approach can be replaced by edge side include, while the cookie parsing can be done with the cookie vmod.

The article claims that despite being the cache being able to do these things, application support is still required. The app has to be modified to output certain cookies in a certain format, e.g. the vary-on-user-permission-level thing. It's the combination that is important.

Have these things been done before? I'm sure they have. Most ideas in computer science are 30+ years old, and it's rare to truly find something new. But again, the point of the article is research. What we're after is not to present a new thing, but to present an idea, and if it's a new thing then we want people to help us to test; if it's not a new thing then we want to hear experiences.

The part about inability to speed up authenticated page loads fails to take into consideration things like ESI.

If the majority of your page is still the same for logged-in users, but they see some pieces of content personalised for them, breaking them out into individually request-able components means you can let software like Varnish or a CDN rely on it's cache of the main page content, and make a very small (and ideally simple to process on the backend) request for the per-user content.

It took me some hunting to find it (no docs for v5 yet apparently) but if your stack already includes a caching layer (e.g. Varnish or a CDN) you may want to disable this extra cache using the config described here: http://blog.phusion.nl/2014/11/25/introducing-phusion-passen...

> What if the cache can parse cookies and vary the cached response by the value of a specific cookie, not the entire header?

> This is caching that Varnish and other “normal” HTTP caches (including CloudFlare) could not have done.

This is false. I'm not at all very familiar with Varnish, but I know this is easily possible, and has been used for many, many years.

E.g. for Drupal + Varnish, i.e. to only keep Drupal's session cookie, I found these examples, in less than a minute of googling:

- https://www.varnish-cache.org/trac/wiki/VarnishAndDrupal

- https://www.lullabot.com/blog/article/configuring-varnish-hi... (grep for "inclusion")

Everything in this article has been well-known for at least half a decade, yet is being presented as major technical breakthroughs. Too much marketing, IMO.

Yes, and we (CloudFlare) offer different caching based on cookies.

Can you link to the relevant documentation?

All I can find is in this whitepaper, and it doesn't go into detail: https://www.cloudflare.com/static/media/pdf/cloudflare-white...

The first example looks nothing like parsing a specific cookie. It merely sanitizes the cookie headers a little, but it doesn't extract out a specific cookie to use as cache key.

And both examples you link to remove cookies. That's not what we're after. We're after the extraction of a specific cookie without removing anything.

It's also not marketing for a commercial product. The research is for an open source project, and the code is public and open source. The entire point of the blog post is to call for research participants who could not only test our ideas, but who could also point out anything we might have missed.

> The first example looks nothing like parsing a specific cookie. It merely sanitizes the cookie headers a little, but it doesn't extract out a specific cookie to use as cache key.

You say you don't parse a specific cookie, but doesn't extract a specific cookie as a cache key? The blog post says the opposite:

> We modified Passenger to parse cookies, and to vary turbocache responses based on the value of this user_id cookie. We invoke Passenger like this: passenger start --vary-turbocache-by-cookie user_id

In other words: parse the user_id cookie.

The mentioned commit even adds a ~250 LoC file (ext/common/ServerKit/CookieUtils.h) to do cookie parsing: https://github.com/phusion/passenger/commit/a760649cd79fde43...

---

Anyway, what you're getting at is that Phusion only parses a certain value from the cookie and then uses it as a custom Vary header, whereas the examples I linked to clean up the Cookie header and then varies on that cleaned up Cookie header. That boils down to exactly the same thing.

P.S.: downvoting? Not very nice.

I am from Phusion. No idea who downmodded you, but this happens often on HN.

But what you mentioned is not the same thing. In the examples you linked to, Google Analytics and other cookies are removed, leaving only a single cookie. But that's not what we're after. We want to vary by a specific cookie, without removing the other cookies.

For example, Rails stores the session data in a session cookie. We advocate introducing a user_id cookie in addition to the session cookie. Using your approach, the session data would be removed, which breaks the application. Our approach leaves all original cookie data intact while still varying on a specific cookie.

Furthermore, another idea that we've described in the blog post is to vary based some user property, e.g. the user's permission level, in order to increase the cardinality of cache entries. This is not something you can do with only Varnish: it requires cooperation from the app.

I had a contract at a company about 13 years ago where I was working on a web-based CMS that had been built entirely in-house. Because each rendered page's content was built up in a hierarchical manner, I added a caching layer that allowed arbitrary portions of the rendered page "tree" to be cached all the way up to the entire page and HTTP response if possible. Each cached portion could be located quickly based on its dependencies (template ID, content ID, CSS etc). If any edits were made to the site, only the relevant portions needed to be flushed and re-rendered. User-specific portions could be cached in the user's session rather than site-wide. (Thinking about it now, each portion could have been rendered in parallel too, though this was back in the days when multicore machines weren't very common and it wasn't something that occurred to me.)

I built this without giving much thought to whether anyone else had attempted something similar. Presumably they had though I do remember being disappointed when researching the various open source caching libraries, they didn't offer much help at the time. Overall I was pretty happy with the way it all worked and the performance boost it provided was like night and day.

Sadly the company is no longer operating, presumably the CMS codebase is long lost.

Most major frameworks include something like this but it's usually harder to use, 'partials' caching as it is called is a pretty effective way to speed up the server side of things when you need that.

Varnish has a similar facility (which allows you to abstract it out of the framework code entirely).

A dependencies ('make') like automatic approach to this would be quite nice to have, maybe your approach could be retro-fitted onto one of the existing CMSs?

Thanks, I hadn't heard of the names "Russian doll" or "partials" before. I haven't worked with a CMS (or even web apps at all really) for over a decade now but I'm surprised you seem to imply that automatic dependency management isn't standard stuff these days already. It was certainly very useful to us, though admittedly tricky to implement robustly.

I have seen this referred to as the "Russian doll" caching strategy. You can find several examples using Rail's partials or Django's cache template tag.

I believe Stack Overflow does some of this too. Though I can't seem to find the source that convinced me of such.

Hmmm. I might be missing something here, but I routinely clean out cookies in "public"/unauthed URLs in Varnish, as well as hashing the cache based on _part_ of a specific cookie (the bit that defines, say, the site's theme, or a generic user role).

They do mention that they didn't investigate how to do this in Varnish, but I recall having picked up the basic technique from one of the author's posts.

This was my first thought too. You can vary by any subset of cookies in Varnish if you simply normalize the Cookie header by stripping out irrelevant cookies. Another trick I've used is to parse the cookie to look for the user ID and set a User-Id header on the backend response and then set Vary: User-Id.

> Normally, non-cacheable page fragments would make every page uncacheable.

AFAICT, ASP.NET supported partial output caching since 2003. And, it can store multiple versions of a cached item via a user-defined parameter (a property on the cached control class). It seems it might be possible to simply create a property that returns the user ID and let the cache vary on that.

Of course, this uses the Web Forms controls-based system, which isn't very popular. But calling this type of caching new seems a bit of a stretch.

http://msdn.microsoft.com/en-us/library/k4he1ds5%28v=vs.71%2...

Edit: And here's a link talking about VaryByCustom, complete with example code that uses the version of the browser to generate unique per-browser-version caches.

http://msdn.microsoft.com/en-us/library/5ecf4420%28v=vs.71%2...

Fragment caching of non-anonymous things on a cache is nothing new. Varnish supports it with ESI, other caches support that with other strategies.

I've successfully build SPAs using that strategy. It's basically Rails russian doll-caching on a dedicated process.

Even ignoring the fact that edge side includes are the way to go when confronted with this problem, it sounds like they're basically saying they'd like to include the userid in the Vary header, but cannot because the Cookie header includes a bunch of other stuff.

Instead of parsing the Cookie header and doing a bunch of additional work in the cache layer, why not just add a separate custom header (X-User-Id, say) and Vary on that?

You're talking about response headers. But this deals with sending responses, you only get requests. Request headers can't contain a custom header that you control (unless your JS is making the requests of course). Hence you have to transform the Cookie header into whatever you need.

I almost always configure Varnish not to cache on Vary Cookie and pair that with some simple changes on the back-end to keep most my pages cacheable.

One, don't blindly set Vary Cookie on every single page. Two, when a page only needs minor variation like a username; store the username in a cookie and use JavaScript to display it on the page.

Yeah, the biggest advantage ESI offers is not requiring JS. If that's important to you, configure it and do extra rendering on the server-side. If not... :)

When you mentioned discourse serving 19k req/sec, does it hit the ruby stack at all? If no, serving 19k req/sec of cached HTML doesn't seem that impressive. What am I missing?

Sorry to barge on this thread - but does have a reasonable varnish config that works for a wordpress site behind https nginx and W3 Total Cache ?

It seems strangely hard to configure something like this.

I used (and modified) one based on Dreamhost's https://github.com/dreamhost/varnish-vcl-collection/blob/mas... but there's a bunch of others on Github if you search for them. Note that this isn't a shortcut so you don't have to learn VCL... and that the way I did it was to put Varnish on port 80 and the web server on another port. I've seen other configs where a load balancer asks Varnish when it knows varnish should have the answer and otherwise sends traffic directly to the web server. And sometimes your config will vary based on asset type or domain name -- e.g. static.example.org assets might be cacheable forever while your site's pages might only require a 2 minute cache. You can invalidate cached-forever pages with commands for Varnish or by varying the URL using a "cache buster" hash or string in the asset filename. Oh and don't forget to set the storage type Varnish uses to malloc instead of file.

thanks for this !

I guess you dont have an SSL termination point in front. What I was thinking of was nginx -> varnish -> nginx.

Ah, actually we do, but that runs on the load balancer. We currently use nginx on the LB but you could use haproxy alone these days.

When you use nginx as ssl termination loadbalancer - do you just do a proxy forward to varnish, or is it anything complicated ?

uWSGI has a powerful cache mechanism, and it's clustered.

and you should be using it:

http://cramer.io/2013/06/27/serving-python-web-applications/

Switching to a single page app seems invasive. Why not just have an iframe for the username and stuff?

Reply on news.ycombinator.com