Context-aware HTTP caching (with Varnish) | Hacker News Reader

Context-aware HTTP caching (with Varnish)

asm89.github.com

69 pointsasm8921 comments

It's worth noting that sometimes you only have content that differs for your "logged in" users. Or, content that is the same with the exception of actions that a user has taken on your site. Say, in the case of a shopping cart.

In that use case, you might find that one cache covers everything except their shopping cart preview. This is where ESI (Edge Side Includes) come in. I've never used it in practice, but the idea is similar to server side includes of yesteryear. Here's a good article:

http://blog.redfin.com/devblog/2010/05/esi_and_caching_trick...

Another option is to populate the cart preview via an ajax request, which allows the initial page to be served entirely from the cache.

Yes, edge side includes are very interesting indeed! We used that for scenarios like you describe, but there are actually some things you should take into consideration when using them. Maybe I can dedicate a blog post to that! :)

Please do, I'd love to learn from somebody else's experience with them

This is the first blog post I wrote and actually published. Any comments on content, but also writing style etc are very welcome!

I think it's really good. One thing I'd change (not that I'm a good writer/blogger) is a clearer description in the first paragraph of why you'd want to do this.

"By using context aware caching, you can achieve high levels of caching for both anonymous and logged in users, unlike a dumb cache which generally excludes logged in user requests" or something like that.

Ah yes, the introduction talks about caching, but doesn't say anything about the title. I'll try to keep this comment in mind next time!

It's just very good, please do continue. Also, the drawings are cool but that dark red is not contrasty enough with the number inside it, you might want to improve that for the upcoming posts. Keep up the good writing!

I took the photographs with my phone, next time I'll use a better camera. Changing the red is probably a good idea anyway. Thanks!

I thought it was a great introduction to the topic. I found the jump to pre-authentication a little jarring, perhaps adding an explanation as to why caching logged-in users is a problem would help.

In the later examples, I don't think it's very clear that you're using the timestamps to show which responses are fresh and which are cached. Coloring the fresh/cached responses differently in the output might help.

Look forward to your next post.

Hm, I'm not sure how to introduce the pre-authentication bit even further. Maybe it isn't the best subtitle.

I agree that colouring the output could make things more clear. I'll keep it in mind for future articles!

White balance those photographs! Also you can make them much smaller with no apparent loss. http://i.imgur.com/EQulv.jpg

Next time I'll take the photographs with a dslr camera instead of my phone. :)

Excellent job... I think a second article in which you give specific examples of how Context-aware caching would work for a simple application would be helpful. Maybe a group chat service would be a great example. Also, adding HTTP requests within Varnish allows for a much more useful cache invalidation scheme... for example, incoming requests could have an content_token and timestamp. An internal HTTP service that records whether a given piece of content has expired (through user action rather than TTL) could then be checked before serving up a cached copy.

Thanks! As for the examples, I think I will write some follow-up articles with more concrete use-cases and features of Varnish.

Interesting as I'm not too familiar with Varnish. However it seems to me like it would be easier in many cases to pull out the session ID and then have a simple script which uses that to set the X-Auth-User header since most sites using authentication do so with session cookies.

The approach described in the blog post can be used with all sorts of authentication schemes. I used an access token as example, but you could also use the "Cookie" header to get a session id.

This article brought back to memory the first paywall like functionality that was implemented in Varnish, since it was done using the cURL VMOD (and the reason why it was developed by Varnish Software in the first place).

"Development of this VMOD has been sponsored by the Norwegian company Aspiro Music AS for usage on their WiMP music streaming service." from <https://github.com/varnish/libvmod-curl#readme>;

This is great thanks. We use Varnish with our current site iteration and it is a life saver. It keeps us running without any significant re-architecting on the backend.

Currently, new directions do not include Varnish but I was starting to re-consider that position and this article was a nice reminder. Thanks!

I think it also depends on how you use Varnish. A lot of web applications can benefit from the reverse proxy capabilities of Varnish, but if you make Varnish part of your application stack from day one you can do things like I describe here. :)

It reminds me of the recent Varnish Paywall article:

http://highscalability.com/blog/2012/9/12/using-varnish-for-...

Reply on news.ycombinator.com