Using Flat Files So Elections Don’t Break Your Server

Doesn't Varnish handle many times more concurrent clients than apache does, at significantly lower system load? It could just be pure optimization.

So I know nothing about haproxy, but what sort of system load does it generate?

They say they ran it on a EC2 micro instance, and the micro instances are specifically designed to handle spikes, not continuous heavy load. In fact, they intentionally throttle under continuous heavy load.

I guess it must have been the right choice for them, I was just surprised to see it was a micro. Would that typically be the right way to go?

If you look at the current webserving stacks propagating out there, wev'e gone from apache to multi-layered stuff, haproxy/varnish/nginx/apache/mongrel/rails/ various fcgi/etc..... and the thing to keep in mind is there is overlap between most of these projects.

Apache can, one way or the other, do most of what the others can do - possibly not as well, and at the risk of a much more complex configuration.

Haproxy does one thing and does it really,really well - it's great for dealing with load balancing, concurrency limiting per user defined resource, and identifying and routing incoming requests to the right infrastructure. It's really good at this - that's what it does.

Varnish does one thing and does it really, really well - it caches content and serves it up (usually out of memory, but even if it's swapped, it's optimized) to keep the load off your application servers. It has various optimizations built into make it really good at this.

So - in this case, the answer might be "not much - we could just put the apache's out front behind haproxy" - but it appears putting the varnish server out front with a 5 second cache dropped the load on the application servers (in this case apache serving static files that it receives over rsync - dont' forget the rsync part - resources are needed for that). This might result in smoother output for the end user, rather than something hitting a node that's busy servicing an rsync update. It may also be their engineers are very familiar with the haproxy/varnish front end setup, as it presumably exists in their current day to day operation as well.. so the people responsible for keeping things up probably decided "Yes, we'd like to keep it there - it makes our lives easier."

There is an operational anti-pattern in there - removing too many elements from a known system is also a kind of added cmoplexity - all your troubleshooting methods disappear.

Their goal here was to de-couple the dynamic elements from the event-driven side of things and turn them into something more resilient (and less flexible) for a short time to deal with unknown and unpredicably large load.

> A static file, or more simple, a HTML file is what they're actually talking about.

Static yes, HTML no, an HTML file could still have e.g. SSI instructions. It's valid HTML, but if the webserver supports SSI it's not going to be static.

Your threshold for amazing is pretty low. :P

You get 1000 requests a second from Reddit? Wow, it's a whole lot bigger than I thought.

Chances are that you will need to do some serious kernel tuning before you will really run in to the limits of what varnish can do.

If you're using linux using IPV4 have a look at the following parameters in /proc/sys/net/ipv4:

tcp_tw_recycle

tcp_tw_reuse

Tuning those will help in allowing faster re-use of sockets in the TIME_WAIT state. This matters because at the defaults your sockets will linger for a long time before being allowed to be re-used (as per the RFC). Technically this is the correct behaviour but it can quickly become a bottle-neck.

Ulimit max open files per process:

ulimit -n 50000

The default is just 1,024, and that's not nearly enough to keep varnish working hard.

Those are the first things to look at, there are many more once you start to bottom out again, but this will get you started.

A good way to see if you've got your kernel tuned properly is when varnish starts to approach the limits of your hardware, I've seen it do well over 600Mbps on an otherwise unloaded box.

Rather than repopulating the same cache entries every 5 seconds you could instead actively purge the Varnish cache entry when the content actually changes.

Why stop at 5 seconds? Unless you're updating that blog entry every 6 seconds, you're missing out on a bunch of good caching by regenerating it 12 times a minute.

How about cache it once at write-time, then again whenever you edit it?

On the drupal side there is Boost: http://drupal.org/project/boost It has some fairly smart cache invalidation logic as well. At work we use the cache invalidation logic to tell varnish what needs to be refreshed. We use this setup for over 1,000 TV station news sites; supported by 10 boxes total. Flat file cache size is around 80GB. Our load is usually under 2 on all boxes.

Sounds very much like it. From the article:

>After each batch of new data was received from the AP, this server determined which pages needed to be re-rendered and, using the Typhoeus libcurl-multi bindings for Ruby, pulled new data for each of these pages from the render pool.

Sounds like their infrastructure already has a farm of servers set up to handle rendering articles into their HTML components, so for this scenario they would save the output as a static file and then write it to disk rather than sending the HTML out to the response / caching layer / etc.

Yes. The diagram explains that their render farm takes data from AP and converts them into pages that are then sync'd with apache.

When dealing with unpredictably high traffic spikes, on days when you cannot afford any kind of downtime - when that downtime comes at great cost, it's not overengineering (and it's hard to determine if it's overengineered until you know the budget and time spent on the actual project - this may have been a relatively simple modification all things considered)

Static content is easy to crank up to web-scale. You can use DNS, any kind of load balancer, all kinds of web services, CDNs, whatever.

Dynamic content is hard to scale (compared to static). You have several layers of added complexity. Yes, you can build wonderful, self-scaleable systems - but at some point they hit a limit, there are many more resources that can be tied up, and troubleshooting and scaling that out beyond anything you've previously imagined on short notice can take time you can't afford.

So - simply de-coupling the dynamic content generation from the static web serving is a great way to make a clean break - you now have a known & tuneable load on your dynamic application (because your'e running it at known intervals, rather than being event driven by user requests) and you have a front-end static infrastructure that you can scale like mad, and even if your back-end collapses, edit by hand.

Surely there are other ways to approach the problem.... but it also depends on the engineers involved, the time taken, and their confidence in their ability to deal with it.

I'm also fairly sure they aren't the first company out there to take this approach to burst scalability issues... but it's curious to note how the NYT actually operates.

TL;DR: Look at the old configuration, and the new configuration. Decide which one will best serve your business in terms of your ability to troubleshoot it when it gets hit by a level of traffic higher than you can plan for, because you have NO idea how high it will go.

In my book, overengineering is defined as doing more work than is necessary to ensure that the risks are within acceptable bounds.

As the author points out, having the Times election site go down on election eve would be a BIG PROBLEM -- massive losses in both reputation and advertising revenue. For the system architect, a failure could possibly mean losing his job.

What the author has laid out is a system that is robust to multiple, simultaneous failures (with possible exception of the loss of AWS, although that's not entirely clear). That just seems like good planning.

The file-system isn't just another data-store: it's I/O and therefore special. On systems with 'sendfile' data can be sent straight from disk to network port without CPU time (and any copies being made). This is used extensively by high-performance webservers and web-accelerators.

Ruby on Rails is generating the flat files, while apache et al. are serving them.

It's risk mitigation... this is one way you can deal with unpredictably large spikes in traffic.... for the NYT - elections probably represent their extreme.

For the regular ebb and flow of data, you can plan, and use varnish/rails/nginx/ all the magic tools you want, and things will work great - but you also have added complexity, and when something goes wrong, the more complex the system, the longer it generally takes to fix. Especially under unexpected heavy load.

Simplifying the system down to flat files and de-coupling the ROR stuff gives you a clear troubleshooting point - if something goes berzerk, you can cut the link between the two and troubleshoot in relative safety before turning replication back on.

I'm rambling - the main point is to reduce the complexity involved in the end-user transaction to be as efficient and fast as possible so you can deal with an unknown load factor coming in on a really important day. Going down that day would be BAD for business.

The load was for election night, which is more than likely a full order of magnitude, or two above their normal traffic levels. So their normal setup of RoR is fine for normal amounts of load, but would fail under that expected spike. So they used all their page-creation code in rails, but instead of serving it out like normal, they create the static files, and serve those. You lose out on interactivity, but make up for it in performance.

> I may sound naive here, but: if all you're doing is serving just 184 flat files, then why do you need all this RoR jazz?

Generate the static files: those files contained live election results, so they had to be regenerated as new results came in.