Ouch | Hacker News Reader

Ouch

antipope.org

123 pointsMaysonL32 comments

Please could you make the submitted title a little more informative? Perhaps:

    Ouch!  (First hand experience of a massive load spike)

Thanks.

> 4. Tune Apache now — you won't get a chance to do it when the stampede arrives!

Or take the easy route and ditch Apache.

I'm not saying that Apache can't be great (people who know much more than me use it), but if you need to quickly serve up a ton of requests without bothering to tune this and tweak that, there are better choices, in particular for static pages.

Cherokee's my preferred, but lighttpd, Nginx, thttpd, etc. are all good, too.

Ditching Apache is only "easy" if (a) your blog software isn't designed around Apache-isms like .htaccess and mod_rewrite, and (b) you have time to find and comprehend the documentation for your alternative server.

Nginx is great, for example, but the docs are primitive compared to those for Apache, which has been around forever and which has spawned thousands of hints and snippets all over the web.

EDIT: At this point I should probably say out loud that for accelerating static pages it might well be simpler to get a reverse proxy, e.g. Varnish, running in front of Apache than to actually ditch Apache.

Let me add: I am paid to write books, not paid to ride herd on servers. Apache's what I know from days of yore; switching to a new platform means taking an opportunity cost hit.

Having said that, I'm looking at Varnish (once the stampede dies down) ...

If you add Varnish, you'll still need Apache.

If you switch to nginx, then you probably won't need varnish at all and can replace Apache.

The nginx config is much, much, much easier to work with than Apache. Transferring complex rewrite rules is the hardest thing to get right when moving over, but everything else is much easier.

Smart lobsters use nginx

I compared changing the Apache MPM, Varnish & nginx for a high-traffic, low memory (256M) server.

* Changing Apache would have required changing other packages - PHP wasn't thread-safe.

* Varnish looked fragile. I wish I still had my notes about why I thought that.

* Nginx in front of Apache was relatively quick to setup & really great in practice - restarting an order of magnitude less often.

If the nginx config file would be useful as a starting point, I offer it.

Edit: formatting.

Honestly - publish it on a blog somewhere or shove it into a github repo or whatever - but if it was useful to you, it's quite possibly useful to somebody else so you should publish by default rather than merely offering.

I'm not sure if it would help me, but I'm almost certain it would help other people - so please, consider your offer accepted if you can find somewhere easy to publish this :)

Why is varnish fragile? It is a bit odd to configure, but that seems like a set and forget kind of a thing.

The Varnish configuration is hairy - manually tweaking cookies to cache images for everybody while avoiding duplication, denormalizing headers so that IE & Firefox can both get the benefit of compression, dealing with pragma & cache-control, etc.

(Compression & other varying content was a headache - separately caching & serving gzip, deflate, & unpacked versions, without any built-in compression.)

Those tweaks looked time-consuming to set up, hard to test, & liable to break quietly. A buffer in the way that simply & quickly gets a response from (& then frees up) Apache, then slowly feeds the response to the client, provides most of the benefit; nginx fills this role well for me.

On one memory-constrained server, I have nginx reverse proxying to Apache, so the Apache-isms are not an immediate problem. I have nginx bypassing Apache for some static files. Having a ton of slow connections was a problem for Apache, but not for nginx.

Cherokee is actually pretty good at following .htaccess and mod_rewrite for the big blogging/cms engines (like drupal, wordpress). It is much easier to configure, though perhaps a bit more tedious, as it is all done through the web gui.

Other than that Apache is pretty much king with compatibility and nginx is pretty hard to get to similar qualities off the bat.

But if you have the space something like a reverse proxy for Apache with sane times set would probably be the better option.

I use cherokee on VPS's though and it works great.

I've moved almost exclusively to Nginx, once you get your head round how to do the "Apache-isms" (great potential neologism that :D) in Nginx configurations it becomes second nature.

Cherokee is a possible alternative too; that has some default configs for Wordpress etc. that work well. But for anything custom I found it a nightmare to get "just so".

Nginx is a neat alternative to Varnish as well; for example I'm having great success sitting it in front of a Cherokee site (serving the http) to provide SSL :)

Is varnish a drop in fix for something like this while everything is going to hell, or is there some monkeying that has to be done to make it play nice with movable type?

nothing really is a drop in solution for traffic spikes, however, there are a plethora of articles about how to make Varnish play nicely with moveable type:

http://www.coldhardcode.com/2009/04/varnish-and-movabletype-...

The same is true for wordpress:

http://www.varnish-cache.org/trac/wiki/VarnishAndWordpress

I understand this is HN and all, but has anyone even bothered to take a peak at the EXTRAORDINARY content that was behind the spike?

In a nutshell, it is a transcript of a British parliamentary session in which the speaker claims a shadowy yet totally legitimate organization--"Foundation X"--is willing to donate billions to the British government no strings attached. It is extremely engrossing.

Yep: http://news.ycombinator.com/item?id=1864932

Here's video of the speech (starting at hour 2, 34 minutes). Exceedingly odd. http://bbc.in/9qS59B

I still don't know if this is to be taken seriously. What exactly could be going on?

Sky news is reporting on it too, and while it looks like it could be a work of fiction (it sure reads like one - Dan Brown comes to mind), it apparently isn't.

http://news.sky.com/skynews/Home/Politics/Foundation-X-Lord-...

I've also experienced similar spikes before. It's important to added that everything will slow way down... that includes SSH access.

Even if you think "I'll just jump in there and correct the issues if/when the time comes" you may not be able to if SSH access slows down to a crawl.

Like the post says, the best course of action is to be prepared ahead of time and not trying to make tweaks while the traffic spike is happening.

And, for God's sakes, turn off Apache KeepAlive. It kills more blogs than cancer.

Or switch to nginx, which eats keepalives like candy.

Or use the event MPM.

Or put your blog on posterous or blogger.

There are several reasons my blog isn't on blogger.

First: I like to own my own IP. And I don't want to be vulnerable to a spurious takedown notice if some random griefer decides to take aim at me.

Second: blogger has some hard limits -- official list here: http://www.google.com/support/blogger/bin/answer.py?hl=en... -- and there are anecdotal reports that there is a limit on how many blog entries you can have.

I seriously considered Typepad, but once I worked out how much it would cost it was cheaper to stick with a dedicated colo server (hint: there's more than one blog on the system I run, and there's also a mailman server and some other stuff).

Also: I've been running on a Linux colo since 1996/7, and I really don't feel the need to take a couple of weeks off work to re-jig my entire online presence. This is the first time I've ever had anything like this load level. If it starts happening regularly, I'll reconsider my strategy. Until then, it's a one-off.

What's the current name for that? Do we still call it "slashdot effect"?

Sometimes it's the reddit effect, or similar. However "slashdotted" should remain understandable to anyone.

Though in this particular case, the site seems to have been Gaimaned.

And boingboing, and reddit x2, here, and a bunch of newspapers in the uk. I couldn't go anywhere yesterday without tripping over that link.

He got slammed.

If you want a picture of the future, imagine an internet stamping on a single server— for a day anyway.

Isn't the general term "flash crowd"?

" I don't have an exact breakdown of what just happened, because my logfile analyser is still churning, from 2am last night.That's because the Apache logfile today is over 500Mb it's normally about 150Kb. " The post is timed at 11:31.

I think another lesson learned is that the log analyzer needs to be analyzed.

If the server's under really hard load - memory or I/O or both, with the swap thrashing - I'd be unsurprised.

I've never checked to see how my stat analyzers fare on half a gig of logs.

Reply on news.ycombinator.com