Comment by thaumaturgy | Hacker News Reader

Comment by thaumaturgy | Hacker News Reader

Parent Full thread

It just so happens that I have the very best test site possible for this argument: one of my alpha customers has a website built by Not-A-Professional with an incredible 156 requests and 2.39MB of data for the landing page. It is exactly the opposite of an optimized website.

So I tested the page load with KeepAlive on and KeepAlive off. I `/etc/init.d/apache2 graceful`'d before each test, and I cleared Chrome's cache before each test. I have nothing else running on my system right now and our pipe is nice and clear. I have a 92ms ping from the office to my web server.

KeepAlive On: 5.31s

KeepAlive Off: 6.87s

1.5 seconds is an appreciable difference. It works out to roughly 10ms per request, unless it's so late that the math part of my brain has quit for the night. If Yahoo.com were loaded from my web server, having KeepAlive on would delay total page load by about 750ms -- a noticeable difference. (However, given that it takes over 4 seconds to reach page load on yahoo.com for 70+ requests, their page might actually load faster if it were hosted on my server...)

Now here's the kicker: that same customer site has been featured on a popular radio show, twice, driving over 6 million hits over 24 hours, with peak traffic at 40Mb/s, thousands of simultaneous connections, and 100+ requests per second, sustained for several hours.

The site's loading time during that period was utterly unaffected. Seriously, never even a blip. (I stayed up during the whole period just to personally watch it.)

I wasn't about to experiment with turning on KeepAlive during all that fun, but I'm reasonably confident that having KeepAlive turned on would have crushed that server, given that my customer's site was receiving substantially more traffic than even Pinboard did immediately after the Delicious announcement (http://blog.pinboard.in/2011/03/anatomy_of_a_crushing/) -- traffic that Pinboard's founder notes will usually kill an unprepared site (or server).

And keep in mind, this was all on a memory-constrained tiny little vm.

So here's my final statement on KeepAlive, which I will link back to from now on: if, as a server admin, you're turning KeepAlive on to speed up your site, then you are optimizing your server and your site in exactly the wrong way.

You're not providing any data to say having KeepAlive on would've 'crushed' the server, actually quite the opposite. For example - with KeepAlive on, your site response time was 5.31s. With KeepAlive off, it was 6.87s. So with your own test it was slower to load.

With keepalive off, your server would spawn new connections for every request rather than re-using them. With mpm-worker and threads, that's not so bad. However I've seen a lot of people run pre-fork which uses processes instead of threads which can be a Bad Time(tm) with heavy spikes in traffic.

Anyway - found this here (not my site) - explains it fairly well: http://abdussamad.com/archives/169-Apache-optimization:-Keep...

From one of my other replies to the KeepAlive discussion: http://news.ycombinator.com/item?id=2588783

I promise you my little server was getting more traffic than that, and since patio11 -- another vocal "turn off KeepAlive" advocate -- personally commented in that thread that turning off KeepAlive directly resolved their site outage, I think I'm going to stick with what I've got.

The server is already using mpm-worker, as well as fcgid carefully tuned to make PHP nice and snappy without eating all of the server's available memory.

Final point: site response times aren't the only metric a server admin should care about. A server admin should also -- and maybe even primarily -- care about concurrent connections. If I can serve thousands of simultaneous connections at 6.87s or hundreds of simultaneous connections at 5.31s, guess which one I choose?

1.5 seconds is a lot of time to save off page load and this shouldn't be just ignored. Other people do ridiculous things to their architecture to shave off just a dozen milliseconds no one cares about.

I mean, KeepAlive shouldn't crash severs. Have you tried tuning other settings, like KeepAliveTimeout? It's either a config issue or general Apache issue, but it should be solvable in both cases. There is no unsolvable problem that would prevent a server from persisting connections under lighter load and dropping them when there is more traffic.

I liked your other comment on this better, fwiw.

I'm trying pretty hard not to get defensive here, so before I respond, I need to point out that I originally posted a comment intended to help people get a better Apache config; I didn't claim anywhere that it was the perfect one, or that there weren't better ways to go about it. I posted the comment to a thread started by a post clearly aimed at total newbies (and SEO), which is why my comment stuck mostly to really really simple stuff, and didn't include other notes, like changing the server's file descriptor limit. Now I'm letting myself get dragged in to a debate over, frankly, minutiae that doesn't belong under a post about "setting up your first web server". I work all the time on a mind-blowingly wide array of technologies, yet I never talk about it because

I

hate

this

shit.

So much.

So, that said: if my customer wants to shave 1.5 seconds off of their page load times, the very first thing they need to do is (drumroll...) _not_have_156_resources_on_their_landing_page_. We're not talking about 1.5 seconds for the average Wordpress site here (54 requests, based on a Wordpress blog picked at semi-random). So let's just be clear that we're talking about a 10ms delay per request over a connection with a 96ms ping.

If they did that, and still weren't happy with the results, then the next thing I would do is pay for a bigger VPS instance.

If I did that, and they still weren't happy with the results, then the next thing I would do is move their site to a server in a data center closer to their geographic location.

If I did that, and they still weren't happy, then the next thing I would do is charge them a brand new premium rate of about 10x what they're paying now, and I would re-configure my PowerDNS servers to do geoip-based results and I would set up nginx proxies (all of which I intend to do eventually anyway).

If I did that, and they still weren't happy, I still wouldn't bother farting around with KeepAlive, because at this point I've entirely removed Apache from the page load equation because doing all of this is easier than screwing around with the Apache config any more than I already have.

Please don't get me wrong. I think performance is really really important. It's a frequent soapbox point for me. I have already sunk a ton of time into making my server setups secure and fast and reliable. I will continue to sink more completely irretrievable time into that. However, as geeks it's easy for us to lose sight of what's actually important: should I spend more time trying to remove a 10ms delay per request without sacrificing reliability, or should I answer some more support requests and get another job done on time?

I also have to confess that part of my problem is that I don't trust any of the web server stress testing tools that I've used so far. Like, at all. It's possible that I'm too dumb to use them, but I can't find one (even a paid service) that I can use and say to myself, "Yeah, so that's what would happen if the website got on the front page of Reddit." Without some kind of really solid tools to use for testing, I'm disinclined to try to squeeze 10ms of extra performance out of the server and take it right up to the hairy edge of what a VPS can do. If you have a recommendation for a stress tester that you feel does a good job of emulating front-page-on-Reddit syndrome, by all means, please share. I'd love to try it, turn some more knobs, and maybe even write a post titled "the best Apache-on-Linode config for surviving the front page of Reddit".

interesting.

do you mind pastebin-ning your apache conf and your mysql conf ? It would be nice to see what parameters are you tweaking.

Which VPS hosting are you using ?

I'm using Linode. I can't say enough good things about them, btw. Their support is mind-blowingly good, and their service isn't too far behind.

I have zero incentive to post my config, sorry. One, it would take me an hour or so to format it for general consumption; two, it would take me many many hours more to justify every single setting in it to anybody who read it.

We've got a website overhaul in the works, which will feature a blog which will actually get used & updated, and one of the first items on there will be our server configurations. Promise.

Reply on news.ycombinator.com