I will also quote myself: "I first learned to disable Apache KeepAlives in 1998. Yes, 1998. It's disheartening that Apache still ships with it enabled by default. It has always allowed a relatively small number of lingering clients to completely DoS your server."
Key points:
• Make sure that wordpress supercaching is on. You can verify this by looking at the last line of the HTML that comes back which has a timestamp for when it was generated.
• Turn off KeepAlives.
• Set MaxClients to 8.
• Use monit to check to make sure it can connect and restart apache with a kill -9 if it can't. (This is optional, but helps if you have some random thing that ends up taking a very long time to execute and eats up connections.)
With that in place even a much smaller server can easily handle a HN top story without breaking a sweat.
LOL what? what's your mpm? did you enable this to keep the backend from blowing up from too many queries? surely you can handle more than 8 connections at a time. is this the proxy layer, and if so are you using web caching on top of wordpress caching?
if monit is restarting apache every time it can't connect (i hope you have a long timeout) you're denying service to a lot of people. connections are supposed to queue so they don't get dropped.
My point, specifically, was how low you can go with a cheapo VPS. We hold up fine during an HN spike. (We're B2B and not a destination site, so our usual load is trivial.) Even during an HN spike you're getting tops of 2-3 visitors per second, which can be dished out reasonably well with 8 workers.
The monit thing kicks in after a 30 second timeout. With the configuration above, we don't get that because of load, but rather when something else has gone wrong (specifically there's a wordpress plugin that our internal status blog uses that sometimes hangs). But given the original poster's issue of apache getting so out of control that it took him several minutes to get a live SSH connection and a system load of 60, having monit kill things (and restart them) is a preferable stop-gap.
(Note: Our actual customer facing stuff is quite different; there we're using multiple servers behind an nginx proxy and using a combination of Rails, Sinatra and Java services. The basic web stuff is segregated off from those primarily for security reasons.)
If you have some free time try deploying your LAMP stack with Buildroot and uClibc. The application size ends up being around an order of magnitude smaller, but i've never bothered checking if private RSS on clunky apps like PHP or Perl is minimized at all.
edit Also for prefork we used to have some scripts that would monitor processes to see if they 'went crazy' and wouldn't ever return, and reap those processes so Apache would fork a new one so we didn't have to restart the whole server. When you're under peak load and you restart a server and it sends all those clients to all your other servers which are already almost at their peak things get very nasty very quickly. I know you only have the one VPS here but for applications with many servers it can be handy.
HTTP keep-alive connections are faster (saves 3-way handshake on subsequent requests and allows pipelining).
The problem is that Apache implements them in a horribly inefficient manner (thread/process and all its memory kept in use just to hold on to a socket).
If you use nginx, lighttpd or shield Apache with haproxy/varnish, then you can easily have keep-alive enabled and clients will see better performance.