For the first 3 years it was Apache+Prefork+mod_php+WordPress (default setup for any on RedHat-based or Debian-based setups for the longest time).
Any time I would get a story on Slashdot or Digg the site would die for at least half a day... god I hate it.
I went from a 1GB RAM VPS to a 12GB dedicated machine in 3.5 years trying to get it to stop crashing whenever I would get a flood of traffic and was never able to. I pushed back on the idea of being a Linux sys admin for so long because I didn't want that hassle, but alas, I had to bother with it.
Finally at about the year 4 mark I decided no REAL site on the planet was running with this configuration since it didn't seem to matter the hardware you threw at it (yes I tweaked the Apache setup/mod-list tirelessly to scale with the improved hardware). I finally started digging into how real human beings setup Apache and came across the argument for using the "MPM worker" as opposed to the default pre-fork worker.
Made sense to me; less 30+ MB processes running around answering questions.
After that change, it helped a little... I have no hard numbers on hand, but it felt like a small improvement.
I kept digging and soon ran across the one-hacky-but-now-officially-supported method of using Apache + a family of PHP VM threads pre-launched and called via FastCGI to execute the .php pages from my WordPress site, the computer-science part of my brain loved this idea... the Java-trained side of me suddenly made me realize that prior to this with Prefork and mod_php, every time someone was connecting I was spinning up a new Apache thread and a new PHP VM every single time (please correct this if wrong... this is how I understood it).
With FastCGI I could have a family of say 20-some PHP VM threads living in harmony and responding to Apache constantly asking them questions.
After rolling that change out at about year 4, I noticed a big improvement; maybe about 50%.
At my next Slashdotting the server got REALLY slow, but hung in there; no crashes. I thought it was odd that all that hardware still couldn't host things snappy... it seemed like every other day I was clicking a link off of Hacker News or Reddit front page to some dude's personal blog that was responding very quickly to me and I was positive these people weren't spending $300/mo like I was on dedicated hardware to run their blog.
So I kept digging.
As you guys probably know, when you start searching for what sucks about Apache two things come up more than any other: "use nginx" or "use lighttpd" -- I had read that early versions of lighttpd had some memory leak issues (I think long-since fixed) and had a handful of Ruby friends that loved nginx... so I decided to stay up all night one night and port the site over.
25mins later I was done.
Yea so that was a lot easier than I expected. The only painful part was using some heavy handed redirect logic to convert my WP-SuperCache rules over to nginx (the author wasn't supporting nginx yet, but I think he does not).
I would point out that the server load with nginx running with NO CACHING (WP-SuperCache disabled, all queries execute PHP and perform a MySQL query) was something like 1/4 what my Apache/MPM/FastCGI/PHP/WP-SuperCache-enabled setup was using.
Once I got WP-SuperCache up and running on nginx, the different was stupid-big. The nginx/WP-SuperCache setup was using 1/8th or 1/10th the system resources that the Apache setup had been using.
Soon there after I got Slashdotted with a peak of roughly 400 users on at a time and the server load barely crawled beyond a 1 (screenshot): http://www.thebuzzmedia.com/wp-content/uploads/2010/12/slash...
Overall I couldn't be happier with nginx. I think there are probably people that live in oxygen-rich test chambers inside of military bunkers who were bread to tweak Apache that can optimize it to have comparable performance, but that wasn't me. Out of the box nginx has been fantastic thus far.
And that is my little story related to this subject... for what it's worth.