The last couple days (especially around peak times) HN seems to have slowed dramatically. Pages take multiple seconds (think, 10 to 20) to get served and even the up/down vote images fail to load.
Is it just me?
1. I checked responses using FireBug plugin and it seems that pages are served uncompressed. For example the front page of the site (only html) is 31K uncompressed, with gzip compression it becomes 6.3K (80% smaller size means 80% less bandwidth to use). Most of web servers support compression.
2. All pages seem to be generated on every request, e.g. no HTTP cache is used on server side. Because I did not found any response headers set by caches. If for example, each page was cached just for 1 to 5 seconds it will reduce the stress on database and cpu significantly. On the other hand it will not impact user experience, because cached version is short lived and fresh results will appear on time. For example the front page is the most requested page of the site, assume that it is requested 100 times per second which means 100 queries to the database etc. If this page is cached for 1 second queries and html composition is reduced to 1 time per second. Personally, HTTP caching on server side is my favorite choice because you don't need to modify the program to use it. (super-fast Varnish cache is the most flexible solution I found).
I don't know how many people are not logged in, though. Those pages could be cached for a minute or so.
One possible solution is to decouple the personalized data and load it separately using AJAX. Votes are already implemented in a similar way.
Generally, I am proponent of HTTP caching. It requires some modifications of web sites in order to be effective but still the whole system remains less complicated, compared to implementations of cache on back-end (like `memcached').
For example reddit, which is similar to HN, is in constant struggle with their cache subsystems (http://blog.reddit.com/2010/05/reddits-may-2010-state-of-ser...). I believe it can be avoided by keeping cache and web site code apart.
Maybe a front-end cache like nginx would help?
I don't know how well something like a front end cache would do to speed up content delivery as I suspect news.yc has a lot more users logged in than normal websites which might take away the advantages of a front end cache.
The page, from the news.ycombinator subdomain is loading quick, as is the tracking code. (180ms latency)
Firefox works the best for me, as it seems to just time out and not request anything.
Caching seems to be fine - I'm getting 304s for stylesheets and images (and they are served from local cache according to firebug and chrome dev. tools) which means it actually takes 20-30s of wait time for the server to just return the headers.
I wonder if it might be problem with Keep-Alive which is on (with 15 sec. timeout I think) - there's probably bunch of apache processes / threads sitting idle waiting for connections to close and with the amount of traffic hn gets I would imagine there's quite a few of them.
I don't know what the backend is, but if it's on a single shared machine it might be too many inquires on the hardware at once. Maybe ask the hosting company to move it to a machine with a lower load.
rtm is the sysadmin for HN (yes, that rtm: http://en.wikipedia.org/wiki/Robert_Tappan_Morris).
It always makes me laugh when someone with no knowledge of a website offers advice to that site's sysadmin. It's especially funny when that sysadmin is of the calibre of RTM.
inspector has flagged 3 elements with "You could save bandwidth by having your web server compress this transfer with gzip or zlib": item, news.css and propres.php