Ask PG: Would you accept donations towards buying a monster server for HN? | Hacker News Reader

Ask PG: Would you accept donations towards buying a monster server for HN?

26 pointsjacquesm17 comments

HN is pretty slow again, and has been for some time now, connections timing out the rule rather than the exception when posting an article or commenting.

I can see why it's hard to scale HN using multiple servers, what with ARC not being the most cluster friendly platform, but there is a different solution, simply getting a really badass server.

I'm sure that you're not wanting for cash, but since we all use this resource and we all profit from it in various ways would you accept donations that would go to buying a server large enough to see HN through the next phase of growth?

View on HN

We have the fastest server we could conveniently get about a year ago. But I just asked Rtm if we could get something better.

Unfortunately processors don't seem to get faster at the rate they used to. Now they just have more cores, and I believe MzScheme can only use one core.

I think the solution is to make the software faster. There's probably some small hack we could make somewhere in the guts of the system that would make it 10x faster. I'm thinking of having a make-HN-faster contest, but first I have to release a more current version of the code.

but first I have to release a more current version of the code.

Please do that, some people are dying to see some of the stuff that goes under the hood. Of course things like hell banning may not be released, but some of the ranking and "ask" page stuff would be awesome to look into.

http://ycombinator.com/arc/arc3.1.tar

If you didn't know, that does not have some updates that pg makes on the fly. I have been managing hackerstreet.in, which uses that source, and I can tell it is outdated. Please actually see links before you are linking.

It's the most recent code that's publicly available. I'm aware of the fact that there has not been an Arc update recently.

I believe MzScheme can only use one core.

How annoying. http://osdir.com/ml/lisp.scheme.plt/2003-01/msg00216.html suggests that you're right, but MzScheme _used_ to support multiple cores.

How difficult would it be to divide HN up into multiple cooperating processes that each get their own core? With a little caching, it should be possible to get a lot more headroom.

Would it be possible to layer arc on top of clojure or is that too far distant from each other?

That's a bit much to resort to when AFAIK caching hasn't been pursued aggressively as it could be.

Typically if you start the work when you need it you're actually too late, if this is a direction that is viable the work could get underway long before there was an immediate need and it would probably be done that much better for that reason.

Again, I have no idea if this is even feasible, the differences between the various dialects of lisp seem to me (as an outsider) to be considerable.

Ignoring myriad other differences, Clojure's syntactic improvements on Lisp/Scheme alone would create far more work than would be worthwhile.

It's simply unnecessary and excessively painful in the face of much better alternatives.

Like cutting the bloody limb off when you haven't even tried a penicillin derivative yet, a common "first step".

To my knowledge, Arc, like Scheme, uses mutable conses as its main data structure; Clojure doesn't use conses in the sense that the cdr-equivalent element of a list cell can't be just anything. Clojure's data structures are also immutable. You'd have to do a lot to re-engineer the basic data structures just for that; I doubt Arc would be the same language at the end of it.

I'm sure Mz/Arc can be made to run workers in other threads or processes in some way, even if it's not trivial. I also get the impression the caching is implemented in Arc code rather than as a HTTP proxy. The latter could run across multiple cores or even servers. Finally, it may be possible to decouple the user-specific data from the generated pages themselves and cache that information on the HTTP level. I admittedly haven't given it a lot of thought, and I also haven't read through the public HN source yet.

Since version 4.2.5 MzScheme can use many cores with future and touch (http://docs.racket-lang.org/guide/performance.html#(part._ef... ) But they apparently they are tricky to use.

For example the manual says that to run the computations in parallel, you should no use operations that mix exact ans inexact numbers. It is possible to do this by hand writing the scheme code, but I think that it is more difficult to make the arc->scheme compiler do this automatically.

I agree, given some representative sample data and a goal ("make rendering this page go fast"), finding speed improvements likely wouldn't be very hard.

PG: Have you considered setting up Varnish on HN? I think this would solve most of the current issues with the performance.

http://www.varnish-cache.org/

I learned about Varnish a few weeks ago and it looks promising. I am setting it up now on one of my sites to test properly.

Varnish is indeed crazy fast...for things you can cache. The complaints I've been seeing center on slowness and timeouts for posting stuff, in which case Varnish (like the recent pagination change) might not help.

I think PG opening up the latest code and giving people a chance to actually diagnose the causes of the slowness is a great a idea. Anything else -- premature optimization.

HN is becoming crazy mainstream. Might as well start figuring out how to use multiple servers now before we're really in a bind.

Reply on news.ycombinator.com