Hacker News Reader

Comment by pg | Hacker News Reader

We have the fastest server we could conveniently get about a year ago. But I just asked Rtm if we could get something better.

Unfortunately processors don't seem to get faster at the rate they used to. Now they just have more cores, and I believe MzScheme can only use one core.

I think the solution is to make the software faster. There's probably some small hack we could make somewhere in the guts of the system that would make it 10x faster. I'm thinking of having a make-HN-faster contest, but first I have to release a more current version of the code.

View on HN

Since version 4.2.5 MzScheme can use many cores with future and touch (http://docs.racket-lang.org/guide/performance.html#(part._ef... ) But they apparently they are tricky to use.

For example the manual says that to run the computations in parallel, you should no use operations that mix exact ans inexact numbers. It is possible to do this by hand writing the scheme code, but I think that it is more difficult to make the arc->scheme compiler do this automatically.

I believe MzScheme can only use one core.

How annoying. http://osdir.com/ml/lisp.scheme.plt/2003-01/msg00216.html suggests that you're right, but MzScheme _used_ to support multiple cores.

How difficult would it be to divide HN up into multiple cooperating processes that each get their own core? With a little caching, it should be possible to get a lot more headroom.

Would it be possible to layer arc on top of clojure or is that too far distant from each other?

That's a bit much to resort to when AFAIK caching hasn't been pursued aggressively as it could be.

Typically if you start the work when you need it you're actually too late, if this is a direction that is viable the work could get underway long before there was an immediate need and it would probably be done that much better for that reason.

Again, I have no idea if this is even feasible, the differences between the various dialects of lisp seem to me (as an outsider) to be considerable.

To my knowledge, Arc, like Scheme, uses mutable conses as its main data structure; Clojure doesn't use conses in the sense that the cdr-equivalent element of a list cell can't be just anything. Clojure's data structures are also immutable. You'd have to do a lot to re-engineer the basic data structures just for that; I doubt Arc would be the same language at the end of it.

I'm sure Mz/Arc can be made to run workers in other threads or processes in some way, even if it's not trivial. I also get the impression the caching is implemented in Arc code rather than as a HTTP proxy. The latter could run across multiple cores or even servers. Finally, it may be possible to decouple the user-specific data from the generated pages themselves and cache that information on the HTTP level. I admittedly haven't given it a lot of thought, and I also haven't read through the public HN source yet.

Ignoring myriad other differences, Clojure's syntactic improvements on Lisp/Scheme alone would create far more work than would be worthwhile.

It's simply unnecessary and excessively painful in the face of much better alternatives.

Like cutting the bloody limb off when you haven't even tried a penicillin derivative yet, a common "first step".

I agree, given some representative sample data and a goal ("make rendering this page go fast"), finding speed improvements likely wouldn't be very hard.

but first I have to release a more current version of the code.

Please do that, some people are dying to see some of the stuff that goes under the hood. Of course things like hell banning may not be released, but some of the ranking and "ask" page stuff would be awesome to look into.

http://ycombinator.com/arc/arc3.1.tar

If you didn't know, that does not have some updates that pg makes on the fly. I have been managing hackerstreet.in, which uses that source, and I can tell it is outdated. Please actually see links before you are linking.

It's the most recent code that's publicly available. I'm aware of the fact that there has not been an Arc update recently.

Reply on news.ycombinator.com