Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines") | Hacker News Reader

Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

lib.store.yahoo.net

29 pointsMobileDigit20 comments

I always assumed it was, and that the closures are stored in the user's session (in memory, on the server). You notice that if you leave a comment reply box up on the screen long enough, you'll get "Unknown or expired session" when you try to submit, and have to back up to the main comment page before you can try again?

If they expire sessions in order to avoid losing memory over users leaving without ever calling the closure, their timer seems short for what it gains.

The link name "fnid=" is a pretty strong giveaway as to what is underneath.

Yes, I find it too short and the end result is annoying but passable for the audience here. Non-technical users wouldn't be so understanding I suspect.

How are hash keys deleted? Age? Or are just the last N generated ones kept?

[silly test to see if they remove the closure from the HT once it's used, and the answer seems no]

Correct, unless the closure explicitly removes itself when executed, of course.

For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging in when you're not logged in and you submit a link through the bookmarklet) is part of the state of the closure; in that case it's a bit like a continuation.

We keep only the most recently generated 20,000 closures. When you click on a link that says it has expired, that means either it has been purged from the closure table, or that we've restarted the server.

Isn't there a risk, that someone can (accidentally or not) enter hash-code of someone else's "closure"?

One could avoid that problem by keeping track of the owner of each closure and ensuring the user is properly authenticated before calling the closure.

The code explicitly protects against that.

If access to the closures are session-specific, you'll have to hijack the session to do that.

"in that case it's a bit like a continuation."

Doesn't MzScheme (assuming it's what Arc is built on) have first-class continuation already?

It does, and so does Arc, but I don't think we've needed them in anything we've written so far. News.YC is a pretty simple application.

Can we expect to see Arc go public in the near future?

From my experience of continuation-based web app, what you're dealing with is not a full continuation captured by call/cc, but rather a delimited (partial) continuation, which captures the continuation of your application logic but not the state of underlying protocols. You can implement delimited continuations on top of call/cc, and I did based on Gasbichler&Sperber paper (ICFP02), but in most cases explicit CPS wrapped in some macros are just as well worked.

What extra data is in the closure for "reply" that means it isn't a simple URL? Is it just what to do after the reply is complete, i.e. what post to return to viewing? Is there a reason this isn't taken from the Referer instead?

I guess this technique doesn't scale too well unless the hash table is either shared across servers or a user is kept on one server, which then causes problems of its own.

I'm guessing 20,000 closures is plenty and I've been bitten by server restarts. Out of curiosity, do you keep track of the minimum age of the 20,000th hash entry prior to deleting to make room for another? If that's something huge, it would confirm server restarts are the normal reason the "unknown fnid" message is seen.

What particular problems are caused by the user being kept on one server?

The redundancy of having five servers isn't helping if I'm always served by server #3, whereas an alternate method where any one of the five can answer my current request allows a machine to fail, or be taken down for upgrading, and the load on the other four just increases by 25% each.

Paul, could you enlighten us how well does this type of development (Lisp macro/closure based) integrate with JS and Ajax? I'm interested in asynchronicity of the Ajax... Ruby and Python with Protoype and Scriptaculous make this very easy indeed...

How much Ajax do you see on this site? (The answer is that I don't know, because I haven't tried.)

In my experience with Seaside (which uses roughly the same technique), extremely well. There's a demo site at http://scriptaculous.seasidehosting.st/, fwiw.

Reply on news.ycombinator.com