Hacker News Reader

Comment by JuDue | Hacker News Reader

Fast is one thing.

BUT I'd much rather get rid of the "this link has expired" syndrome.

Fast is only impressive if the results are modern.

The superior results from my browser's back button far outweigh the inconvenience of the expired link. I click on a link from the "Ask" front page, reply to the top comment, and my comment appears in the thread. Next I click "back" on my browser and I'm back at the front page of "ask".

I do that much more often than I hit an expired link.

That doesn't really have much to do with the expired link issue. That issue exists because the targets of links are stored as closures on the server. What you want is to serialize those closures into the URL itself, instead of letting the URL be a pointer to a closure stored on the server. In traditional web applications, you'd have to do that serialization manually (although if you've never used such a closure based web framework you might not even be aware that you're doing this, just like a C programmer who has never used a high level language might not consciously realize that he is implementing objects or closures or garbage collection manually -- or like an old Fortran programmer who is not aware that he is implementing recursion manually). The problem used to be much worse, but then PG did this transformation manually for the important subset of the site (most importantly for links to the comments section -- these links no longer expire).

Here's an example. Currently if you go to the home page and then click the "More" link at the bottom to go to the next page, and wait long enough, the more link expires. That's because it's currently implemented as something like this (not sure about ARC syntax, I'll use Scheme syntax here, but you get the idea):

    (define (show-list-of-posts page-number)
      ... display the rest of the homepage ...
      (link "More" (lambda () (show-list-of-posts (+ 1 page-number)))))

This stores the closure `(lambda () (show-list-of-posts (+ 1 page-number)))` with the free variable `page-number` in a hash table on the server. The URL becomes an index into that hash table. But the only information needed to reconstruct such a closure, is the function body and the free variables. So if we defined an auxilary function:

    (define (foo page-number)
      (show-list-of-posts (+ 1 page-number)))

then we could represent the closure as the pair ("foo",page-number). If we encode that in the URL, then instead of looking up the closure from the hash table, we can reconstruct the closure on the fly. Hence we do no longer have to store anything on the server, and no links can expire anymore.

There are some challenges when you want to do this transformation automatically, but they can be overcome.

"The problem used to be much worse, but then PG did this transformation manually for the important subset of the site (most importantly for links to the comments section -- these links no longer expire)."

I don't think this is strictly true. The comments and reply links simply link to a normal (non-closure) URL on the site and the page is generated from that URL in the normal manner.

I don't understand the disagreement...isn't that the same thing I said? IIRC it used to be the case that all links could expire, then he did this transformation from closures stored on the server to closures manually serialized in the URLs (which results in a normal URL as you say). What I'm describing is how a web framework could do that automatically, which means you get the behavior and expressiveness of closures without the link expired issue. Though I can see that the explanation isn't very clear, but I don't know how to make it more clear. Edit: don't downvote him people, it is a perfectly fine comment.

> then he did this transformation from closures stored on the server to closures manually serialized in the URLs (which results in a normal URL as you say).

The reply links have a parameter 'id' - the id of the comment to which the reply is being posted. I would guess it's just passed to a normal function which adds the reply to the post.

May be you mean the same thing, but "serializing closures in the url"(where? all I see is the id of the post being replied to) isn't the same as params in the url which are then passed to a function.

Yes, my point is that it is the same thing. What's happening is just manual closure conversion (http://en.wikipedia.org/wiki/Lambda_lifting), except the resulting data gets saved in the URL rather than the server. On the server, closures saved in the hash table, also have "params which are then passed to a function", except that the params (i.e. the free variables) are saved on the server inside the closure rather than in the URL.

If you have a lambda expression like `(lambda () (show-list-of-posts (+ 1 page-number)))`, the Scheme implementation does these things:

1. Introduce a global function definition with the same body as the lambda expression but with an extra parameter for the free variables. The free variables in the body of the function get replaced with expressions that extract their value from the extra parameter that has the free variables.

2. Convert the lambda expression to an expression that builds a pair of a function pointer to that global function, plus the values of the free variables.

For example the code:

    (define (show-list-of-posts page-number)
       ... display the rest of the homepage ...
       (link "More" (lambda () (show-list-of-posts (+ 1 page-number))))

Will get converted to something like this:

    ;; this is the extra global function that has the body of the lambda expression
    ;; note that the reference to `page-number` got 
    ;; replaced by `(extract-value "page-number" params)`
    (define (closure-324 params)
       (show-list-of-posts (+ 1 (extract-value "page-number" params))))

    ;; note that the lambda expression gets replaced by a create-closure expression
    (define (show-list-of-posts page-number)
       ... display the rest of the homepage ...
       (link "More" (create-closure closure-324 "page-number" page-number)))

create-closure creates a closure data structure where the first argument is the function pointer, and the rest of the arguments are the free variables.

Now, what happens if you want to remove this closure business in a web application, and instead use normal URLs?

First, you introduce a global request handler for the body of the lambda:

    (define-handler (post-list-handler params)
       (show-list-of-posts (+ 1 (extract-value "page-number" params))))

This would define the handler for news.ycombinator.com/post-list-handler?page-number=12.

Then, instead of the (link ...) with a closure, you just link to that url in the show-list-of-posts function:

    (define (show-list-of-posts page-number)
       ... display the rest of the homepage ...
       (link "More" (create-url post-list-handler "page-number" page-number)))

Compare these code snippets to the one above. Do you see the similarity to closure conversion? In both cases we:

1. Introduce a global function/handler for the body of the lambda.

2. That function/handler gets a `params` argument that has the free variables.

3. Everywhere a free variable is referenced in the body, it gets replaced by an expression that extracts the value from the params argument.

4. In place of the lambda expression, we have respectively a (create-closure func free-vars...) or a (create-url handler free-vars...)

So it's really completely analogous. That's why I say that we are just serializing the closure here, and this could be done automatically. Hopefully this makes it more clear what I mean, but maybe these details just make it less clear if you're not familiar with how closures are implemented (closure conversion)...

What I would love would be if the "More ..." link at the bottom of any given page would link to Page N+1 of whatever the current ordering of pages is, rather than page N+1 of a listing of articles that is now out of date and gives me an error message.

Great explanation of what's causing that - thank you!

Still, why not link to a page offset and just show what is current? Or include it with the current linking and automatically redirect? Pretty basic ux optimization.

Because the functions which generate pages get garbage collected. Once the function is gone, there is no reference point.

I suspect hindering spam and vote manipulation plays a part in the architectural decisions. It also makes it possible to create different HN's for different users - e.g. the hellbanned, royals, and plebeians.

I know about hellbanned, but what are the royals and the plebians?

Theoretical particles required by my unified theory of HN.

Interested. Link?

Royals and plebeians might be a little strong. YC folks have a slightly different interface. I think the main differences are that YC usernames show up in orange and there is a link along the top that displays the most recent submissions from YC folks/alumni.

"Royals" came to mind from FlameWarriors - though maybe my predispostion to think of PG as the Philosopher King of HN played a role. Anyway, "Plebians" was just pushing the political analogy a little further. I'll take your word about the differences in interface. I really was just conjecturing.

http://redwing.hutman.net/~mreed/warriorshtm/royals.htm

The worst is when you write a big long comment, submit it, and get "this link has expired" -- then you hit back, and the textbox is empty.

This is particularly infuriating on a cellphone.

One quick fix: in the code that prints this message, check to see if the request was a comment-post action. If so, append, "...but for your convenience, here's the text you tried to post, so it's not lost forever: [...]"

(Just watch out for XSS!)

There are extensions that help with that.

Reply on news.ycombinator.com