Comment by codetrotter | Hacker News Reader

Comment by codetrotter | Hacker News Reader

Parent Full thread

From the page you linked:

> Check out the note for the first entry: Come on, everyone! Let's beat the dead horse one more time!

And indeed, it does say that at https://hn.algolia.com/?q=the+lisp+curse below the entry for https://news.ycombinator.com/item?id=2450973

But what's weird is it doesn't appear on the page https://news.ycombinator.com/item?id=2450973 itself nor anywhere in the page source that I can find.

But going back to https://hn.algolia.com/?q=the+lisp+curse, and looking at the source it says that it's the story comment.

So if I understand correctly, even though when you submit a story with a link it does not include the text if you write one, it seems that it's still stored on the HN server and returned from the HN API that Algolia gets its data from.

Just found this interesting.

And I wonder how many other stories have a "hidden text" like this.

If the Google BigQuery or what it's called dataset includes this same data it should be a simple thing to find all such stories.

I noticed that also. I've never seen it on anything other than that one search.

I gave it a go with Google BigQuery:

https://console.cloud.google.com/marketplace/details/y-combi...

First, number of stories where url begins with "http".

    SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE INSTR(url, "http") = 1;

Output:

Then, number of stories with non-zero length text:

    SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE CHAR_LENGTH(text) > 0;

Output:

And then, this should be the number of stories that has both.

    SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;

Output:

Seemed like an awfully high number though.

    SELECT * FROM `bigquery-public-data.hacker_news.stories` WHERE INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0 LIMIT 3;

Ah. I see then that there's a lot of spam posts. (Note: The order of the rows may be different from run to run I think, and I didn't feel like adding "order by" stuff. So it could be that you don't see the spam from issuing this one query above.)

Anyways, let's require score > 0.

    SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE score > 0 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;

Output:

Still really high. Let's limit ourselves to score > 25.

    SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE score > 25 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;

Output:

Ah, that's better.

And somewhere between? Say, with score > 8.

    SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE score > 8 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;

Output:

And let's look at some of them.

    SELECT * FROM `bigquery-public-data.hacker_news.stories` WHERE score > 8 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0 LIMIT 5;

Here's one of the ones I get:

Title: Calling the NSA: "I accidentally deleted an e-mail, can you help me recover it?".

Text: A nice thought and practically the same thing that Google does. You hand in your privacy, but you get services back in return.

Id: 6306219. Discussion: https://news.ycombinator.com/item?id=6306219

Seems pretty clear that this is an example of a comment to go along with the submission. And it was preserved, although not shown in the thread as we know.

And with Algolia we indeed get the story comment shown as well. https://hn.algolia.com/?q=%09Calling+the+NSA%3A+%22I+acciden...

Well, there you have it!

So this also makes me realize that third-party HN clients could use this to offer users the ability to write and to see story texts that don't display on HN itself due to there being an URL attached to the story, but which are still stored on the HN servers and returned by the HN API. That's pretty neat :)

Reply on news.ycombinator.com