HN-Index | Hacker News Reader

HN-Index

alexmolas.com

40 pointsalexmolas15 comments

Note I did a scrape four years ago which I saved on archive.org [0] and provided some pedestrian analysis on it [1], along with source code [2].

It looks like a sqlite3 database of all items, including comments and stories, was about 5Gb, so I would expect it to be within the range of 10Gb or so now.

I think h-index is an interesting idea but the post only really goes into it for story submission. The author scoffs at doing h-index for everyone but I don't think it would really be that bad.

What does an h-index for comments look like? A commentor with h comments that have at least h karma each? Or h replies? Or should it be h comments with `alpha *` h karma, for some value of alpha?

It's amazing to me how much progress has happened in 4 years. We now have LLMs. It'd be interesting to try and use the HN data for that type of purpose. Even using it to pick out links for people with high karma (or high h-index comment) could help with blog discovery.

[0] https://archive.org/details/HackerNews-Data-2020-06-28

[1] https://abetusk.github.io/yahnda/

[2] https://github.com/abetusk/yahnda

The author says he's looking for blogs to follow. So why compute this score based on submitted stories only?

People most often submit stories not written by themselves, so it does not really reflect what may be found on their blog (if they have any).

If they have a blog, then my guess would be that their blog posts are likely more similar to their comments than their submitted stories, since it's written by them for sure in both cases.

reads blog and sees own name

I wonder if you’d see a bimodal distribution of some sort for posters and commenters. It feels about right that you’ll have people who post a lot and people who comment a lot and then mixes of the two modes.

Or is it that people who post a lot also tend to comment a lot too?

If people have missed we collected less than a year ago personal blogs of HN users https://github.com/outcoldman/hackernews-personal-blogs

The open Google Bigquery table of Hacker News might possibly be a nice way to get this data. (Although last time I checked it was a few years out of date, so maybe not... Hoping that someone reads this and starts updating it again...)

There is a constantly updated table in ClickHouse with no login required for querying: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNvdW50K...

Nice, here's an attempt to compute the HN-index for all users. (Differs from blog post; not sure if because the data from the two sources is different, or there's a mistake somewhere):

    WITH Ranking AS (
        SELECT 
            by,
            score,
            RANK () OVER (PARTITION BY by ORDER BY score DESC) AS rank
        FROM hackernews_history
    )
    SELECT by, MIN(rank - 1) as h_index
    FROM Ranking
    WHERE rank > score
    GROUP BY by
    ORDER BY h_index DESC
    LIMIT 100

Does that still work? I'm getting "Code: 60. DB::Exception: Table default.hackernews_history does not exist."

The table name has been changed to "hackernews", although it also hasn't been updated for 7 months.

It appears to be working for me:

    SELECT count(), min(time), max(time) FROM hackernews_history

39838828, 1970-01-01 00:00:00, 2024-03-11 13:04:47

I created this table on the European server, but forgot to enable replication to the US.

Wow. Awesome resource!

if OP is reading this, HN also exposes an Algolia search API. Perhaps they could investigate if it would give them an easier time with their analysis.

Probably, as it returns directly a list of all stories (without comments) and their vote count, and then quickly paginate it, way fewer requests than using the firebase API

For example for the user alexmolas, filtered to only show stories and show 50 items per page (the maximum):

https://hn.algolia.com/api/v1/search?tags=story,author_alexm...

Also adding &numericFilters=points>1 filters out all posts that don't have any votes (all posts have at least one point).

Because there is almost no cost to submitting a story, either on HN or for the calculation of an h-index it doesn't work.

Comments are more interesting, but the data is hidden.

It's interesting to look at for insights.

A low h-index among top users is an imperfect way to see who adds the most value. It works in the opposite way.

Reply on news.ycombinator.com