Hacker News Reader

Comment by remram | Hacker News Reader

Is there a dataset of HN titles? This made me want to fiddle with this, but step one is to get the data, and I don't want to crawl HN if the data has already been collected.

View on HN

There are a few sources. There's the official API [0], the Algolia search API [1], and the BigQuery dataset which is pretty up to date [2].

I used the Algolia search API, it has extremely generous rate limits and page limits.

[0]: https://github.com/HackerNews/API [1]: https://hn.algolia.com/api [2]: https://console.cloud.google.com/bigquery?p=bigquery-public-...

There's an API[0] but it's frustratingly limited in capabilities (albeit not rate-limited.) You'll have to iterate all post IDs, download each post as JSON and get the titles that way.

There's also a Google dataset but I don't know the URL for it or if it's up to date.

[0]https://github.com/HackerNews/API

> frustratingly limited in capabilities

What's missing precisely? Seems to be good enough for every use I could think of.

One time I even downloaded every single item from it, with a threaded fetcher of I think 16 threads, iterating from 1 up to latest ID and it was done in some like 2 hours I think.

No ability to directly download threads with a single request, for one, or query it like a database to sort or filter results, exclude unwanted fields, etc.

Those things should be trivial to achieve with most general purpose languages, as the API is so simple. No need for pagination or other things, just request things by ID recursively and you get the full thing, then after than filter/select whatever you want.

Pseudo-code to show how simple it would be:

    function get_thread(id) {
      let item = http.get(`{api}/?id={id}`).body
      if item.childs {
        item.fetched_children = item.childs.map((id) => {
          return get_thread(id)
        })
      }
      return item
    }

(untested, but you don't really need more than that, besides checking if the item was deleted)

Yes, and i've done it. But you still wind up having to make a separate request for each item, which makes building threads incredibly slow. It's also a waste of time if you're filtering out anything, because you still have to make the request and download the item to filter it out.

Which is why it would be preferable for the API itself to support these features.

Can you somehow fine-tune GPT3 on a dataset? I just assumed the OP generated them using a prompt like "top hacker news threads" or somehing like that.

Reply on news.ycombinator.com