Hacker News Reader

Comment by pavel_lishin | Hacker News Reader

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676

It's 118 gigabytes of JSON.

> It's 118 gigabytes of JSON.

118.0 GB of ZST compressed JSON (https://zenodo.org/records/15170676). The actual uncompressed JSON would most likely be much, much larger.

I imagine it can be reduced a fair bit simply by stripping out unneeded fields from the messages; I imagine it can be reduced even further by removing unneeded messages entirely (i.e., bot welcome messages), deduping messages, etc

I downloaded it and decompressed it — it's approximately 2.1 terabytes in size.

hey there, would you mind sharing the file please?

I managed to download the file, but since they restricted the downloads, it seems they hid the checksums too. Could you possibly share the md5/sha256 of the .zst file?

  sha256sum dataset.zst 
  0196416253fab4bce08504737bc81215927d9afdc6ccc81f75345518109266a4  dataset.zst

magnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst

Here's the top 500 servers in the dataset: https://pastebin.com/pANc3WkT

Anyone who has it, please post the SHA384 or BLAKE3 (BLAKE3 will be way faster on 100+ GB) so I can verify it if I get a torrent later. Zenodo requires a sign-in, and I won't auth with GitHub as they want some WILD permissions on my GitHub.

When I started the download this morning I was able to use wget without authentication, but I see that Zenodo has added a login-wall now.

The decompressed data appears to be JSONL, but at least the version I downloaded has a little binary garbage at the front. The first readable JSON object has the author "Fortnite Germany".

Size as .zst: 117,962,356,699 bytes

ZST SHA384: b8863645654610f1fde2859bb20bd87d913865af7791e0ec33741402944d5b9bdfdaaf65c2dc610730efb01f446e2588

It is not just a login wall, they have restricted access even for logged in users, presumably to only the uploaders. A magnet would be nice.

About the decompressed data:

zstdcat dataset.zst | sha384sum

0812f3876a7e319081f596a5545321e5c8e8def501add3a4f5ff039568fe59aa5d4ac5d2c3e549532f529bd09b887596

zstdcat dataset.zst | wc

2059116741 22128178392 2099550453760

If you can post your email and you have a sftp server or other accessible means to receive this large file, I'll contact you and then maybe you can help distribute it more widely.

(Offer also applies to anyone else reading this thread.)

Hi, if able to, contact me at <ruooens AT protonmail DOT com>

Hi, could you also send an info to below? Have webDAV from my end. <iliiilili AT protonmail dot com>

hi there, would you mind sharing with me please: kedaiapps AT gmail DOT com

hi there, would you mind sharing with me please: <kedaiapps AT gmail DOT com>

Hi Philip. Would you be able to share with me just a few anonymized user IDs (random ones are fine)? I wanted to double-check their format, since I'm pretty sure their method is broken. My email address is there → https://gynvael.coldwind.pl/?id=50 (contact section)

Received, thanks!

magnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst

This is a pet peeve of mine. Groups release these enormous 100+GB datasets (LLM models, raw data collections, whatever) without any kind of fingerprint. Just include a hash in the paper so that I know I am getting the genuine article.

Is there a faster way to download than the link there? It's steady, but a roughly 11 hour download.

hmm they should have uploaded a torrent link to download the data, I wonder why they didn't do that?

I'm sure someone will make a magnet for it eventually

magnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst

Site says the file is restricted for me. Dang.

Reply on news.ycombinator.com