Hacker News Reader

Comment by diggan | Hacker News Reader

> It's 118 gigabytes of JSON.

118.0 GB of ZST compressed JSON (https://zenodo.org/records/15170676). The actual uncompressed JSON would most likely be much, much larger.

View on HN

I downloaded it and decompressed it — it's approximately 2.1 terabytes in size.

hey there, would you mind sharing the file please?

I managed to download the file, but since they restricted the downloads, it seems they hid the checksums too. Could you possibly share the md5/sha256 of the .zst file?

  sha256sum dataset.zst 
  0196416253fab4bce08504737bc81215927d9afdc6ccc81f75345518109266a4  dataset.zst

magnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst

I imagine it can be reduced a fair bit simply by stripping out unneeded fields from the messages; I imagine it can be reduced even further by removing unneeded messages entirely (i.e., bot welcome messages), deduping messages, etc

Reply on news.ycombinator.com