Hacker News Reader

Text is very compressible. It can probably fit in a single consumer HDD (8TB or so) compressed with zstd.

It would be neat of there could be defined curated chunks of standardize text data sets for LLMs training distributed over bittorrent.

Maybe there already is and I just haven’t looked hard enough.

> The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.