Libgen-text: complete Library Genesis, re-packaged as text only (1.2TB)
offlineos.com
offlineos.com
However, it should be of great use for indexing and searching in them!
[0]: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...
1) I assume 1TB (not TiB) of uncompressed (?) text
2) I assume one character is one byte
3) I assume 5 (actually it seems to be 4.7 in English) characters per word
So 1TB/1B/5 = 1.0E12/5 ~= 2.0E11 = 0.2T = 200B words.
Your article mentioned that Chinchilla is trained on 1.4T tokens. So there is quite some difference.
The article also mentions different mysterious book data sets with 27B tokens, 560B tokens, or 390B tokens.
The latter datasets were made by Google. So you are still behind Google massive book dataset even if you use probably the largest book dataset "available" to people or instituions outside of Google.
EDIT: I thought I made a mistake, but T stands for trillion or tera which are both 1E12.
Perhaps not the best trade, than again, the choice might not be necessary at all: xz is rather slow on decompression. What were to happen if the format was .zstd instead of .zip (with/without .tar)? We should be able to achieve a lower size than .zip without being so slow.
* Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.
People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone".
> Yes, there's a lot better compression available these days
Tell me more!
Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.
I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially. At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd.
The start of the files had a header with pointers to the manifest.
The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary.
Edit: If there's some interest, I can try to convert the dataset to this format.
Outline of the technique is here https://news.ycombinator.com/item?id=28013347
One is that the books are organized into thousands of zip files, while I'd personally rather they just be organized into directories. I get that most people will benefit from the compression, but I suspect that relying on btrfs's transparent zstd compression will do just fine for me, so I'd rather have a simpler access pattern that doesn't involve extracting or scanning through a zip file whenever I want to look for a book. Not a huge deal. I'll just extract them all I guess, just feels like a waste of IO.
Another thing, why txt? I'd much rather this be a collection of epubs instead, so normal ebook readers can open them with proper formatting, and they'd already be similarly compressed. There'd certainly be some increase in overall size, but I think it'd be worth it.
Another problem is that the index files don't seem to be clean? There are references to books that don't exist, apparently because if whatever automated tool was used to create this couldn't extract the text from a book, the book simply isn't present but it still has an ID and an entry in the index files. Another thing that isn't a huge deal and that I can fix locally, but is a bit of an annoyance.
Once I have this all set up in a way I want, I expect to just be able to serve it with a simple web server and a url rewrite rule instead of the software shipped as part of this project. Nothing against that software, but I'd rather just use a filesystem to organize text files. I guess I'll have plenty of time to think about how best to do it, because the current download ETA is over a week.
About the missing books fiction and non fiction
> if the book is missing (15%), that means the text could not be extracted.
> if the book is missing (10%), that means the text could not be extracted.
===
| why plain text?
===
- text is smaller (figures are for old .tar.xz, compressed .zip is about double the size)
+----------+------------------------------------+------------------------+
| | size | number of files |
+----------+----------+-------+-----------------+-----------+------------+
| data set | original | text | compressed text | original | text |
+----------+----------+-------+-----------------+-----------+------------+
| lg | 32TB | 0.8TB | 0.32TB | 2,450,000 | 2,220,000 |
+----------+----------+-------+-----------------+-----------+------------+
| ff | 2.4TB | 0.5TB | 0.18TB | 2,180,000 | 1,890,000 |
+----------+----------+-------+-----------------+-----------+------------+
- text can be viewed on every device.
- plain text cannot contain malicious code. pdfs can.
- text can be searched and indexed more easily.
- why csv? anyone can use csv. it can be read raw in a text editor, opened in excel, or imported into a database. database dumps can be used only by experts.> if the book is missing (15%), that means the text could not be extracted.
I saw that, I just don't understand why a book that isn't included has an entry in the CSV file. If the automated tool fails to extract the text from the book to include in the archive, then why put it in the index file?
So now you have a list of stuff that should have been included but isn't? Then maybe chase down why and fix?
I'm downloading the torrent right now and it's 1.2TB of mostly zip files each containing many txt files. So the 1.2TB does appear to be the compressed size (unless the zip files aren't actually compressed, but that would be weird)