Perhaps not the best trade, than again, the choice might not be necessary at all: xz is rather slow on decompression. What were to happen if the format was .zstd instead of .zip (with/without .tar)? We should be able to achieve a lower size than .zip without being so slow.
I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially. At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd.
The start of the files had a header with pointers to the manifest.
The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary.
Edit: If there's some interest, I can try to convert the dataset to this format.
The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.
Outline of the technique is here https://news.ycombinator.com/item?id=28013347
People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone".
> Yes, there's a lot better compression available these days
Tell me more!
Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
* Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.