The biggest theft in history with governments as complices
But no, I don’t think governments are accomplices. The smartest move would be for China to wait and see how this plays out. It could significantly shift the epicenter of AI work to China. The single biggest limiting factor for open source LLMs is data, and no one in America or Europe wants to be sued.
High-speed access available for anyone who can do at-scale text extraction, or who can supply us with new collections.
Please focus on your opsec. The more visible you become, the progressively angrier people will get. Don’t do anything silly like edit your Wikipedia page from your house.
With that out of the way, someone I know happens to have the original books3 epub files. I think they can be convinced to send them to you. It’s only 200,000 books, but that could theoretically grow your collection by 10% or so. I don’t know whether that would be helpful to you (you’ve far surpassed books3 at this point), but if so, let me know.
Given the legal risks, the best course of action for AI companies is probably to ignore English and European books entirely. There is plenty of Chinese data, and the models would learn all the same concepts without exposing anyone to lawsuits.
Basically, you download a client, say "allocate 2 TB of my disks to whatever archive.org/donate/disk.rss" says, and the server/client combination ensures you download and seed the rarest 2TB of the collection.
This design is also open, in the sense that the server can share the database of torrents it contains, and anyone can use it to fetch any of the files in the dataset from the swarm.
Would something like this be at all useful? I've emailed a few archivists, but I got no response, and the one person I've managed to talk to about this said there have been a few attempts on this, but they always fail for one reason or another.
The hard part is that those who donate their space have authority over that space. It’s the Byzantine fault tolerance problem: imagine if 4chan donated their space, then started serving CSAM instead of the expected data. You can use hashes to verify integrity, but then the question becomes who gets to decide which hashes are ok. And hashing makes it impossible to edit large files, which is a frequent occurrence in LLM work. You’re constantly tweaking your datasets and spitting out new blobs.
Direct answer: yes, you’re doing good work, and you should keep doing it. I would personally use this for storing books3 transformations.
The other hard part is that you’ll want at least some redundancy — see 6.824 distributed systems, or the GFS paper. It’s why I’ve been implementing Raft and toying with some kind of distributed consensus without a blockchain. (Such consensus is still possible if the researchers were granted authority over what can be stored — which is the whole reason people are donating their disk space in the first place.)
Another issue is sudden bandwidth loss. Data storage is one part of it. The other half is rapid transfer. By replicating the data, you can pull it from multiple replicas at once (I.e. there are more seeders). This also protects against someone suddenly getting throttled, or just having a power outage. The protocol should prioritize donors with high bandwidth over vast storage space.
Feel free to DM me on Twitter if you’d like to toss around some design ideas more seriously, and thank you for trying to build this.
Edit: Looks like Twitter wants me to pay to DM, I'll email you instead.
I responded with some telegram info if that helps.
If you're doing this in bittorrent then you might want a client that's configured to optimize for a different goal than most torrent clients.
Potential goals, somewhat conflicting:
A) keep data with a low mirroring degree available. either this needs to be centrally coordinated or some sort of randomized algorithm where clients pick underseeded torrents but not everyone picks the same
B) bandwidth matching. to not consume more resources than are provided a client maybe should only download 1 piece of data for every N times it uploaded any piece. This is much less greedy than what you'd have in a normal torrent client but ensures that caches themselves don't take up much bandwidth compared to users who actually want to download something. Otherwise a misconfigured cache (e.g. behind NAT) could accidentally always download data without ever giving much back.
One could easily use IPFS as the storage layer and add $magic_sauce to manage the distribution of the books within the sub-network kind of like git-annex does to manage people's porn collections. There's one project I saw that does this (using the IPFS-daemon's RPC API) to run a cluster of IPFS nodes for whatever reasons people would want to do something like this.
Short summary of The Right to Read, and evidently the world you want.