Hacker News Reader

Comment by stavros | Hacker News Reader

I wanted to write a piece of FOSS that would allow dissemination of a large dataset. It would basically work kind of like a torrent swarm, you'd have a "tracker" (the Internet Archive) deciding what it wants each person to store (usually the rarest content would get stored preferentially), and the user could say "I want to donate 2 TB of space to the Internet Archive" and would download whatever files the IA thought were most at risk of being lost.

This would have the added benefit that, if the IA went down, the public could reconstruct (some of) the dataset from this swarm.

I spoke to a few archiving organizations about whether they'd find it useful, but there wasn't much interest. Too bad, I think a lot of people would like to donate some disk space right about now.

View on HN

You could also do some kind of distributed proof that they still have it: ask every host that claims to have a file compute the hash(prefix+content) of the file, where prefixes differ: distributed k prefixes to each client, and then compare them to get a proof that at least some number of copies exist.

It's an improvement over regular BitTorrent because it sounds like there's some coodination about how to spread the data across contributing nodes. BitTorrent doesn't do that out of the box.

Maybe I'm just being slow, but how is that different from a normal bittorrent (with DHT)?

Are you asking for coordination of which parts are most rare? I though bittorrent already prioritized the least common parts in a swarm.

It has quite a few differences, namely it's not one swarm, each file (or collection) is its own torrent. This allows for easy updates (clients can get new files without having to re-download everything), it has protection against some attacks, etc.

That is supported in the bittorrent standard: https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0039.html

Not sure what the client support looks like.

This is interesting, thank you!

IPFS is meant for this, you could also use torrents with RSS (a lot of clients support downloading torrents from a feed, but there's no signaling for replacement/removal).

IPFS doesn't do most of what I described, though.

It doesn't track how much each file is being stored or pick what's most at risk. That seems particularly difficult, even impossible in the presence of bad actors.

Unless you make a system where the clients don't know what they are storing nor who else does too. That would be novel I think.

Whatever you are describing I want to run a node

Yeah, but the Internet Archive would have to run the server for it, and I don't know if they want to :(

Reply on news.ycombinator.com