Hacker News Reader

Comment by sxates | Hacker News Reader

I've got a synology NAS with at least 10TB spare capacity I could donate... If only there were a way to use it, and thousands more, to back up important projects like this in a decentralized way.

View on HN

IPFS seems perfect for the purpose. Files are content-addressed and a request for a part of a file can resolve to any accessible peer that has it. A peer has it if they have pinned it or if they have accessed it themselves recently so that it is cached on their node. This way, users share the load for popular files, donating both bandwidth and storage.

Torrents also seem like a perfect way to donate bandwidth and storage, and the IA already uses those.

The only problem with these approaches or generally any distributed file store is that someone will have to share a copy, so you'll mostly be donating bandwidth by storing a redundant copy of data that they likely have to store themselves as well to guarantee accessibility.

I keep thinking that the Bittorrent protocol would be an ideal fit for this; specifically, it would need a way to have all peers agree on what chunks of a file they themselves host and share to ensure the most complete copies available. And it would need a system for keeping it updated.

I wouldn't mind hosting a NAS in my closet and some bandwidth + storage to donate to the IA.

I also think cloud providers - all of them - have the capacity to host backups of the IA; I'd prefer if companies like Amazon, Google would donate their storage and bandwidth capacity to the IA instead of just money.

Bittorrent would require the totality of the archive to be preserved in multiple copies. I can't have a copy of that ledger here at home. We'd need a distributed highly redundant filesystem for that.

44 million items in the Internet Archive are available as torrents. It's pretty straightforward to enumerate them, download them, and serve them up in a swarm.

That doesn't make it sure the torrented data is always available. The index is important, but the data and its metadata are much more important.

Right, which is why you want to start getting the data served by seeders outside of the Internet Archive proper.

The Pirate Bay is all magnet links. A model to be considered.

That still doesn't ensure all the data is available all the time.

A solution should be one where whoever hosts the data can't pick which pieces they want to host. Otherwise content that's not popular will disappear.

git-annex is much better for that situation.

https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

and

http://iabak.archiveteam.org/

I'd go out and buy a NAS and fill it up with backups from IA if it would help the collection survive. Point me at the project.

Reply on news.ycombinator.com