Hacker News Reader

Comment by IanCal | Hacker News Reader

> Distribute your public data globally for free to ensure it is available forever!

What steps are in place to ensure this over reasonable timescales (20-50 years)?

The project is run by the U.S. 501(c)3 Non-profit called Institute for Reproducible Research (http://reproducibilityinstitute.org) and this site has an overhead cost of ~$500/year. We plan to fund this project for at least the next 30 years. The community hosts the data and we also coordinate donations of hosting from our sponsors (listed on the home page).

We also run the project ShortScience.org! Check it out!

> The community hosts the data

This is the key thing for me, so there are no guarantees of the data being available? Or is all the data backed by an owner hosted box (and backups)?

The main goal is to facilitate distribution. We work with research groups and provide "best effort" hosting with the resources from our sponsors. We expect the uploader has a backup of the data in all cases.

What would the Millennium Clock[0] version of a data storage device look like?

[0]http://longnow.org/essays/millennium-clock/

It wouldn't be a single device. That's far too subject to attacks. Instead, let's picture a decentralized system like BitTorrent, but with the goal of data resiliency.

I picture a system where a person adds themselves as a node to a certain group of files- like BitTorrent- but instead of downloading everything, they choose how much space and bandwidth they're offering and the system grabs the "best" pieces. Best in this case is about spreading the data out to improve resiliency, like any dates storage service would. It's not a human choice but an optimization problem.

If 250 people each offered 100MB of space, a 5GB file could be maintained with quadruple redundancy. Nodes come and go, and the system would minimize data transfers while aiming to maximize redundancy. Try to spread the files far and wide geographically, to put popular pieces of the data onto nodes offering more bandwidth.

Hmm. This sounds like a fun project to try...

If you're going to build something like this, especially when the data sets are typically large, this is a very strong case for erasure coding.

If you split a 5GB file into (20) 250MB chunks and then mirror each of them four times, there is a nontrivial probability that one of the chunks loses all four mirrors. Especially when you're using notoriously unreliable volunteer hosting.

If you split the same 5GB file into (20) 250MB data chunks and (60) 250MB erasure chunks, you consume the same amount of total storage but have to lose 61 chunks (>75% of all the hosts) before you lose any data.

Offtopic: Have you looked at PAR2? I seem to recall it being really good but having some critical bug.

And here I go on a wiki spiral. Thanks!

That sounds a lot like IA.BAK [1], a git-annex backed attempt at making a backup copy of the Internet Archive.

[1] https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

Edit: https://git-annex.branchable.com/design/iabackup/ has more details.

It would be a collection of lithographically etched plates. These are stable across a wide collection of conditions. The coding scheme would be reed solomon or a variant of it, or possible even just massive duplication with checksums. Multiple duplicate backups of the entire data would be stored in geographically disparate locations. This reduces the "asteroid hits <city>" problem. A wide range of different technologies would be used, randomly, across the copies. This reduces the likelihood of systematic catastrophic failure due to a specific technology not working as expected.

This is not that different from how current millenia-scale data storage has worked for... well, millenia. The primary challenge is in finding a universal key (if somebody comes along a million years later, will they share enough context to decode the archive?)

From the same people: http://rosettaproject.org/disk/concept/

It doesn't really scale, but then neither does the Clock.

Reply on news.ycombinator.com