Hacker News Reader

Comment by fdsary | Hacker News Reader

Does archive.org let you download dumps of all of their data? If they make a weekly torrent containing their whole data-set it would be easily immortalised...

View on HN

They hit 10 petabytes two years ago* - which is 10000 terabytes. Torrents don't really cut it at that scale.

* http://blog.archive.org/2012/10/26/10000000000000000-bytes-a...

> They hit 10 petabytes two years ago* - which is 10000 terabytes. Torrents don't really cut it at that scale.

I see no reason why torrents couldn't handle this. Data transport through bittorrent still works even with large numbers of bytes.

At these scales, bandwidth to disks isn't trivial - at a gigabyte a second, it would still take over 130 days to build the checksums for the torrent. I don't think Bittorrent fundamentally allows multithreaded torrent generation.

I think they bundle their data into WARCs of a few (dozen?) megabytes. Copying those around is much more straightforward, with much lower latency.

I think he could have meant that some DHT could have been used, it needn't be specifically torrents. You could then volunteer your machine to redundantly store, say, 5GB of the archive.

If you are willing to spend time copying files around you should donate: your pay-per-hour probably far exceeds the time it would take you to dilly dally with billions of WARCs. Even if you flip burgers for a living. There are far better ways to contribute than by typing 'cp'.

You could use floppies to transfer terabytes of data, but that doesn't make it a good idea.

A good portion of the images/video are spam - take that into account.

Reply on news.ycombinator.com