Hacker News Reader

Comment by DaiPlusPlus | Hacker News Reader

Does that 1.5m figure include redundancy (e.g. RAID)? What about if the data is compressed (either trivial gzipping each "file" the archive has, or using a compression window that spans multiple files)? I imagine the HTML/plaintext content of the Internet Archive would compress very well.

View on HN

No, that's for a single raw copy (it could go lower based on implementation -- the sweet spot on $/bit pricing is around 8TB/disk right now, but that would actually be more expensive for us in total cost because of the increased infra/space/power necessary to run them). We could probably get a relatively trivial 20-30% savings on space for Wayback contents via compression, maybe more with work (various projects are underway to do this), but much of the rest of the contents are difficult to compress, or already compressed (music, imagery for books, video, software archives, etc). We have also historically been very reluctant to deduplicate heavily, though we are experimenting with it for certain types of content -- one principle of operation is that as an archive of last resort, we're unable to have a true deaccession plan as some other archives have. A compromise we make is that our hard drives are "landfill-ready" -- that is, the contents of a drive (assuming you can read the filesystem) are inherently meaningful, content is housed with its metadata, and so forth. This produces some unusual restrictions on how far we can take compression and certain types of bitwise redundancy.

By unique data, that excluding generated data? Can you please estimate the space for just the wayback machine? It's the actual target.

What's the wayback machine with and without images? Is it possible that we could distribute the ASCII/Unicode content now?

It's the actual target.

It may be your target, but I would not be so dismissive of the other data in the Archive. The Software Library, the tens of millions of scanned books, the music, etc etc. On top of that, the raw scrape data driving the Wayback Machine is not currently made available to download from the archive. It's stored in WARC files, which would include both the images and text of all scrapes and would not be trivial to disentangle.

I'm not dismissing any other section of the archive. The the wayback machine has the big red target on it. It's a snapshot of recent history, I have lost track of how many times I have needed it for things that would otherwise be memoryholed.

Please, consider making a subscription service for the warc files, let us pay to get access to a query interface. archive.org could raise significant defense funds.

Reply on news.ycombinator.com