Hacker News Reader

Comment by fermienrico | Hacker News Reader

I've always wondered this - How does Archive.org work in terms of storage? Internet is massive and caching every single site periodically for years on, isn't that unreasonably huge amount of data?

Edit: I just checked Wikipedia, it says they're using about 15 PB of storage.

Edit 2: 15 PB cost => 15,000 TB x $30/TB = $450,000. Ofcourse, back of the napkin cost (no maintenance, power, etc). That's not too bad actually.

View on HN

The Archive currently has about 46 Petabytes of content ("bytes archived"), and over 120 PB of raw disk capacity; the difference is due to data replication, "currently filling" storage, non-storage infrastructure, etc.

We save a lot on web content storage by de-duplicating "revists" when the page hasn't changed. This works out to save a whole lot for content like jQuery served from a common CDN URL; it doesn't work well when there is a page counter or any trivial changing content on a page.

If you are interested in the storage back-end, it's actually pretty simple: HTTP requests/responses are concatenated and compressed in WARC files (sort of like .tar.gz) that get stored on regular old ext4 filesystems. An index of "what URL captures are in what WARC files on what servers" is continuously generated in the form of, basically, a giant sorted (and shareded) .tsv file; replay requests on web.archive.org look up the URL and timestamp and get a reference to a machine, file, and file offset, and make an HTTP 1.1 range request for the content in question. There are a bunch of other details, like checking robots.txt status, but the core design is super simple, cheap, and (relatively) easy to operate at scale.

Apart from web crawl content (including, these days, "heavy" video content which is difficult to de-dupe), we have a large amount of live recorded TV, scanned books (raw photos), etc.

(I currently work at IA)

Have you evaluated compression algorithms that support custom dictionaries, like zstd? You could generate a compression dictionary for each domain, or just for those above a certain size.

Would be nice if you could store a diff, so a changing counter would only have to store the changed counter after the initial save.

Wondering the same. How if used git protocol itself? Not sure how efficient is git, but if it is then it's a relatively easier change

Git doesn't store diffs, it stores whole files.

No, git packs do store base blobs and diffs, which are not visible to end users; the git plumbing hides this and only presents blobs to the user.

You can do this if you group all site visits into one common WARC and compress it (or dedup it otherwise).

WARC itself does have a method of dedup if the response is the same (or mostly the same) but terrible if content changes.

How do you combat bit-rot/silent corruption?

> (including, these days, "heavy" video content which is difficult to de-dupe)

Still waiting for the Shazam for video to know that 2 videos are the same even when they are of different codecs/framesizes/etc; just based on the visual imagery.

Isn't that Youtube Content ID? Upload a video and see who sends you a takedown

120PB is still an insane amount of data.

I imagine at least a million dollars an year to keep the lights on just for the infrastructure.

Does IA have a big endowment to keep it going for a while?

A lot more than 15PB; it was over 20PB last year. And that figure doesn't count the duplicate copies they keep of everything.

According to their website, it's over 35PB now.

"We preserve 750 million Web pages per week! We’ve saved 35 petabytes (that’s 35,000,000,000,000,000 bytes) of data."

According to this 9-28-2018 article,

"Today, the Wayback Machine houses some 388B web pages, and its parent ... The Internet Archive’s collection, which spans not just the web, but books, audio 78rpm records, videos, images, and software, amounts to more than 40 petabytes, or 40 million gigabytes, of data. The Wayback Machine makes up about 63% of that."

https://thehustle.co/inside-wayback-machine-internet-archive

It's essentially the only cure for the serious link-rot problems in Wikipedia's references.

That doesn't account for redundancy and backups. I have asked some storage cost questions on /r/sysadmin and /r/datahoarder in the past about cost per TB and I tend to see numbers ranging from $300 to $1000 a TB of useable space.

That must be for SSDs, which are ~$150/tb vs ~$20/tb for HDDs. I expect IA must use HDDs for the bulk of their data.

Or even tape. For offline redundancy it’s still cheaper.

SSDs are about $400/TB for raw storage. Prices are substantially different for consumer and enterprise spaces, but 4TB disks (Micron, Samsung, etc) tend to be about $1500 and 8TB around $3200. If you factor in redundancy, you're looking at around $1000-1500 per TB of usable storage.

This also isn't a matter of "just go with cheaper consumer disks". If you care at all about data integrity you won't use consumer-grade SSDs.

3-way mirrored spinning storage would cost roughly $100/TB. 12TB disks are about $350/ piece these days. There are a whole bunch of possible considerations and alternate scenarios, but this is probably the worst-case.

That does not factor in compression / dedup, which can gain you a substantial amount of savings depending on your work profile; I bet if Archive.org wanted to, they could slash their storage using block-level dedup. Of course that creates potential recoverability headaches, so it's possible they don't.

If the internet grows exponentially, and sites decay in finite time, then that's about the same as caching the current internet, i.e. Google.

Blockchain

I see this is downvoted. Actually, if a blockchain used proof of actual useful storage or computation, that'd be great. (instead of wasting energy by computing "useless" hashes)

It's probably downvoted because replying 'blockchain' to a question about how Archive.org stores their data is not even a useful answer.

There may well be fascinating ways one can use blockchain for archives of this sort, but the grandparent comment was essentially useless noise.

I suppose that would be kind of interesting. Although if it's conceivable that someone could just store all the useful information then you can't just request a random piece of information as a proof of 'work', so you'd need something clever.

Anyway the comment was probably downvoted because it consisted of a single buzzword.

Proof of storage is used in siacoin, but not to secure the network. I haven't seen a scheme where proof of storage can be used to secure the network.

In terms of making sure the work that is done is useful it's fairly hard to do that because if you let someone dictate what work should be done then you potentially allow them to do the work upfront and a 51% attack becomes pretty easy.

"you potentially allow them to do the work upfront and a 51% attack becomes pretty easy."

I do wonder if there are potential consensus algorithms that rely upon game theory, the threat of being booted off the network, and the need for some investment before they're "trusted" by other peers. The basic idea is that yes, you could 51%-attack the network with a lot less than 51% computing power, but it would never be profitable because if you're discovered you're booted off into your own little partition where nobody plays with you, and the value you get from being part of the network in the future is greater than the value you would get from rewriting history into something only you believe.

51% attacks are possible now - roughly 75% of Bitcoin hashing power is located in China, so if the Chinese government decides that they want to shut it down, they send the military after the largest Bitcoin miners, say "Hash these transactions or else", and take control of the chain. They haven't found it profitable to do so yet, though - what would they gain from it?

The thing is that reputation isn't built into the protocol. Everyone is basically anonymous strangers.

When space-intensive computations are used for consensus, it's usually called proof of space. There are a few projects doing it already.

https://en.wikipedia.org/wiki/Proof-of-space

Proof of space and proof of storage are two different things.

Blockchain actually could be used to provide data integrity in a distributed environment with known bad actors (presuming that they fall under the 50% threshold), but that is not applicable in this case since archive data is not stored in a distributive fashion by clients (as far as I know). Having worked on a production application that uses blockchain, my take away was it cleverly provides distributed ledger transparency, but at a price that makes it very inefficient as a data store.

Reply on news.ycombinator.com