Comment by Spooky23 | Hacker News Reader

Comment by Spooky23 | Hacker News Reader

Parent Full thread

The use-case that Glacier is intended for is long-term archival preservation. When archivists say "long-term", they are thinking decades at a minimum.

When you're thinking like that, you need to ask yourself what 1TB actually means. Your USB disk the the equivalent of a VHS tape in 1980... it holds a few hours of video, but to keep the quality of the original, you need to maintain it. Will a USB drive in a safe deposit box work in 20 years? Will you have a USB3 interface handy in 20 years?

Digital media is really tough to archive over long periods of time vs. paper. I worked with a US State archivist on a project, and they stuggle with the issue. They have a good handle on the papers of the colonial government from the 1600's, but have a difficult time accessing the early digital records from the 1980's.

So putting stuff in glacier, and letting Amazon handle storage migration, break/fix, replication, etc for $120 per TB/year sounds like a value to me.

If we assume that Amazon's service survives the next twenty years, I still don't see how we can reasonably expect to be able to read the files we stored there. File formats become obsolete all the time. If you don't maintain your data it is quite likely that it is useless independent of where you store it.

That is a problem, but it's a problem that you have with "roll your own infrastructure" as well. Digital data isn't like paper -- archiving doesn't mean keeping tapes in a vault. You need to periodically refresh and check the integrity of the media.

Amazon's service gives you an expectation that 99.999999999% of your data will survive intact on average every year. So while you need to maintain your data logically, you can set clear expectations with respect to the integrity of that data.

The fact that Amazon's infrastructure may not be around in 20 years isn't especially relevant either -- neither will any maintained infrastructure. Tapes need to be refreshed, archival storage infrastructure usually ages out in a 8-12 years, and you need to do a TCO analysis each time to refresh.

The other thing is that you get physical isolation. Flooding hasn't been a problem in lower Manhattan or Staten Island in recent memory. How many safe deposit boxes do you think were destroyed in those two places recently?

I know Amazon says they have 11 9s, but merely given that there have been 5 mass extinction events in the past 500M years, you probably can't do better than 8 9s. (11 9s gives you even odds that your data will last 70 billion years, as long as you pay Amazon's fee.)

You're probably right that Amazon holds your data for as long as that, however, there's another way to look at 11 nines: If you upload one hundred billion (10^11) files on Glacier, you expect to lose one of them per year, and not more. That's probably realistically achievable.

Compare this with S3, where you will lose 1 in one billion files per year. Shocking!

While physical media formats and hardware interfaces become obsolete all the time, it's something of a myth that file formats do the same. Even proprietary formats can usually be read by digging up a copy of the software that created them (in theory this software may be lost, but typically it is not). The one major exception is DRM'd files where the keys have been lost.

This is truer today than it once was. Go back, say, 15 years and you'll find any number of word processor and presentation formats that are only readable by long-gone software. I agree that, today, we've (mostly) coalesced around a modest number of standards that are either open or, at least, widely used and well-documented.

I'm not sure what you mean by file formats becoming obsolete all the time. This may be true for some proprietary formats that are no longer readable by a modern application, but for any reasonably widespread format there should not be much difficultly. I don't see any reason why we wouldn't be able to read most of our text, image, and video data decades down the road.

Remember, we're supposedly entering the "post-PC" world. How do you intend to open a WordStar document or Quattro in 2040, after 30 years of app stores and sandboxed apps?

The presentation of documents are critical as well. Will complex documents created in Word 95 render properly on the iPad XXXVI in 2040?

Archival file formats like PDF/A, open specification formats like OOXML and open source formats like ODF will help with this issue, but I'm sure there will be many audio and video files that will be unintelligible in 30 years.

The other alternative might be to review your current archival storage format every year, and upgrade while the interface is still available but new storage formats are coming out.

Reply on news.ycombinator.com