Hacker News Reader

Comment by adrianN | Hacker News Reader

If we assume that Amazon's service survives the next twenty years, I still don't see how we can reasonably expect to be able to read the files we stored there. File formats become obsolete all the time. If you don't maintain your data it is quite likely that it is useless independent of where you store it.

View on HN

That is a problem, but it's a problem that you have with "roll your own infrastructure" as well. Digital data isn't like paper -- archiving doesn't mean keeping tapes in a vault. You need to periodically refresh and check the integrity of the media.

Amazon's service gives you an expectation that 99.999999999% of your data will survive intact on average every year. So while you need to maintain your data logically, you can set clear expectations with respect to the integrity of that data.

The fact that Amazon's infrastructure may not be around in 20 years isn't especially relevant either -- neither will any maintained infrastructure. Tapes need to be refreshed, archival storage infrastructure usually ages out in a 8-12 years, and you need to do a TCO analysis each time to refresh.

The other thing is that you get physical isolation. Flooding hasn't been a problem in lower Manhattan or Staten Island in recent memory. How many safe deposit boxes do you think were destroyed in those two places recently?

I know Amazon says they have 11 9s, but merely given that there have been 5 mass extinction events in the past 500M years, you probably can't do better than 8 9s. (11 9s gives you even odds that your data will last 70 billion years, as long as you pay Amazon's fee.)

You're probably right that Amazon holds your data for as long as that, however, there's another way to look at 11 nines: If you upload one hundred billion (10^11) files on Glacier, you expect to lose one of them per year, and not more. That's probably realistically achievable.

Compare this with S3, where you will lose 1 in one billion files per year. Shocking!

While physical media formats and hardware interfaces become obsolete all the time, it's something of a myth that file formats do the same. Even proprietary formats can usually be read by digging up a copy of the software that created them (in theory this software may be lost, but typically it is not). The one major exception is DRM'd files where the keys have been lost.

This is truer today than it once was. Go back, say, 15 years and you'll find any number of word processor and presentation formats that are only readable by long-gone software. I agree that, today, we've (mostly) coalesced around a modest number of standards that are either open or, at least, widely used and well-documented.

I'm not sure what you mean by file formats becoming obsolete all the time. This may be true for some proprietary formats that are no longer readable by a modern application, but for any reasonably widespread format there should not be much difficultly. I don't see any reason why we wouldn't be able to read most of our text, image, and video data decades down the road.

Remember, we're supposedly entering the "post-PC" world. How do you intend to open a WordStar document or Quattro in 2040, after 30 years of app stores and sandboxed apps?

The presentation of documents are critical as well. Will complex documents created in Word 95 render properly on the iPad XXXVI in 2040?

Archival file formats like PDF/A, open specification formats like OOXML and open source formats like ODF will help with this issue, but I'm sure there will be many audio and video files that will be unintelligible in 30 years.

Reply on news.ycombinator.com