Hacker News Reader

Comment by phil | Hacker News Reader

Storage experts: I'd love to know more about what might be backing this service.

What kind of system has Amazon most likely built that takes 3-4 hours to perform retrieval? What are some examples of similar systems, and where are they installed?

View on HN

Typically they are tiered.

There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access.

Then there'll be a robotic tape library. Any restore request will go in a queue annd when an arm-tapedrive becomes free they'll seek to the data and read it into the HDD array.

Waiting for a slot with the robot arm - tape drive is what will take 4 hours.

EMC(kinda), Fujitsu etc make these.

http://en.wikipedia.org/wiki/Tape_library

http://www.theregister.co.uk/2012/06/26/emc_tape_sucks_no_mo...

Wouldn't there also need to be a lot of logic to prevent fragmentation? You'd probably want data from one user near other data from that user, i.e. on the same tape.

The multiple-hour window could give you a lot of wiggle room here though. It's unlikely to take 3 hours to restore from a single tape, so even if you have to visit 2-3 tapes then you have plenty of time.

I'm sure that there is a general tiered storage platform (as mentioned above) which keeps some of the data online as well. That would let you run a "defrag" algorithm later if you find you need it.

I'd guess that they ignore that problem and have baked the time it takes to get data from several tapes into the 3-4 hour estimate.

If you think about it, writes are more common than reads on average, so it's more efficient to just write to whatever tape is online and deal with the fragmentation problem on the read end, as opposed to queueing writes until the 'correct' tape can be brought online just save some time reading. Also in backup situations like this, it's more important to get the backup done in a timely manner.

That's true, but they're obviously using a multi-tier solution, with about a 90 day buffer before things go to tape:

  In addition, there is a pro-rated charge of $0.033 per gigabyte for items deleted prior to 90 days.

So it may just be feasible to organise writes so they end up together. But, yes, ultimately it is probably not worth it to do so.

Given it takes 3.5-4 hours to retrieve a backup before you can download it, probably not.

Thanks -- I had no idea you could just buy a tape library off the rack (more or less).

Do you think there is redundancy baked into the tape drive system?

Close. Tiered, yes. But remember who we're talking about.

First, no tape. The areal storage density of tape is lower than hard disks. Too many moving parts involved. Too hard to perform integrity checks on in a scalable, automated fashion without impacting incoming work.

Second, in order to claim the durability that they do (99.999999999%), that means every spot along the pipe needs to meet those requirements. That means the "near-line HDD array" for warm, incoming data needs to meet those requirements. Additionally, if the customer has specified that the data be encrypted, it needs to be encrypted during this staging period as well. It also needs to be able to scale to tens if not hundreds of thousands of concurrent requests per second (though, for something like Glacier, this might be overkill).

They've already built something that does all that. It's called S3. The upload operations likely proxy to S3 internally (with a bit of magic), and use that as staging space.

After that, the bottleneck is likely I/O to Glacier's underlying storage - but again, not tapes. See this post for deets: http://news.ycombinator.com/item?id=4416065

Could be for example some tape robot where you can have huge amounts of tapes in the storage but only have few devices for reading/writing them. With tape you can't really stream the data to the web. Instead you would probably first copy it somewhere. If there is lots of data, say few terabytes, even this process takes some time.

Or in case they are using regular hard drives, you might want to have this kind of time limits in order to pool requests going to a specific set of drives. This would enable them to power down the drives for longer periods of time.

The 3-4 hour estimate may also be artificial. Even if you can in most cases retrieve the data faster, it would be good to give an estimate you can always meet. They might also want to differentiate this more clearly from standard S3.

And we should not forget that it does take time to transfer for example one terabyte of data over network.

Take a look at Linear Tape File System (LTFS), which allows for ad-hoc file retrieval. CrossRoads Systems out of Austin has the leading implementation, and they did bid on this Amazon contract. I have no idea if they won it, though (next corp conference call is Aug 29th). They just closed a joint investment with Iron Mountain, so my money is on an LTFS solution with CrossRoads as a systems vendor.

See my post up top: http://news.ycombinator.com/item?id=4416065

Reply on news.ycombinator.com