Hacker News Reader

Comment by teraflop | Hacker News Reader

I would temper the suggestion to use Glacier with a warning: make sure you thoroughly understand the pricing structure. If you don't read the details about retrieval fees in the FAQ, it's easy to shoot yourself in the foot and run up a bill that's vastly higher than necessary. You get charged based on the peak retrieval rate, not just the total amount you retrieve. Details here: http://aws.amazon.com/glacier/faqs/

For example, suppose you have 1TB of data in Glacier, and you need to restore a 100GB backup file. If you have the luxury of spreading out the retrieval in small chunks over a week, you only pay about $4. But if you request it all at once, the charge is more like $175.

In the worst case, you might have a single multi-terabyte archive and ask for it to be retrieved in a single chunk. I've never been foolhardy enough to test this, but according to the docs, Amazon will happily bill you tens of thousands of dollars for that single HTTP request.

View on HN

They explain it really weirdly, but it's not that hard to cap your expenditure if you translate it into bandwidth units rather than storage units, since that's what's really being billed. The footnote on the main Pricing page is the most misleading part of their explanation:

If you choose to retrieve more than this amount of data in a month, you are charged a retrieval fee starting at $0.01 per gigabyte.

From this you might think the retrieval fee is some function of the gigabytes retrieved, but isn't really in any direct way: you're charged ((max gigabytes retrieved in any one hour in a month) x (720 hours in the month) x per-GB price), modulo some free quota. So you can really get charged up to "720 gigabytes" for a single gigabyte retrieved, which is a bit silly to think of in terms of per-gigabyte pricing.

It makes more sense to me if you think of it as priced by your peak retrieval bandwidth. Then you can cap your expenditure by just rate-throttling your retrieval. Make sure to throttle the actual requests, using HTTP range queries on very large files if necessary (many Glacier front-ends support this)... throttling just your network or router will not have the desired effect, since pricing doesn't depend on download completion.

Anyway, the retrieval pricing in units of Mbps is ~$3/Mbps, billed at the peak hour-average retrieval rate for the month above the free quota. If you use that as a rule if thumb it's not bad for personal use. For example given my pipe, it's perfectly reasonable for me never to pull down more than 10 Mbps anyway, in which case my max retrieval costs are < $30/month.

Yeah, that's a good summary.

Another way to look at it is that if you retrieve data as fast as possible -- say, to S3 or EC2 -- it costs about $1.80/GB, minus the free quota. So for large objects, it's cheaper to store them in Glacier than S3 if you don't expect to need them in a hurry any time in the next 18 months. The free retrieval quota starts making a difference if your typical "ASAP-retrieval" size is less than 0.1% of your total archived data.

Reply on news.ycombinator.com