Hacker News Reader

Comment by re | Hacker News Reader

Glacier's pricing structure is complicated, but fortunately it's now fairly straightforward to set up a policy to cap your data retrieval rate and limit your costs. This was only introduced a year ago, so if like Marko you started using Glacier before that it could be easy to miss, but it's probably something that anyone using Glacier should do.

http://docs.aws.amazon.com/amazonglacier/latest/dev/data-ret... https://aws.amazon.com/blogs/aws/data-retrieval-policies-aud...

View on HN

> fortunately it is fairly strait forward to cap your data retreival rates.

Amazon has done a great job with this feature. By doing a poor job implementing something for an extremely narrow usecase, in a technology that is outdated and then providing the most complicated pricing structure surrounding every aspect of the product one can't helpbut use the feature: any other provider or service.

Like, wtf would be the usecase for amazon glacier in 2016? I dont think I would put hubdreds of petabytes of sata into 20 year cold storage, and the author of this post certainly wouldnt use it again. The fact that i need to read 2 pages of pricing docs and then the 2 pages you linked to control them because I cant estimate them myself, is a sure sign this is absurd

Tape storage is still the most optimal form of long-term storage. If you need to store things for an exceptionally long time, such as financial data, scientific data, etc, then you're going to get the most bang for your buck on tape.

Yes but if you were in this target market you'd likely want your information on your own tape with a reading infrastructure you control.

Depends. Having an in-house key and shipping everything to Amazon encrypted means that you have all the infrastructure there and waiting, and not capable of being read. Additionally, that tape library would need to be stored, and periodically tested so that tapes can be rotated out as needed. Sending that data to a service like glacier means you've shown due diligence, but at the same time, don't need to maintain a schedule of testing every disk every year.

don't need to maintain a schedule of testing every disk every year.

So you trust Glacier or Google Nearline (or any similar provider) without testing? No testing ever???

I wouldn't feel happy with my critical data floating around in the cloud, without my checking it at least once a year to make sure it really exists.

And once you do start verifying that data, you will incur all sorts of charges to access it.

this was what I meant. thanks. You can buy a 5tb hard drive for ~138.00. You could likely buy several of them at discount to get started. As you go forward in time, these will become much cheaper, allowing you to continue purchasing them on demand from the market for much less money.

This allows you to trivially share, copy, move and retrieve that data quickly as well as fully control who has access and when.

I am sure there are use cases for this but in a situation where you have petabyte scale data, it is often the case that you also have the infrastructure to save it. How many places would need to store >5tb of data a week that

* don't have this capability in house

* will almost never need to access it again.

* will not need it in a timely manner, if they do need to access it again.

* don't have the money to implement their dedicated server and storage on site for this purpose.

I am not saying that this rules everyone out, but the prices are so low, and tape must be so annoying, I couldn't imagine why they keep offering this. Obviously, some peopel must be using it but in 2016 with storage prices being so low already, i don't know how many places have this amount of data and meet the above requirements.

2 Words: Offsite Backup

The post states he paid $150 yo retrieve 60gb of data. For $150 you can buy a 5tb hdd drive.

What usecase could the price delta make sense to have a 4 hour feedback loop and all of your important data locked in someone elses data center.

the usecase where your data is so properly massive that this makes sense && you don't have the storage infrastructure in place, is so narrow that it doesnt make sense.

It is basically one research student's crawl data

Edit: also, s3 is pretty cheap. So again, i dont really see the usecase here. How much room is in the market between your own physical or digital system and amazon s3 or an equivalent. you would have to have a massive amount of data you dont care about and be very price sensitive.

Durability is one reason. A physical hard drive in my desk is so much more susceptible to destruction or loss or theft.

This.

The data I care about is already backed up on two different multi-TB drives at home, and another one at work.

Glacier is the contingency for "something took out the original data and all three backups in two different locations 7 or 8km apart - if I'm still alive after whatever just happened, I'll consider whether or not to pay Amazon a grand or so to retrieve it quickly from Glacier, or wait ~20 months to get it all for single-digit-dollars".

If you're talking about personal data, why not just use backblaze and amazon's consumer unlimited cloud storage?

That gives you 2 backup providers that can durably store everything and it's free and quick to access. Why deal with all the harddrives and glacier?

Neither of those seem free - am I looking in the wrong place? Also there's no Linux client for either AFAICT.

Right the services cost money but retrieval is free. Going by the cost of the harddrives amortized out, it'll probably be the same or less. You get far more durability and less complexity with universal web access.

I believe there are other similar services for Linux or you can just use browser to upload files with Amazon.

A user posted a long example below: https://news.ycombinator.com/item?id=10921709

You don't have to pay $150 for retrieval of 60GB. And you don't do long-term storage for X TB / 5 TB * $150. You might have to rent space in someone elses datacenter to put your own external backup... or you could pay Amazon for Glacier and not deal with maintenance etc. Might be worth it even if you have infrastructure for all data that isn't glacier-cold.

He paid $150 plus between 50 and 80 CENTS a month to hold a backup of 60GB for four plus years.

There are some use cases listed here, and they seem pretty reasonable to me: https://aws.amazon.com/glacier

Lots of businesses have data retention requirements, and it can be difficult and time consuming to make sure this data is backed up in a way that is secure and can survive a catastrophe.

The author's use case (and most other personal use cases) might not be a good fit for Glacier, but he's not the target market.

SOX compliance, legal requirements to save communications, etc. There are a lot of places where there are needs to maintain a huge amount of information that you're probably never going to need again.

Not all products are for all people. If you foresee a need to recover a large amount of data all at once, then glacier's not for you. If you might occasionally need a filing from 6 years ago, then glacier would be great.

It's not about recovering a large amount of data; its about recovering a large % of your own stored data.

Amazon starts to charge you extra anytime you exceed restoring 5% of the data.

If for example you save all your tax-related documents in Glacier, then you are audited then the accounts department or the government will want all the information. Not 5% of it. Not 10% of it. Everything. At that point Amazon will have you over a barrel, because getting the data out at a reasonable time frame will cost exponentially more than dripping out the data over the course of 20 months.

>If for example you save all your tax-related documents in Glacier, then you are audited then the accounts department or the government will want all the information. Not 5% of it. Not 10% of it. Everything.

Are you sure about that? I haven't worked with tax litigation specifically, but I've worked with e-discovery w.r.t. e-mail and I can assure you that no one ever asks for all the e-mails sent by a particular company over all time. It's always a matter of asking for the e-mails sent by or received by a select group of people, over a fairly discrete time period. For something like this, a Glacier store might make sense, if it was coupled with an online metadata cache stored in e.g. S3.

With tax litigation the issue is that you have to prove you didn't simply shift money and accounting briefs around, and the only way to realistically prove it is to show all the statements in the time period that you'd required to keep them (I think that's the last 6 years).

The government basically comes to you and says they think you owe X, and you have to prove that false to their satisfaction. The more data you give your CPA to work with, the better.

Glacier is about storing a large amount of data in a cost-efficient manner, when you do not anticipate needing it 1) cheaply and 2) quickly.

Legal issues, at least in the US, do not have those requirements.

I have had situations in the past 12 months where recovering past data would be worth >$1k for the right 100k of data.

> Amazon starts to charge you extra anytime you exceed restoring 5% of the data.

Isn't one way to get past this... increasing your data usage by 20x? If OP used less than a $1 a month, then if he uploaded $20 of junk data, he can get the 5% original data back "for free". Sure, it's $20, but it beats out $150+.

A better way would be if you had 20+ categories of data that are totally unrelated, like; your tax-stuff, your code, your diary from 1995-2010, ... Since these are very unrelated, you are not likely to need every one of at the same time, ASAP.

Though it's hard for me to imagine having so many categories of unrelated, useful and important data.

It looks like it is even more complicated than that. You can get 5% out per month at no charge, but that it only if you spread it out across the entire month. The extra charges happen the first time you exceed 5%/30 in a single day.

Reply on news.ycombinator.com