Glacier redux

> Old hard drives that are no longer economical for more intensive service, supported by disk-handling robotics.

I am with the author in that I do not think they are using older disks. From personal experience (working with petabytes of disk based storage), I would put money on the fact that AWS is not using stock off the shelf older disks and then powering them on/off as a method of storage. The last thing you want to do with an older disk (5+ years) is ever power it off. The chances of it coming back to life decrease rapidly with each cycle. Picture seized bearings, etc. Your best hope it to keep them continually moving, but never powered down for extended periods of time.

Also, the GB/power/physical space required ratio does not make sense after X number of years. For example, you could replace 3 shelves of 1TB disks, with one shelf of 3TB disks. Yielding less power consumption in less physical space! When you are at scale, data centre space and power are big factors. For these reasons, I do not think they are using old stock disks.

NOTE: Maybe they are using some custom make disks, but from what I know about standard disks, even enterprise, they do not like being powered on/off when they get old.

The author handwaves away the power savings of powering down disks by talking about the capital cost of providing power.

"Unless the prices of copper, PDUs and diesel-generators have started following Moore’s Law, this is probably more true today than in 2007."

But this is fallaciously assuming that Glacier servers would need to be like EC2 or S3 servers that are switched on for x random hours per day. This isn't the case at all.

Diesel generators, for instance - what's the use? If you're replicating the data around the world, it doesn't matter if your Glacier servers in one location are powered down for days.

Cooling, power - just switch off every single Glacier server at each location during the peak hours of the day. No additional capital costs, because you're never adding to peak power usage.

Distribution of reads - Profile customers based on how frequently they're reading from the data store. I bet 50% of customers never read a single byte back. Colocate these customers on the same servers, and you only have to power up the server once a month to check if the data is still there.

Bonus - Profile data access patterns with simple heuristics to determine what's likely to be read back in, and temporarily store that data in S3. Imagine a company that archives everything to Glacier daily, but restores day, week and month old backups regularly. Keep all data less than a month old in S3, and the rest on the indefinitely powered down servers talked about above.

I actually quite liked the author's case for BDXL, but it seems he's doing a straw man on other possible solutions. Well implemented BDXL more cost efficient than a naively implemented disk strategy? Not exactly news.

If you have a lot of heavily-read data distributed across machines, you're probably constrained by available spindles rather than available storage space. So co-locating data that will almost never be read with heavily-read data is effectively free. I would guess that Amazon stores Glacier data alongside S3 data, and the lower price reflects the fact that the limiting factor in their storage system is IO rather than capacity.

  But NONE of the Hacker News commenters addressed Sony and 
  Panasonic’s continued investment in high-density optical 
  disc technology. [...] There has to be a business reason 
  for the continued investment, i.e. customers prepared to 
  buy a lot of product in the future and buying a lot right 
  now.

If Amazon was the only customer for high-density optical storage, they'd be crazy to invest in developing it, because Amazon will have all the negotiating power in the relationship.

There must be other customers somewhere if two companies are continuing to develop this stuff.

The deletion charge is interesting. Does deletion guarantee a scrubbing of the data as soon as possible? If so, I could see that as justification for the fee since some sort of significant work is involved in retrieving data, implied by the hours of wait time. If not though, the author has a good point in that Amazon needs to make at least .03/GB in order to be profitable.

Okay, this is beyond silly. Still ignoring tape.

Explain how BDXL, a new format, which has never proved itself beyond a few years, which costs $45 for a few hundred GB, which AFAIK has very little data on re-writability (which is probably terrible and close to one-time use) could be any more profitable or reliable, or useful than tape.

I don't buy it. Tape would be the logical short-term choice to get started, because Amazon could just go buy an off-the-shelf tape library and add tape as they needed from Oracle, versus again, engineering their own BDXL library on the expectation that it would cost less than tape, taking into account factors such as:

1. Reliability and degradation 2. Supply 3. Cost 4. Reusability/Re-Writability

What I'd like to see is a storage product that aggregates the unused storage space on EC2 instances.

* By default instance storage isn't attached, so there's probably a lot of completely available capacity.

* Even if attached, it's rare that the full capacity would be used, so thin provisioning would leave some space available.

* Some host machines won't be fully allocated.

I imagine taking this pool of capacity and using erasure coding and replication to build reliable storage. As your volumes come and go, you need to make sure it remains available, which is why I imagine erasure coding across a large number of customers. By integrating with the guest -> host assignment function you can ensure that you never lose data, if need be delaying scheduling until you've copied data elsewhere.

You'd have to throttle reads & writes to ensure that the guests weren't unduly impacted (easier with SSD's predictable IOPS), and the splitting and erasure coding would make for slow reads & writes as well. But this makes the economics a lot more attractive (free!).

One thing that suggests Glacier could be doing this: if I was AWS, and I was doing this, I would not be in any rush to tell EC2 customers that I was "stealing" their unused capacity!

The Amazon Glacier Secret is hidden in plan sight. I think is just a virtual product. Probably unused S3 capacity, at a much lower price, not a different technology.

Full disclosure: I work at UpCloud, a cloud hosting provider and I think business in this industry day and night.

One thing that gets overlooked in almost all comparisons is the pricing models. I actually use Glacier through Arq (brilliant backup for Mac), but the catch is in the requests. I recently uploaded 200GB of photos to glacier and the upload process cost me about $10. The monthly storage is about $2.

The thing is that you shouldn't compare the sole storage price, but the total of cost of storing your data in glacier, which was also overlooked in the original article.

I'm sure AWS has understood this through their S3 storage lifecycle and thus developed an appropriate pricing model for glacier that arouses interest in the product like no other.

    Glacier is significantly cheaper than S3

Yes, as long as you put something in and almost never take it out.

    They charge for deletions in the first 3 months

What if this is just disincentive to pull content out and treat Glacier like S3?

    Power is not the driving cost for Internet scale infrastructure

It is not the only cost, but it is still one of the largest factors, no?

    Sony and Panasonic continue to invest in a product that has no visible commercial uptake

That means nothing in and of itself. The only person who will win such a market needs to be one of the first there, innovator's dilemma, etc.

    Facebook believes optical is a reasonable solution to their archive needs

Do they? I saw one mention in the author's previous post of James Hamilton commenting on a Facebook cold storage system using Blu-ray but is unclear to me if it is in production.

Assuming it is true though, it is likely an apples-to-oranges comparison. Glacier provides archival restoration for presumably largely enterprise-level customers. Facebook backs up data from users, and I'd presume this is from deactivated accounts, etc., and unlikely to need urgent restoration.

Other than burying the cite (2 links to his own blog) to why "cost of provisioning a single watt of power is more expensive than 10 years of power consumption," the author's canonical source is eventually a 404:

http://static.googleusercontent.com/media/labs.google.com/en...

The data is also > 7 years old.

EDIT: working link:

http://static.googleusercontent.com/media/research.google.co...

If power provisioning and space are an issue it seems unlikely to me that they're sitting with many drives all hooked up for immediate power-up. It would make more sense to separate the storage of media and the power connection so you get the most from each power connection. Therefore: removable media, whether those are hard drives or optical.

There's a data retrieval period of multiple hours [1]. That doesn't sound like they're just powering up a drive. It sounds like they're moving something around or doing some kind of linear read (as opposed to random-access). I'd bet on "a retrieval job fetches this stack of read-only media and connects it to the powered device, which loads it onto hard drives for quick download".

[1] "Retrieval jobs typically complete within 3-5 hours" - http://aws.amazon.com/glacier/faqs/

I've always thought that the deletion fee for <3 month old data was a way to limit "misuse of the service" - call it a detergent if you will - more than actually a way to recuperating costs. Because I guess the costs will be there no matter when you delete data

I still think it is possible to make the whole thing work using a tape library.

If the goal is to manage capital costs, I don't see how funding the development of an entirely new data storage hardware ecosystem is a reasonable answer. You control capital costs with commodity hardware, not cutting edge.

Yet that is what the author thinks Amazon is doing with BDXL.

So who's funding high-capacity optical storage? Hmm, can we think of a customer who ingests huge amounts of data, wants to keep it for a long time, and has no fear if funding cutting-edge product development? Yes: defense departments.

My friend Robin Harris wrote the original article.

A little off topic, but it seems really strange to me that Amazon is not transparent on the technology. Because of the high charge for fast reads, I tend to believe that the underlying storage is some form of media that gets mounted, perhaps like Facebook's bluray archival system.

Is data really so heterogeneous? How many files in your home directory are generic, with bit-exact duplicates existing in the home directories of many other users? And the remaining files, which are uniquely yours -- what percent of each of them consists of generic data, like file headers?

Don't forget that S3 costs additional $0.12/GB for transfer out.

So either cheap VPSs or dedicated servers are still much cheaper if you don't want to deal with Glacier.

another question: what is the difference in power consumption between a HDD that's powered up but idle, and one that's being used to read from or write to?

As an aside, erasure codes allow you to reduce power consumption as the redundant fragments are only necessary for safety, not for regular retrieval. You don't need glacier to benefit from that. (But glacier might be an optimization of that strategy)

Don't forget that S3 costs additional $0.12/GB for transfer out.

Dedicated servers are still much cheaper than even Glacier.

question: how does Amazon's glacier relate to this? http://dl.acm.org/citation.cfm?id=1251214

Erm, isn't it getting a bit meta to have a post to respond to the comments on HN for a previous post of yours? Surely, that is exactly what the HN comments are for?