Amazon CTO: 'You should be able to walk away' from cloud providers | Hacker News Reader

Amazon CTO: 'You should be able to walk away' from cloud providers

networkworld.com

53 pointsalphadoggs37 comments

"Koffler says it's not necessarily hard to move data out of S3, it just may be more expensive than putting it in."

This is a ridiculous statement. PUTs are $0.01 for 1,000 requests, and GETs are $0.01 for 10,000. Deletion is free. Data transfer out per GB is comparable to the monthly cost per GB.

Sure, putting data in is cheap because you don't get charged for bandwidth, but storing it costs just as much per month as moving it out would. If you've got somewhere better to store your data than S3, you'll come out ahead price-wise after a single month, so you should do it. That's not lock-in, not even close.

Convenience-wise, yes, there's a lot of lock-in with AWS, no question. But that's a different problem, it's not fair to imply that you're locked in because of charges to actually get your data out.

You'd be pretty silly if you had all your data only stored on S3 without a backup under your own control. So moving out should come at a cost of $0. You should be able to cancel your account at a moments notice and not lose a single bit, if only because the other side can do the same.

where do you put this backup? Another cloud provider? Servers you manage? Etc.

I think you're understating the cost of keeping such a backup, and the complexity that may come with it.

I think you're underestimating the cost of not keeping such a backup. That's a terminal mistake if there ever was one.

I would love to hear both of your estimates of the costs.

Not having a backup: your company goes out of business.

Having a backup: $1000 a year for a dedicated server you physically own.

Not all businesses would go out of business if some data was not backed up. If Chartbeat lost its data, it would be a minor inconvenience that the feature of their real-time dashboard where you can rewind and look back at an earlier point of the day wouldn't work until it repopulated the next day. They probably wouldn't even lose a customer over it. When they were a young company without much funding, it might've made business sense to only back up their accounts and not the historical data for the sites being tracked.

Not all data can be backed up for only $1000 a year either; it's not just a matter of storage costs and a server. A write-heavy service like Foursquare can't just turn off and do a dump at night; to have backups they need enough server capacity to replicate data as it comes in. For those servers to keep up, they have to be as beefy as the ones they're replicating -- a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth.

> a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth.

You are mixing data, memory and processing power. In the case Amazon crashes and burns, they need that CPU+Ram+bandwidth etc. near the data.

But to just keep a copy of the data (not available online) they don't need a really beefy server -- just store all the updates from a node, and replay them later at your leisure. 5400rpm will often be fast enough for this purpose (of course, you would need days to recover in this setup ... but that might be ok if past data is not needed online)

Take twitter or foursquare for example: They need everyone's most recent max(4, tweets/checkins in the last day) available online at any given time. But if last years' tweets/checkins are not available online when the system is degraded in case of backup restoration etc -- that may be acceptable.

Absolutely right, assuming the data you keep on S3 is mission critical (for me it's mostly just archival stuff, logs and such, that we never touch but have no particular reason to delete since S3 is so cheap, so we don't worry about backups).

But even if your data does matter, spending $1000 to back it up separately is not a no-brainer, depending what your data is actually worth to your business and how likely you think it is that S3 will lose it. Assuming your data is worth a million dollars, then spending $1000 to back it up implies that you think there's at least a 1 in 1000 chance that S3 will lose it all.

I don't know what probability I'd assign to "S3 loses all my data", but I do know that the particular data I store there is not valuable enough to make it worth backing up elsewhere. If my data actually mattered, I'd work out the math and consider alternatives. I will say this, as a word of warning - I would never use Amazon's durability claims ("Amazon S3 is designed to provide 99.999999999% durability of objects over a given year") when doing such a risk vs. cost assessment, because the real risk is not that the expected failure modes will "get you", but that something totally unknown and unpredictable will happen, i.e. someone at Amazon totally fucks up a staging push and sends it live to production instead, and the replication system goes nuts and starts corrupting everyone's data irretrievably before anyone realizes what's going on, or something like that. SLAs and durability claims don't account for stuff like that, but it's still a risk.

Of course, there's also the "cover my ass" and "I'm not paying for this anyways" factors to consider - if you would likely lose your job if the data in question was lost, and your personal finances are not affected one bit by the company spending an extra $1k/$10k/$100k on backups, then even if math tells you it's a waste for the company, you might as well do it, because as long as it's not too much of a pain to set up it will help protect you. The calculus always changes when it's other people's money you're playing with...

that's not a very reliable server if it's $1000/year. Are you really trusting that with data that's so important your company will go out of business if you lost it? Are you trolling?

But the real cost is the time people spend working on it. Sys admins, programming the process to get it onto the server, etc.

I don't think any of the data I have in s3 would cause my company to go out of business if we lost it. Most of them are thumbnails from videos that could be regenerated.

Try maintaining a backup of a bucket with tens of millions of objects that are constantly changing. Not such a trivial or cheap problem to solve now. Downloading the bucket listing alone can take hours.

Btw if anyone has any suggestions I'd love to hear them. So far the best approaches seem to be either to upload to multiple providers in the first place or to organise the buckets in such a way that bucket listings are not required to copy all files changed since a certain date.

I don't think he meant in terms of bandwith. Creating an s3 bucket and using it is quite easy. If you wanted to take that an replicate it by having some sort of redundant storage built yourself that's going to be expensive.

That's how I took it anyhow.

Well, in that case, the "lock in" comes from S3 being better and cheaper than the alternatives. In which case that's something to be applauded, not criticized.

What about AWS import/export? You just ship them drives and they plug it in, charged by the hour.

http://aws.amazon.com/importexport/

My friend Tom and I did a talk about this at various conferences, wow, 2 years ago. The interesting thing is that we're following the same evolution as the old snail-mail network and the SMS network. Eventually we'll realize that "cloud peering" is the best solution. But there will be lots of sturm und drang before we get there.

http://assets.en.oreilly.com/1/event/31/The%20Cloud_s%20Hidd...

http://www.slideshare.net/sh1mmer/the-clouds-hidden-lockin-n...

There were four separate things needed to bring about what we think of as the modern international mail system. First you needed better infrastructure inside and between countries. You needed literally more portable objects, envelopes with stamps on them, instead of loose sheets, scrolls, wax seals, etc. You also needed standardization of the rates and address formats so one letter can travel anywhere. Last was a uniform rate, no-questions-asked promise to deliver via optimized routes, what we would now call a "peering agreement". Two ends of the agreement would treat each other as peers, and honor each others' comminucations as they would their own.

It's that kind of system we don't have, but should, in the cloud. We need better infrastructure in the form of optimized routes between clouds. We need to be able to move our virtual machines and configurations around without special help. We need to make sure we don't get locked-in by screwball APIs or data formats. Most of all, we need the various cloud and web services vendors to commit to honoring each other's traffic without clobbering us, their customers, with metered billing.

Hi there,

This is super-interesting - may I ask where you learned about this part of the history of the postal service?

I started researching peering, and came across references to the SMS study and the Treaty of Berne. From those two clues I found everything else via Wikipedia and the library. Iirc there is a book about the history of the Berne treaty but the name escapes me.

Also: http://www.philippinephilatelist.net/Reading/Spanish/spanish...

I just tried to find the original source of the quote "the best SLA is choice" that I originally heard from Matt Mullenweg from WordPress. It seems relevant now.

EC2 is one thing and is fairly simple to move away from, more problematic is S3, DyanmoDB or Google App engine or anything like that. They touch only lightly on the subject to be honest. ORM's lol will not solve your problem in this case.

This article is way to fluffy to offer any proper advice.

While there are a lot of AWS offerings/cloud services which have vendor lockin, the core EC2 API is supported by Eucalyptus (http://eucalyptus.com) and OpenStack (http://openstack.com).

I don't think it's fair to call richer services lockin. GAE has lockin, in my opinion, because basic services use GAE-specific APIs and common needs (like scheduled jobs) are accomplished in uncommon ways.

If you're using auto-scaling or beanstalk or whatever, that's an example of AWS offering something where there isn't a norm, isn't a "normal way" to do it, and if you need that thing, your other alternative is to build it yourself - which is still an option should you decide to move away from AWS.

Lockin is avoidable switching cost which benefit the party imposing the switching cost. Are there examples of AWS doing that?

One I can think of is related to RDS - if you run your own MySQL instances, you can pretty easily migrate away to another set of (MySQL) machines by replicating off the master onto a new slave, and then promoting the slave to master and shutting down the original master. RDS doesn't allow (external) replication, so your only option to get away from RDS altogether is to do a mysqldump, which can get nasty if you are trying to migrate away without downtime (a full export/import via mysqldump can take a long time, and you'll miss any updates during the process). At the very least, it involves either a chunk of downtime or a decent bit of manual migration, which is unfortunate.

That said, I don't think they set it up this way to specifically cause lock-in, it's just technically difficult to allow users to manually manage replication alongside RDSs automated system.

Their messaging is a non-standard protocol. AMQP standard would be kick ass.

Same with their cloud search - sooo obviously solr/lucene-based but 0 exposure to the lucene syntax.

Not a big deal though, it's not like it totally locks you in to use these.

> Same with their cloud search - sooo obviously solr/lucene-based

Except it isn't. http://aws.amazon.com/cloudsearch/

"Amazon CloudSearch was created from the same A9 technology that powers search on Amazon.com."

Well my team and I have been working on AppScale for the past 3-4 years to do away with the lock-in for GAE. They iterate pretty fast on their platform, so we're always playing catch up. Eucalyptus and AWS did recently partner, so AWS has come around to supporting other implementations of their API. The EC2 API getting further reach is a good thing in my opinion for AWS even if they aren't directly making money off of it. Google has been cognizant of this from the beginning and hence why they helped fund AppScale.

Another project forwarded me by a fellow KPS engineer is the DeltaCloud project from the Apache Software Foundation. An API abstracting differences between clouds.

Not using it (yet), but it's on our viewscope.

http://deltacloud.apache.org/

For reference, here is the original blog post on Data Gravity:

http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds...

It's ironic that two vendors are so desperately trying to convince people that there is no issue with storing large amounts of data with them because it is so trivial to get it out. Never taking into account the dependencies that are created (and then broken if you try to move the data). Dependencies include: Latency and Bandwidth, Data that you don't own, the sheer volume of the data (try moving 10PB around), proprietary or specialized APIs, regulatory, and cost...

None of those dependencies are unique to the two vendors. With large amounts of data, most of those apply, even if you're on a fully open stack.

Absolutely agree, there is no magic fix for the effects that Data Gravity has (there are ways to lessen the effects). The only exception is in the API, a fully open stack gives you greater portability of your app and export of your data is easier because transformation isn't required (just export/import or replication/copy).

Something like this is why I'm really, really excited for Ubuntu 12.10, Juju and Awsome. A huge step in the right direction for sure.

If they really meant this they would AGPL their stack. It's the only way to provide your customers with the sort of ecosystem that allows them to freely choose their suppliers.

Clearly, they do not mean this in the way that you're implying. Amazon is not about to change their business model to become a professional services company. They are still retail first, cloud second (though that may change, of course).

I wasn't implying that Amazon should change their business model to focus on services any more than they are currently.

Apple, as always, was not available for comment...

Downvoters should turn on their brains before voting. This was very much on topic. gggrrrr

Reply on news.ycombinator.com