Why we ditched Amazon AWS

Except people were doing that long before the "cloud" marketing nonsense started,

Agreed, except that your internal documentation can't abstract away all the infrastructure. At least for the company as a whole.

No, thats ridiculous. You are still paying for all the servers in the server farm, even if you have decided to waste another year of Moores law to add a bulky abstraction layer on top.

The whole point of a cloud to a business is that you can adapt to changing needs without having to pay for all the infrastructure all the time.

(You didn't think the suits talk about "the cloud" all the time because they fancy the technology, right?)

There's a maturity cycle -- risk shift with time.

On day one, buying equipment, labor and software to stand up a computing environment is a risk. You don't truly understand your needs.

After a couple of years of operations, you should have a better idea of your typical run state and growth patterns. At that point the risk may shift. If I can deliver compute with AWS for $X, but can also deliver that compute through some other means for $X-Y, you have a new risk/cost evaluation to do.

The key is continuing to use the cloud thought process.

You're looking at this wrong. You're thinking of TestingBot as the customer, so you see one customer replacing AWS with dedicated servers and calling it a cloud. That's not the actual scenario, though. TestingBot built a cloud for their customers. They built a way for their customers to offload the risk and costs of scaling their test infrastructures. They built a simple[1], automated way for their customers to easily scale. They built a way to share the same hardware among many customers and thus lower costs and improve utilization. This is a cloud.

I also don't really agree with your claim that the key attribute of "the cloud" is offloading the risk and cost of scaling. From my perspective, "the cloud" is primarily about simplifying and automating scaling, thus reducing the human and time costs. It's not just about dumping the cost of physical machines onto someone else. I see taking less risk on machine purchases as a nice benefit, but not the core purpose.

[1] I haven't actually used TestingBot, so this isn't an endorsement. I have no idea if they actually do a good job or not.

That's not at all the case. "Cloud" isn't about external hosting, its about using virtualization to allow dynamic provisioning of resources. Outsourced external hosting was around a long time before cloud computing. Its a key application of cloud computing, but they are no more equivalent than "internal compustion engine" is equivalent to "wheeled land vehicle".

Like an hôtel particulier? That's the origin of the word hotel.

The relevant and important difference being that this is a one-time expense. It contributes almost nothing to the cost unless you do this then immediately drop the platform.

Exactly, now we guarantee a pristine VM for every single test, something which was not possible with our setup on EC2.

This can solved by containerization.

I think you get billed for at least 10min... still better than 1 hour though.

Azure bills by the minute as well.

http://weblogs.asp.net/scottgu/archive/2013/06/03/windows-az...

Are many people using so few aws services that they are able to move to google? It seems like a hassle to use google for VMs and file storage, while having to still use amazon for all the other services google doesn't offer.

The reality is that once you reach a reasonable level of scale (multiple physical machines), the cost savings from going to your own cloud will easily allow you to afford that extra capacity and then some.

Might be true. If you have stable load moving off of EC2 can save you at least 50% over reserved instances, and >90% on bandwidth costs. If you no longer need amazon's scaling advantage, it's a very easy call to make.

Of course that's assuming you haven't locked yourself into amazon's databases/load balancer/... But that would be a horrible idea anyway, right up there with running a microsoft stack licenced per year.

It's a pretty arbitrary constraint. I wonder if he's looked into docker or other methods of containerisation?

RDS recently added cross region snapshot copies, and more importantly, cross region replication for at least MySQL. Those were the last two features I was waiting for to jump to RDS. In fact, I was in the process of setting up cross region backups using just the cross region snapshot copy when they released the new cross region read replicas feature.

I'm very much looking forward to moving our manually managed cross data center MySQL replication/admin to RDS. But I also don't have to scale MySQL to ridiculous levels either, I just need very high availability.

EC2 doesn't do that, AWS does, for many things.

E.g. you can have a nice app running with elastic beanstalk, file storage on S3, database on RDS, full text search on CloudSearch, without worrying once about apt/yum, incompatible JVM versions, backups, stuff in /etc or setting up iptables and ssh, which is what most people think of as sysadmin work.

No the paper is saying the opposite -- that this situation can be the case -- what I was interested in was hearing their evidence that this is happening to them.

With a username like 'badman_ting', you must be a fellow Jamaican :). How can I contact you? There's no email in your profile.

>But I think if EC2 didn't let you borrow CPU, nobody would use it because they would realize how absurdly underpowered EC2 offerings really are.

yeah, that's pretty much the crux of virtualization; it's a lot like buying bandwidth. Yes, your upstream is oversubscribing. Yes, if they do this right, 99% of the time, you won't notice the oversubscribe; you will get more service for less money vs. something that isn't oversubscribed. But, oversubscription needs to be managed carefully.

One thing I've noticed? If your upstream has a 1000mbps line, and sells 1000 unlimited 10Mbps ports, you are almost never going to see contention, even though it's a 10x oversubscribe.

If your upstream has a 1000Mbps port, and sells 10 1000Mbps ports? that's the same 10x oversubscribe. But I /guarantee/ that you will hit contention at least once a week. Probably more often. If people expect 1000Mbps reliable off that, they will be very unhappy. (Of course, if you setup your QoS properly, and tell the customers that it's 100Mbps CIR and up to a gigabit of best-effort burst? it can work out just fine. But nobody is going to get a reliable full gigabit out of that deal.)

90% of your users are using like 10% of your resources. But you've always got a few who are running torrents (or, in the case of CPU, mining primecoins) In the 1000 10Mbps port situation? it doesn't really matter if you've got a few bittorrent users. In the 10 users who can all completely fill the pipe situation? it matters a /lot/

That's the thing about CPU sharing, though; most of the time you don't put that many guests on one machine, and often you give each guest the ability to use the whole machine (when it's otherwise idle) - so you are in the situation of selling 10 1000Mbps links when you only have 1 1000mbps uplink, which ends in tears if anyone actually expects a reliable 1000Mbps uplink. (Now, if everyone understands that it's actually 100Mbps CIR that can burst to 1000Mbps, then sure, people can be happy. but you have to be careful with those expectations. With the 1000 10Mbps links on a 1gbps uplink, customers can treat their 10Mbps link as 10Mbps dedicated, and 99% of the time, they will get what they expect.)

What you say might be true for CPU usage, maybe even for network usage, but definitely not for I/O. In that case it's definitely noisy neighbors causing problems for others, but in a way it doesn't matter. Whether it's noisy neighbors or hard resource limits, the variability is still there. For single-machine stuff you just have to assume the worst case. For things that are distributed among large enough numbers of nodes you can do a certain amount of averaging, but high variability still forces you close to the worst-case assumption.

I've run a lot of tests in this area, and even presented some of the results at LISA last month. When it comes to I/O, EC2 is considerably less consistent than many others[1] and that's what really hurts their users. Their prices might be only 2x someone else's if you're only looking at the average case, but it's more like 10x if you consider their extreme variability as well.

[1] Side note: this is a hint that Amazon has an unusually high oversubscription ratio. The same practice has been evident in shared web hosting, with the same effect, since that industry was created.

actually that may be a good use case for it!

it's real seamless and nice just ends up being expensive after a while if you are trying to do enterprise stuff & their add-on services aren't particularly cheap. The customizability of AWS & private stuff means you have to do a bit of server admin but it is generally cheaper & can give better performance.

EDIT: Oh also note that (my biggest gripe) I see big performance swings / queue-ing issues that aren't really correlated with traffic. Plus they introduce some platform/API changes intermittently that make their admin UIs kinda buggy. Or you have to change your workflow to integrate with their services (though the APIs for this are usually ok). I don't know, I feel like they make a lot of money draining people on threads since the performance of RoR overall is kinda questionable, along with the performance of their platform. I'm going to try with Play framework soon & see if there are less of these issues.

nothing's wrong with it (although it's expensive). I think the point was the Heroku is a service built on top of EC2. Switching from Amazon EC2 to Heroku isn't switching your platform at all... it's just paying for convenience. Personally I think Heroku's convenience is worth the difference. The question is more if the base EC2 pricing (which Heroku adds to) is worth it for your needs.

Cheap, easy to start. Expensive as hell and limiting when you need more flexiblity and custom tools on your server.

You're missing the point. EC2 specifically didn't work for them because they were constantly spinning up new instances and killing old ones, so they were repeatedly incurring the minimum one hour cost. If your business model doesn't call for that sort of a workflow, the article is irrelevant to you.