How and why we switched to Amazon EC2 | Hacker News Reader

How and why we switched to Amazon EC2

blog.railsonfire.com

As I was asked about the implementation details I'll write a second blogpost at the end of the week that goes into the details of our implementation.

If the main motivator here is price (ability to charge per hour vs per month), and EC2 is so much more expensive than a VPS provider, have you done the math to see if buying a couple extra VPS's for the month would not break even or be cheaper overall? Linode is kind of pricey compared to other VPS providers.

The larger your site gets the less you're going to need to rely on "hourly" scaling (as your traffic models also become more predictable). There's also the question of uptime, which we've all seen is not a guarantee from just using Amazon.

From the link:

As a hosted continuous deployment service releasing changes and scaling automatically is a necessity.

I think the key here is that this blog is talking about a hosted service which responds on demand to their client's needs on an hourly basis. EC2 has a lot of infrastructure which does this for them (as long as they can automate it) and is specifically built for this, whereas they'd have to write it themselves by interfacing with a VPS API otherwise. So EC2 might be worth the money to them. VPS providers are mostly more focussed on giving you 1 or 2 boxes to use, and then you leave your site running with occasional adjustments in capacity etc, not on spinning up and down servers on demand, which is a bit more specialised, and might require tens or even hundreds of servers.

For a normal site that's not going to be relevant, and a VPS is cheaper, for this site, it seems EC2 is a better fit...

Exactly. Price itself is not the main issue, but flexibility is.

For example on Weekends we typically see 20%-30% the traffic that we see during the week, simply because people are not at work. Scaling up and down is an absolute necessity there.

I think you're making a lot of assumptions about how a service like their could work and does work. You don't need to write a lot of code to leverage VPS's or scale them, and you don't always need to scale up or down on an hourly basis.

I'll bet you that if they took the historical metrics of their services and their load use and averaged it, buying one or two boxes would easily break even and at the same time not be dependent on a specialized provider or functionality like an auto-scaling VPS.

But that's just a guess.

We actually went that route with historical data and it wasn't any good to be honest. Work patterns change daily, so we really need the auto scaling.

Working with Historical data means you either totally oversubscribe certain days, or totally underdo it. It has to be automated due to the latest load.

> Linode is kind of pricey compared to other VPS providers.

Can you qualify this? The cost is lower than Amazon and Rackspace Cloud.

There are dime-store VPS providers out there that are technically less expensive, but they're usually not able to provide the same level of service. Linode always seems to provide exceptional customer support.

Do you know how many "dime-store VPS providers" there are, and how many of them have existed as long as Linode and longer? Linode provides a valuable, quality service, and has lots of extras that you may not need.

Some VPS providers specialize and have a good history of both uptime and support, and provide it for less than the big-name competitors. I've found many to be exceptionally reliable with good support, and there are communities where you can find reviews and status on such providers to back up their claims.

And in the end, realistically, if you're doing small scaling by the hour you probably don't need the biggest providers in the game. A small service provider for a small company is often a great fit.

You can check out reviews on http://www.lowendbox.com/ and http://www.webhostingtalk.com/ and compare prices, services and reputation. In general, a standard small-sized VPS will be half the cost of Linode, meaning you can buy double the VPS's (on different providers, for example) and get double the capacity and redundancy for the same cost. If you spread them out by region you can implement cheap software GeoIP load-balancing and serve the content by servers closest to your users.

I too have found quite a few usable offers from lowendbox.com (LEB) and lowendtalk.com (LET). Searching the websites bring up quite a few attractive deals.

However it may be tricky to pick a good host. It is usually a good idea to look for past experiences before signing up. The community is active and helps make that decision: http://www.lowendtalk.com/discussion/5136/best-leb-host-q3-2...

The VPS market is crap if want to look at the low end market.

This comes from my own experience running my own small scale VPS host for a few years, with a purposely small customer base (mostly people I knew personally).

Look at WHT and follow it for a while. That market is crowded, a race to the bottom in pricing AND support, and is full of drama. Some examples I've seen:

* Lack of experience in production systems with no backup procedure... some systems go down, they loose all data, entire business caves.

* Super tight margins and low capital... fall behind on a build, their provider shuts them down (most these guys are leasing dedicated servers or reselling from another provider)

* Don't know who really is in control... reselling from someone and don't know who? On dedicated servers with a provider who is known for a problematic network?

* They're some 16 year old and their dad grounds them from the computer so they can't answer support tickets.

* Don't understand the technology they're using for virtualization and get noisy neighbors, don't properly secure exploits, don't understand the inner working of their COTS control panel, etc.

* Look up the history of Lxadmin. VERY popular control panel package, company behind it, but most coding/knowledge behind the CEO. Big exploit found, 100k servers exploited, CEO commits suicide. Company is dangling. Believe licensing servers went offline. All providers using them were screwed. They were using COTS software, didn't know their own stack, left scrambling.

Stick with providers that are a real company. Employees of their own, support staff of their own, own their own equipment, etc.

Linode is great, EC2 is great, and there are several other providers.

My first experience with the VPS market was hilarious. IRC support... which should have been the sign to run away. Went to IRC, kick banned because they thought I was some other user that came in under another IP address to avoid a ban when I went to ask for help.

Run away.

> Linode is great, EC2 is great, and there are several other providers.

What are some of the others?

http://www.lowendbox.com/blog/top-low-end-vps-providers-2012...

http://www.lowendtalk.com/discussion/5136/best-leb-host-q3-2...

Why do a lot of these providers differentiate between OpenVZ and KVM? Usually there's a significant pricing difference, but little in the way of explanation as to why.

KVM is stronger isolation, its a hardware layer virtualization. You can (though the provider might not let you) run a custom kernel, etc.

OpenVZ is more OS virtualization, there is less isolation between instances.

OpenVZ usually has "guaranteed CPU/RAM" and "burst CPU/RAM", while KVM always gives you what its assigned. The guaranteed/burst is typically BS. More often these budget providers like OpenVZ because you get greater density, which means more noisy neighbor and taxed resources like disk IO.

So basically OpenVZ is what gave VPS nodes a bad name when they came on the market.

Are you asking for a list as a counter argument? A list of ones to look at? Or a list of companies that match my personal criteria?

Rackspace, HP Cloud, VPS.NET, Gandi, RIMU, Slicehost (back in the day), Webbynode, Liquidweb, StormOnDemand (though a part of Liquidweb), Burst.net, PowerVPS, FDC Servers, ...

List could go on. I usually look for whether they have an established history, a site that doesn't look like a crappy overused template, a real physical address, WHOIS info on the domain, Google them and research, whether they have a real status site, maybe lookup if they have an ARIN number, traceroute to their site, etc.

>Are you asking for a list as a counter argument? A list of ones to look at? Or a list of companies that match my personal criteria?

The last two.

Not to mention Linode is the only VPS provider I've seen (and I've seen quite a few) to offer Slackware boxes. For someone who grew up using Slack, this is a huge plus.

How do you get around the IO issue? We've been looking at EC2 for, well, forever, but IO performance has always been a huge issue -- to the point where it'd cost more than hosting it ourselves just to scale IO performance to the point where it'd equal a couple self-hosted servers.

Not all services are I/O bound. Hell, it's routine in the modern world to have datasets for real world products that fit in, what, $20 of RAM?

Which is exactly the case in our environment. As soon as all libraries are loaded from EBS IO is not that much of an issue any more

We store all the code and everything that changes in a Ram disk, so really fast.

The virtual machine lies on EBS, but it is really fast to start and performance is not limited by EBS at all in our case.

when i last used ec2, it was almost exclusively for async backend processes. in those cases, high throughput is important, but latency is not a factor in the same way. If there's a 300ms delay talking to the system from the outside world, it's no big deal.

that said, netflix seems to have solved the IO issue with ec2. I'd be curious to know more about their findings.

Main way we solve IO issues: We don't rely on EBS -- most data is stored on the local drives using Cassandra.

For cache throughput, our cache library adds zone affinity to memcache, so the clients are hitting the cache in the same zone first.

Also, we're now exploring their SSD options (http://techblog.netflix.com/2012/07/benchmarking-high-perfor...)

As it hasn't been mentioned yet, Amazon released EBS Optimized and Provisioned IOPS a few months ago that vastly improves EBS throughput.

http://aws.typepad.com/aws/2012/08/fast-forward-provisioned-...

Their new SSD-based instances are supposed to be a lot faster, but has anyone benchmarked these?

My experience with EBS was the the performance was decent most of the time, but would brown out intermittently for reasons beyond your control.

Netflix managed to discover that the new instances cut server costs in half.

http://techblog.netflix.com/2012/07/benchmarking-high-perfor...

Keep in mind this is purely research and it may (or may not) have already been implemented as part of their production systems.

What networking issues did you experience with Hetzner?

Builds failed from time to time because the network was simply gone. We worked around the issues by running commands again when they timed out, but it was one of the reasons (the main reason being that having your own servers is simply not flexible enough) for switching from Hetzner

Network partitioning is also a common occurrence on AWS, and people have to plan for that.

Of course, but it happened with a regularity that just didn't work out for us any more. Although it was definitely not the main issue to change.

So far though we haven't seen any of those problems on EC2 (or at least only very very very few instances)

Linode was out of the picture fast, as they do not provide per hour pricing.

I thought that Linode is competitively priced - is that a wrong assumption? (What is the pricing of Amazon's EC2 per year compared to Linode's?)

Linode is very competitively priced, but we need to scale up/down on an hourly basis. when there are less builds to run we simply want less servers to run.

Linode is much cheaper than Amazon when you go over the whole year, but it just doesn't fit our needs.

Linode is cheaper than Amazon, but then every service that charges per-month (not per-hour) is cheaper than Amazon.

CPU time on Linode is cheaper than standard-priced EC2 instances. Compared to a reserved instance they lose badly, though. And last I checked, storage on Linode was 8x higher than EBS.

Linode's storage is prohibitively expensive compared to EBS and S3. Their instance sizes are all based on a $1/GB flat rate with no volume discounts.

If anything, this has to be slashed in half somehow. It's been a few years since this has been adjusted.

OK, I didn't remember storage being that bad. Also, reserved instances are billed per-year, so my per-hour comparison stands.

Linode charges up-front for the month, but if you terminate an instance early a pro-rated amount is credited to your account. The granularity is at least a day.

Did you consider Reserved Instances?

As far as I am concerned, Reserved Instances are the way to go, if you consider to get to be be a heavy user. I opened a spreadsheet recently on my own startup idea, and with Reserved Instances Amazon EC2 got very competitive.

Reserved Instances are the next thing we are looking into as well to get our costs down dramatically

Recently stumbled upon very competitively priced DigitalOcean: https://www.digitalocean.com/price-comparison-chart

They have APIs to manage on-demand instances as well.

Digital Ocean looks pretty sweet, but the advantages by going with EC2 for us are a large number of different Instance Types we can use for every task we have.

It provides us with a lot of flexibility on how to build our service.

That is true. Amazon has been in cloud hosting space for a long time to cover almost all areas required to run a server. I am rooting for competitors though who could offer more services. That should bring interesting services and drive down cost even further.

Thank you for the link! Their offering looks pretty solid; they are apparently using SSDs for all disk space which is one of the first I've seen doing this while using an 'on-demand' pricing model that's competitive with Linode.

Some key things I saw when I briefly checked it out:

-Their answers [1] regarding bandwidth bursting and fair bandwidth usage / sharing policies [2] is slightly ambiguous, or worrisome at least IMO, although maybe I'm just a cynic heh.

-They do include CPU bursting [3] similar to Linode.

-No private IP addresses [4]. No mention of IPv6 either but perhaps they do support it? =(

-Their 'Features' page looks great but I can't find more specific information & details on some of the things mentioned such as: A) US & Europe data center; is casually mentioned as being New York & Amsterdam only though. B) The ability to upgrade a 'Droplet' VM to a dedicated instance that can use all the host resources. C) No mention of VM resize speed / more info on the 'private network' aspects.

[1] = https://www.digitalocean.com/community/questions/do-you-limi...

[2] = https://www.digitalocean.com/community/questions/do-you-have...

[3] = https://www.digitalocean.com/community/questions/how-does-cp...

[4] = https://www.digitalocean.com/community/questions/do-you-offe...

Does anyone have any first-hand experience with DigitalOcean that could provide some insight?

I'll probably give it a whirl in the next few days as it looks to be a solid offering; would definitely be interested if anyone that may have gone with them already though!

As nice as their offering looks, their TOS prohibits hosting content that is "pornographic, obscene, fraudulent, or discriminatory, including any containing nudity, erotica, profanity, or obscenities" which is extremely restrictive. If you host a website that allows user content, someone posting the word "shit" or posting nude art photography could get your account closed. No thanks.

They'll refund unused time.

Dont forget www.atlantic.net/cloud fast io competitive price per sec billing API etc

Reply on news.ycombinator.com