Hacker News Reader

Comment by nashequilibrium | Hacker News Reader

Is the AWS cost so high because they primarily stored pics? If so what can be done to reduce this cost in terms how to store the pics?

View on HN

I'm currently someplace where we are treating AWS as a VM provider and it's freaking nuts how much it costs. (We might be looking at the problem wrong or failing to appreciate the fact that we can always scale with enough money.)

AWS excels at scalability. There are certainly cheaper ways to have a bunch of VMs running 24/7.

If you have a fixed workload (i.e. X servers always online) then take a look at reserved instances. The break even point is about 6 months for the light instances and a bit more for the medium ones. I don't recommend the heavy ones as the cost structure is different (you pay regardless of whether tt's on so ... if you later change your instance types you still have to pay hourly for the full term of the reserve).

Even better is if you can plan your infrastructure on AWS around using spot instances. They can be really cheap (we're talking a 5x times cheaper then on-demand and 3-4x than reserved). If your instances are used in a stateless fashion with all persistent state saved externally (DB, S3, etc) then you can do some pretty cheap scaling with spot instances.

One setup I've played with is a core set of non-spot instances phalanxed by a number of spot instances (at a couple different price points) for stateless web traffic. As long as the spot price stays below your bid you have significantly more instances available (which should give your users better response times). When the spot price rises your spot instances die and things slow down, but your app would still be alive thanks to the non spot instances. Again it takes quite a bit more engineering to get a setup like this but this is the kind of thing you need to do to take advantage of elastic computing.

At Everpix, we were already extensively using spot instances for the vast majority of our EC2 needs (which were pretty intense considering the type of computation and image science we were doing). That did save quite a bit of money when we did the switch and it was easier to do than we expected, so strongly recommended!

Did you guys look at bare metal at all? (either dedicated or colocated)? Spot gets you in the same ballpark of price on a "unit", which can be mentally deceiving since an AWS unit is pretty weak.

In our experience with CPU or IO intensive workloads, Spots are still 2-5x slower than what you can get at decent dedicated provider..

No, there was no time nor did we have the in-team experience. EC2 is one thing, which you can get down to pretty cheap if leveraging spot instances, but most of our infrastructure costs was on storage.

The break even point is about 6 months for the light instances and a bit more for the medium ones.

Actually I ran the numbers for some of our (higher end ec2 instances). And some of them you break even in 6 weeks

We had already very efficient image optimization for full-resolution photos using 4X to 5X less space than competitors. Without that our AWS bill would have been insane. So as far as it goes for pure storage on S3, we were as low as possible.

However, on the database side of things, there was a lot of room for optimization (and we only got to some of it).

You could move some of the pics to your own server, leaving AWS to handle growth in storage only. The raw costs could be very significant, though most likely you would need another person to create that system make it fast, which in SF probably means losing all the gains at that scale.

In rough order of difficulty/startup cost:

- AWS reserved instances (doesn't help with storage costs though)

- dedicated servers at e.g. SoftLayer

- colocation

There are _a lot_ of dedicated options. In terms of price, Softlayer's at the high end. If Softlayer gives you a price/performace improvement over AWS of 2.5x, you can get another 2-3x that by picking a cheap (but not necessarily bad!) provider.

Please provide links to these providers you speak of.

On the cheap end, you're looking at OVH and Hetzner. They've both been reliable for me, but I still wouldn't run anything mission critical on them (from anecdotes). Hetzner is Europe only, while OVH does have a DC in Quebec. If you're already using spot instances, which can vanish pretty quickly, you might have the code in place to handle these potentially flaky providers.

Over the years, I've had good experience with WebNX which has good connectivity to Asia. They've been supposed to open a NY location for a while now...Speaking of NY, there's reliablesite.net which is ok. You have Hivelocity in Florida (which I didn't use specifically for South America, but I imagine they'd be good for that).

There's a handful of companies which I've never used but that I've been hearing good things about for years. NetDepot, SingleHop and LimeLight Networks for example.

I have used 100TB in both Singapore and Washington. They are a Softlayer reseller. The extra level of indirection sucks when something goes bad, but they are much cheaper (what you can get from 100TB for $400 might cost $1200 at Softlayer). A few of the locations they actually have staff at, so I imagine that's much better.

I recommend you spend some time on webhostingtalk.com. It's great. There's a forum specifically for deals which can be useful to pick something up.

For anyone who's only familiar with AWS/clouds: most of these take 1-2 days to set up machines, their control panels are horrible, they don't have an API and you might not even get a dedicated KVM switch.

Wow, I feel like a huge advertising billboard this morning!

Hi. Do you know why the above "cheap" services do not user something like OpenStack to bundle their services? This would supposedly provide them at least with some dashboards and API's. thanks.

I can only guess.

Many of these are older than OpenStack by a wide margin. OVH in particular is huge, they own many data centers, thousands of employees...you just don't make that sort of switch overnight. The cost and risk might not be justifiable.

Reply on news.ycombinator.com