Hacker News Reader

You should talk to people who have done this IRL, not on HN. This is a difficult thing to pull off, and like any complex decision, there's a lot of trade-offs. Without knowing your specific situation, it's fair to say that volume helps out Netflix a lot, and you won't get anything close to the unit costs they have.

The major thing that sways people to one side or the other is the kind of problems they want to build an organization to cope with. Do you want to build an engineering team that can handle a whole availability zone in Amazon going out, or do you want to hire one that eeks performance out of bare metal? Do you want a sales organization that has to go to the mat over security details, or a marketing organization that has to cope with justifying why "you're not a cloud company, you don't use the cloud" in their brand? There's going to be bumps either way.

Source: I used to sell monitoring software, so I got to talk to a lot of people making this decision. We tended to find big opps when people made the move.

As someone who works on a far smaller video platform than Netflix commands, I can definitely say that AWS has given us plenty of freedom in places that planning our tiny slice of a DC has not. We dabble with dedicated hardware, but it takes so much more planning and budgeting - mostly of human resources. The budget we stick to is barely monetary, as it eventually gets quite a bit cheaper to run things (especially video transcoding) upon metal, but hours and mindspace are far more costly. The things we have started moving to metal are tested on the cushy and forviging pillow that is "AWS". We build and monitor there long before we consider moving the services to our own hardware and that conversation includes total cost, including those that are sunk into the massively larger upfront investment (buying / maintaining servers).

That said, we try not to build our system _within_ the world of AWS, but rather "on top" of it. Our backend software is written to run on any recent ubuntu distro, and moving elsewhere, whether it be another cloud platform or dedicated hardware, is more of an Operational issue than something requiring significant changes in our soft machinery. For instance, we stick with our own messaging services rather than SQS as it's more portable. And our databsaes are RDS (MySQL). I assume this is why AWS sticks to well-used protocols for its larger services, because it allows us to use their services with an open mind and not worry as much about vendor lock-in. This gives us room to grow and worry about the expense when the expense becomes a valid issue.

We're allowed to build on their platform without worrying about whether we can move on to another, provided we keep that in mind. As it stands, we're almost completely on AWS, but I run our entire platform on my [ubuntu] desktop without issue. The Ops are different, as I'm installing our software and dependencies manually, but the software itself can live on any server. And even as we consider dedicated servers, we keep in mind that we may want to continue using AWS for scaling during spikes, and so we continue to write our software and manage our systems accordingly - keeping in mind the idea that it should run on any system that meets our minimum requirements, whether it be bare metal, managed, or in the "cloud".

This depends on your current infrastructure, your expected growth over the next few years and obviously your software stack. In my experience, unless you are doing something with extreme latency or scaling requirements, AWS is likely cheaper in TCO, at least in the beginning where your fleet is less than a few hundreds machines. Also note that AWS is not just EC2. Other services like S3 or Dynamo can be life savers. The downside of AWS is that you can become over reliant of it.

Another perspective; location and staffing.

Last first: Getting enough competent people to whatever you have. Just getting people that knows the difference between rm -rf / and rm -rf ./, not to plug both ends of a cable into the same switch/router, ... can be quite hard. If you have a premium product running on a few hundred machines, AWS doesn't actually eat that much margin compared to having your own 24/7 staff & all that entails.

And when the company then decides to open up "somewhere over in that big asian market", you've got to do it all over again. You can get a lot done with "remote hands" and whatnot, but again - something that eats your margin, just as AWS does...

you still need ops guys just not hardware guys. for a smaller company it would be the same guy anyway no? a company with 10 or 20 servers doesn't need a dedicated hardware tech.

If you watch any of Cockcroft's presentations you can see that there's nothing easy about it. They developed a significant amount of tooling just to work around quirks in EC2, but apparently the elasticity is worth it.

Assuming Amazon can do it slightly cheaper than Netflix could (i.e. Netflix doesn't have to duplicate Amazon's fixed costs, such as engineering), and that Netflix has enough bargaining power to pay Amazon very little over what it costs them to provide it....seems like an economic decision to me.

It depends on your IT architecture and more importantly on your software architecture. I did this IRL and I did it both ways and I'm doing it again. If you want some ideas or guidance or want to ask specific questions - feel free. My email is in my profile, "about" section.

The question is, do you need as much elasticity as Netflix does?

Not like in "I had a surge in traffic of millions of visitors"

But yes like in "Some hardware failed, but yeah that's not my problem anymore!"

Sorry, I don't get it - why would you consider using cloud servers if you don't have significant spikes in your traffic?

Regarding hardware failures - it's not that hard to arrange proper redundancies in your systems, and it will certainly be cheaper than outsourcing it to Amazon. Cloud only becomes cost effective when your spikes are several times larger than your normal traffic, and you must maintain good QoS at all times.

Because sometimes cloud servers are very cost effective, easy, and fast to deploy: Digital Ocean for example.

$40 / month for 4gb of ram, 4tb of transfer, and two cores, is very cost effective. Linode is also very reasonable.

If all I need is that scale of computing, what should I use instead that makes sense other than cloud services? Setting up colo or a dedicated box for that would be more expensive and a much larger hassle.

Arguably, Digital Ocean is more like a regular hosting provider with better onboarding. Plenty of hosters offer comparable infrastructure for the same price (you can even get something dedicated).

They're also far cheaper than AWS, where you pay a premium for the elasticity and access to the ecosystem. If there's anything to compare AWS with, it would be Google Cloud Platform, Azure and maybe Rackspace.

Digital Ocean is comparable to EC2. But AWS does LOTS more.

Well, I admit, those prices are pretty good. I looked at AWS pricing 3 years ago at a SaaS company with steadily growing traffic, and at the time it didn't make sense. It might now.

I worked at a company where the whole server room was flooded. They had to replace everything and restore stuff from backups.

Two weeks later, they had to do it all over again because something went wrong with the air conditioning and the servers got cooked.

I kept recommending going with AWS because of that... I no longer work there, but as far as I know they're still running their own servers.

Why AWS and not just co-locating the servers somewhere?

I wasn't dead set on AWS, I was just using it as an example. Any sufficiently large company would have worked.

>it's not that hard to arrange proper redundancies in your systems

If it really were like 'apt-get install redundancy' I'm pretty sure that AWS would have 1/10th the clients it has now.

At work, we use a dedicated server provider, so we're; when we identify faulty hardware, we file a ticket, and they swap it out. We still need to have enough bare metal machines allocated so we can turn one off while they fix it, etc; but we don't worry about staffing or supplying the datacenter.

I wouldn't want to switch to AWS from this; bare metal plus a good account manager gives us a lot of visibility and reliability; Netflix blogs a lot about the chaos that is AWS, and we save a lot of effort by having machines that for the msot part just work, consistently; instead of having to deal with instances where performance is variable, etc. Running our own datacenter might be nice for some reasons: more control and information about the network, ability to make hardware choices that don't match the providers offerings, etc; but running a datacenter is a big change.

Is elasticity really an advantage of EC2? Dedicated is so much cheaper that you can just overprovision massively and still come out cheaper than EC2.

Disregarding the fact that the E stands for Elasticity...

Yes, Elasticity is THE advantage for Netflix when it comes to using EC2. Consider the vast differences in usage they have in North America during the evening after dinner vs during the middle of the night. I can only imagine their fleet capacity goes up and down drastically every single day. This amounts to huge savings over having the bare metal.

Not just North America... but all around the world. Netflix is global, and EC2 is global.

Does it really though? The netflix stats show that at most it's a 2x saving: http://3.bp.blogspot.com/-72J2TkcP4j0/TxcdvDk58VI/AAAAAAAABA...

And EC2 is more than 2x as expensive as dedicated...

Don't get me wrong, EC2 has advantages, but it's not cost (unless Netflix is paying much less than the public prices, which is possible).

That's only for NA

That really depends on a type of workloads and requirements you have. But basically you (your team) have to be smart about how AWS or any other cloud provider is used. No cloud buzzword alone will help you avoid cost, availability issues etc otherwise. As for the "perks" i.e. discounts talk to AWS account rep.

My first thought was perhaps Amazon also got to pick up some of Netflix's employees in a deal but that seems like it would be difficult to enforce.

They likely get a good deal, but either way their content licensing costs are huge in comparison and make their Amazon bill a drop in the ocean.