Netflix to Shut Down Last Datacenter | Hacker News Reader

Netflix to Shut Down Last Datacenter

blogs.wsj.com

86 pointselectic81 comments

Do you think they have a special deal w/ Amazon? Otherwise I imagine that's a really expensive bill.

Do you think they're doing it because its cheaper when you account for things at a larger scale? Like, reduced liabilities, payrolls, CapEx, whatever? Or is it more expensive but still "easier" and they go with it?

I really REALLY would like to hear everything from everyone about this, because right now I'm facing that decision for my company. I'm not as big as Netflix obv, but it is a big deal for me. It is really attractive to have all the scale you need at will and forget about maintenance at all; even if it is more expensive I would gladly pay as much as double my operating costs if I can rest easily at night. I would even eat as much as 10%-20% of my profits if I don't have to deal with that.

But the devil is in the details, I think is relatively easier to go full-cloud than to later find out it was a mistake and try to migrate back to your private premises. So, what could be the downsides of going full-cloud just as Netflix did? Also, taking into account that I'm not a personal friend of Bezos neither another Netflix, I would be just another smallish guy and I'm pretty sure that won't get me all the perks that come with the former.

You should talk to people who have done this IRL, not on HN. This is a difficult thing to pull off, and like any complex decision, there's a lot of trade-offs. Without knowing your specific situation, it's fair to say that volume helps out Netflix a lot, and you won't get anything close to the unit costs they have.

The major thing that sways people to one side or the other is the kind of problems they want to build an organization to cope with. Do you want to build an engineering team that can handle a whole availability zone in Amazon going out, or do you want to hire one that eeks performance out of bare metal? Do you want a sales organization that has to go to the mat over security details, or a marketing organization that has to cope with justifying why "you're not a cloud company, you don't use the cloud" in their brand? There's going to be bumps either way.

Source: I used to sell monitoring software, so I got to talk to a lot of people making this decision. We tended to find big opps when people made the move.

As someone who works on a far smaller video platform than Netflix commands, I can definitely say that AWS has given us plenty of freedom in places that planning our tiny slice of a DC has not. We dabble with dedicated hardware, but it takes so much more planning and budgeting - mostly of human resources. The budget we stick to is barely monetary, as it eventually gets quite a bit cheaper to run things (especially video transcoding) upon metal, but hours and mindspace are far more costly. The things we have started moving to metal are tested on the cushy and forviging pillow that is "AWS". We build and monitor there long before we consider moving the services to our own hardware and that conversation includes total cost, including those that are sunk into the massively larger upfront investment (buying / maintaining servers).

That said, we try not to build our system _within_ the world of AWS, but rather "on top" of it. Our backend software is written to run on any recent ubuntu distro, and moving elsewhere, whether it be another cloud platform or dedicated hardware, is more of an Operational issue than something requiring significant changes in our soft machinery. For instance, we stick with our own messaging services rather than SQS as it's more portable. And our databsaes are RDS (MySQL). I assume this is why AWS sticks to well-used protocols for its larger services, because it allows us to use their services with an open mind and not worry as much about vendor lock-in. This gives us room to grow and worry about the expense when the expense becomes a valid issue.

We're allowed to build on their platform without worrying about whether we can move on to another, provided we keep that in mind. As it stands, we're almost completely on AWS, but I run our entire platform on my [ubuntu] desktop without issue. The Ops are different, as I'm installing our software and dependencies manually, but the software itself can live on any server. And even as we consider dedicated servers, we keep in mind that we may want to continue using AWS for scaling during spikes, and so we continue to write our software and manage our systems accordingly - keeping in mind the idea that it should run on any system that meets our minimum requirements, whether it be bare metal, managed, or in the "cloud".

This depends on your current infrastructure, your expected growth over the next few years and obviously your software stack. In my experience, unless you are doing something with extreme latency or scaling requirements, AWS is likely cheaper in TCO, at least in the beginning where your fleet is less than a few hundreds machines. Also note that AWS is not just EC2. Other services like S3 or Dynamo can be life savers. The downside of AWS is that you can become over reliant of it.

Another perspective; location and staffing.

Last first: Getting enough competent people to whatever you have. Just getting people that knows the difference between rm -rf / and rm -rf ./, not to plug both ends of a cable into the same switch/router, ... can be quite hard. If you have a premium product running on a few hundred machines, AWS doesn't actually eat that much margin compared to having your own 24/7 staff & all that entails.

And when the company then decides to open up "somewhere over in that big asian market", you've got to do it all over again. You can get a lot done with "remote hands" and whatnot, but again - something that eats your margin, just as AWS does...

you still need ops guys just not hardware guys. for a smaller company it would be the same guy anyway no? a company with 10 or 20 servers doesn't need a dedicated hardware tech.

If you watch any of Cockcroft's presentations you can see that there's nothing easy about it. They developed a significant amount of tooling just to work around quirks in EC2, but apparently the elasticity is worth it.

Assuming Amazon can do it slightly cheaper than Netflix could (i.e. Netflix doesn't have to duplicate Amazon's fixed costs, such as engineering), and that Netflix has enough bargaining power to pay Amazon very little over what it costs them to provide it....seems like an economic decision to me.

It depends on your IT architecture and more importantly on your software architecture. I did this IRL and I did it both ways and I'm doing it again. If you want some ideas or guidance or want to ask specific questions - feel free. My email is in my profile, "about" section.

The question is, do you need as much elasticity as Netflix does?

Is elasticity really an advantage of EC2? Dedicated is so much cheaper that you can just overprovision massively and still come out cheaper than EC2.

Disregarding the fact that the E stands for Elasticity...

Yes, Elasticity is THE advantage for Netflix when it comes to using EC2. Consider the vast differences in usage they have in North America during the evening after dinner vs during the middle of the night. I can only imagine their fleet capacity goes up and down drastically every single day. This amounts to huge savings over having the bare metal.

Does it really though? The netflix stats show that at most it's a 2x saving: http://3.bp.blogspot.com/-72J2TkcP4j0/TxcdvDk58VI/AAAAAAAABA...

And EC2 is more than 2x as expensive as dedicated...

Don't get me wrong, EC2 has advantages, but it's not cost (unless Netflix is paying much less than the public prices, which is possible).

That's only for NA

Not just North America... but all around the world. Netflix is global, and EC2 is global.

Not like in "I had a surge in traffic of millions of visitors"

But yes like in "Some hardware failed, but yeah that's not my problem anymore!"

At work, we use a dedicated server provider, so we're; when we identify faulty hardware, we file a ticket, and they swap it out. We still need to have enough bare metal machines allocated so we can turn one off while they fix it, etc; but we don't worry about staffing or supplying the datacenter.

I wouldn't want to switch to AWS from this; bare metal plus a good account manager gives us a lot of visibility and reliability; Netflix blogs a lot about the chaos that is AWS, and we save a lot of effort by having machines that for the msot part just work, consistently; instead of having to deal with instances where performance is variable, etc. Running our own datacenter might be nice for some reasons: more control and information about the network, ability to make hardware choices that don't match the providers offerings, etc; but running a datacenter is a big change.

Sorry, I don't get it - why would you consider using cloud servers if you don't have significant spikes in your traffic?

Regarding hardware failures - it's not that hard to arrange proper redundancies in your systems, and it will certainly be cheaper than outsourcing it to Amazon. Cloud only becomes cost effective when your spikes are several times larger than your normal traffic, and you must maintain good QoS at all times.

I worked at a company where the whole server room was flooded. They had to replace everything and restore stuff from backups.

Two weeks later, they had to do it all over again because something went wrong with the air conditioning and the servers got cooked.

I kept recommending going with AWS because of that... I no longer work there, but as far as I know they're still running their own servers.

Why AWS and not just co-locating the servers somewhere?

I wasn't dead set on AWS, I was just using it as an example. Any sufficiently large company would have worked.

>it's not that hard to arrange proper redundancies in your systems

If it really were like 'apt-get install redundancy' I'm pretty sure that AWS would have 1/10th the clients it has now.

Because sometimes cloud servers are very cost effective, easy, and fast to deploy: Digital Ocean for example.

$40 / month for 4gb of ram, 4tb of transfer, and two cores, is very cost effective. Linode is also very reasonable.

If all I need is that scale of computing, what should I use instead that makes sense other than cloud services? Setting up colo or a dedicated box for that would be more expensive and a much larger hassle.

Arguably, Digital Ocean is more like a regular hosting provider with better onboarding. Plenty of hosters offer comparable infrastructure for the same price (you can even get something dedicated).

They're also far cheaper than AWS, where you pay a premium for the elasticity and access to the ecosystem. If there's anything to compare AWS with, it would be Google Cloud Platform, Azure and maybe Rackspace.

Digital Ocean is comparable to EC2. But AWS does LOTS more.

Well, I admit, those prices are pretty good. I looked at AWS pricing 3 years ago at a SaaS company with steadily growing traffic, and at the time it didn't make sense. It might now.

That really depends on a type of workloads and requirements you have. But basically you (your team) have to be smart about how AWS or any other cloud provider is used. No cloud buzzword alone will help you avoid cost, availability issues etc otherwise. As for the "perks" i.e. discounts talk to AWS account rep.

My first thought was perhaps Amazon also got to pick up some of Netflix's employees in a deal but that seems like it would be difficult to enforce.

They likely get a good deal, but either way their content licensing costs are huge in comparison and make their Amazon bill a drop in the ocean.

Netflix is still running its own CDNs / using CDNs outside of Amazon.

"The company, which competes with Amazon in the video-streaming business, elected to keep control of its own content delivery network."

So Netflix is still probably doing the bulk of their traffic themselves. Application servers with AWS is probably not a huge bill versus massive video traffic.

Fascinating. I bet they are running their own CDN to prevent Amazon from having full access to their traffic volume/streaming demand. Would be huge competitive advantage for amazon streaming if they could see per-title demand.

No, it would be due to the sophisticated peering agreements - https://openconnect.netflix.com/

Or because it's cheaper.

I was wondering about that. Open Connect was significant in the disagreement between Netflix and Comcast that spurred the net neutrality debate. Unless I'm mistaken Netflix ended up agreeing to the peering fees, though after negotiating down a bit. So does using AWS for mundane traffic actually save them money by avoiding the CDN bandwidth that Netflix has to pay ISPs to use?

AFAIK Amazon charges more for bandwidth than the greediest ISP, but I admit I haven't looked at the EB/month transfer tier.

There's literally no way to operate a CDN-like service from AWS; no matter what discount you get from Amazon, you're paying per-byte instead of 95th percentile billing and settlement-free peering.

This is very strange. Amazon is in the pay per view streaming video business itself.[1] Netflix is outsourcing their entire business to a direct competitor. At what point will Amazon decide that they don't need Netflix?

Or perhaps there's an acquisition or merger coming.

[1] http://www.amazon.com/Prime-Instant-Video/b?node=2676882011

At what point will Amazon decide that they don't need Netflix?

And do what? "We'll jack up the prices if you compete with us" is not good publicity.

Besides, it's not that uncommon. For example, the next iPhone will have a CPU from Samsung.

> But many companies still run sensitive software in their data centers or in private clouds, in which a company has dedicated cloud resources from a third-party or within its own premises.

(Emphasis mine)

Ok, honest question: If I have a "private cloud" on my own premises, how is that not a data center?

> If I have a "private cloud" on my own premises, how is that not a data center?

It is, but "private cloud" on top of that implies a lot more about how you run that datacenter: using virtual machines, abstracting hardware, using APIs to manage servers and storage (ideally the same APIs that would let you just as easily run on someone else's cloud infrastructure), and otherwise trying to minimize your reliance on any particular piece of hardware.

Thanks. I was looking at it from the power, cooling, and square-footage perspective—didn't even consider the sysadmin portion—but I see the massive savings that could be had by uncoupling metal pizza boxes and one-off hardware requirements from the applications that run on them

I've never understood why so many companies are comfortable giving Amazon such leverage over their business infrastructure.

Could someone explain why this is a good move in spite of the risk that Amazon could jack their prices up in the future?

The free market is the basic guarantee Netflix is counting on.[1] If Amazon 'jacks up it's prices', Netflix will simply move to another provider. Amazon is also unlikely to make sudden moves which disadvantage its customers, as any such moves will come at a tremendous cost of reputation, and many customers will leave and never come back (; as Amazon has no power to compel customers to purchase their services). In a market, no one has leverage, everyone simply has a position and an offer.

[1] https://en.wikipedia.org/wiki/Free_market

> If Amazon 'jacks up it's prices', Netflix will simply move to another provider.

Like hell they will. The switching cost for Netflix, and really most heavy AWS users, would be extreme at this point. A good portion of the engineering org would pretty much have to dedicate themselves to the task of switching. That isn't to say other organizations can't be nimble between providers, but it would be hard for Netflix at this point.

AWS is already engaging in a commodity hardware market in order to run the DCs running. AWS fears becoming a pure extension of the same thing, and they have a decent bit of lock-in designed to their services for that reason.

I remember seeing a talk by a Netflix engineer (can't find it unfortunately) where he talked about multi-cloud strategies and how they aimed towards decoupling some of the components of his project from implementation details specific to AWS. So it's quite possible they're already slowly paving the way towards becoming a cloud-agnostic or multi-cloud platform. Not to say that it wouldn't still be a huge undertaking, but I don't think this never crossed their minds before.

Considering how many millions Netflix pays Amazon every single month, a team of highly skilled engineers is a drop in the bucket, and if switching means they can save enough, they'll do it.

"The closing of Netflix’s last data center has been seven years in the making."

Does that sound like a drop in the bucket?

Compared to the running cost of their streaming service, yes. Besides, I like to believe (yes believe, because non of us outside of Netflix would be able to verify) that they've spend that much time such that they're far less dependent on the underlying architecture.

> I've never understood why so many companies are comfortable giving Amazon such leverage over their business infrastructure.

You have to be a pretty big company in order to not let another company have leverage over your infrastructure. Hosting your networked servers in your own office? You're depending on Verizon, AT&T or Qwest, even if those are not the companies selling you your pipes.

Co-locating your servers, getting feeds from 2-3 carriers, while doing BGP with your high-end Cisco routers? You're depending on the hosting facility, such as Telehouse.

So you want to survive a blackout in one section of the US. Now you need at least two server facilities. Your remote facility needs at least one full-time tech.

Corporate infrastructure can get in the way as well. Your division of the conglomerate is in San Francisco, but another division is in New York. Perfect, your division's remote IT person can work out of that division's New York office. But the conglomerate might be structured so that is not really possible. The other division may as well be another company.

So a small company can't afford to do this. A big company can get so big that it's not feasible either. If the big company has one central IT department that all divisions use, it becomes more feasible, but how many big companies only have one centralized IT department? Even if such a creature exists - the big company buys a small company, or merges with another big company - boom, two IT departments again.

Wait... since when has AWS ever raised their prices? They've consistently lowered prices for better services... something like 50+ price reductions in the past 7 years.

Total guess on my part, but I'd imagine Netflix has locked-in long-term pricing. Of course, there's the opposite danger of the rest of the world getting cheaper...

Reminds me of how airlines have been bitten in the butt with a similar future-speculation method that they use to purchase fuel.

Except that this isn't fuel... the prices have consistently trended down for 7 years and have no indications of stopping.

If they're savvy, they'll have locked in long-term maximum pricing. Something along the lines of "prices will at most be the minimum of this discount price, or x% above the standard retail price".

I wouldn't be surprised if Netflix had some sort of price lock in for at least a few years.

From my understanding this is a simple decision: do not focus on something that isn't your core business. Amazon is doing it well, Microsoft is, the cloud is becoming more competitive every day, and they're big enough not to use the public pricing, so it's even more economical for them.

In addition to this, they only need a few "machines" (https://openconnect.netflix.com/hardware/) to have full ISP-local caches of Netflix; why bother with a full datacenter ?

Feeling dejavu on this announcement.

"By the end of next year Reed aims to be 100% on AWS and to be the biggest business entirely hosted on AWS apart from Amazon Retail."

Source: http://techblog.netflix.com/2012/12/aws-reinvent-was-awesome...

People on HN just don't understand that the Basic Hosting company won't give a scalable Network for free. And with scalable Network I mean a Physical / Virtual Network where you could connect / disconnect Nodes and connect it with arbitrary Networks. And that's basically something which costs a huge amount of money especially when you want to connect two Locations. When you need HA and don't get such a network you would need to create a Overlay Network over VPN, etc by yourself which could be quite hard to manage (even for a low amount of servers) In germany we could buy Virtual servers as low as 5-10 Euro / month however you can't buy 3 in different locations and they are connected with an private internal network for free (there aren't even many hosters that will do that, i don't think there are any outside of the saas platforms) however you could rent non virtual servers and buy additional stuff on these Hosters which adds a huge costs per month. So running on AWS / Google Cloud / Azure / etc could be really cheap. It just depends on the Things you need.

Edit: Also AWS gives you a lot of credits for just doing surveys for their new stuff. I have so many credits so that I run my private gitlab instance on AWS and it costs me nothing. I never got a credit of my previous Hoster which I used over 10 years. On AWS I just got a 200 USD Credit at the start since I missed my Free year (also only cloud providers giving you credits, the most other Hosters won't do that and they bareley give you something new on AWS. You could now run a Website for basically nothing just with AWS Lambda which costs nearly nothing.)

This may just be a semantic issue, but the term "public" seems entirely misused in this article.

For better or worse, "public cloud, in which users share the resources of a service provider" is now an accepted term of art in the industry. It doesn't really matter whether it agrees with people's commonsense understanding of the word "public".

This is 3rd article this week I've been blocked from reading. How are people continually reading and upvoting these links? Is everyone a subscriber?

Screenshot: http://i.imgur.com/FGY1lSv.png

Here you go :) https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&c...

I'm confused how this is still allowed by Google. I thought you were not allowed to show Google one thing while other users another?

You're not allowed to show google's bots a different thing than what a user sees when they click a search link.

Which is not the same as visiting the link via other methods (entering into URL bar directly, for example).

May I ask how you did that?

This "Come From Google" (CFG) paywall bypass bookmarklet usually does the trick for me:

https://umur.io/paywall-bypass-bookmarklet-come-from-google/

Thanks for the heads up on CFG - sounds useful, though it doesn't seem to work for me.

it shows the article if the refer(r)er is google

Dev tools, copied the link. :)

For me it didn't even load the ad, I just saw the "log in" at the bottom and a blank page. "who needs an article when you can have a headline?"

Same for me. It must be the ad-blocker (uBlock)

I'm on FF without adblock extension, so at least for me it wasn't. I'm in Japan however, so I'm probably completely irrelevant to the operators of that webzone. "What are you telling me, Johnny? There's people outside the US who have access to the intertubes? Getoutofhere!"

It's even more messed up with an adblocker. You don't even know what is going on http://imgur.com/K9VvuWs

I'm a bit mystified.

The link goes to an empty article but if you google the exact title, you will be taken to the full article which... has the exact same URL.

I don't know what kind of wizardry this is.

They're looking at the HTTP referer [sic]: https://en.wikipedia.org/wiki/HTTP_referer

Most newspapers don't show paywalls to users coming from a search engine, as part of a compromise with Google to prevent them from being penalized in rankings.

Not wizardry. Checking referer.

<https://en.wikipedia.org/wiki/HTTP_referer>

Indistinguishible form magic

Isn't it just referer headers?

I'm supposed to read the article before up voting?

How would you know whether the article is worth recommending if you don't read it?

probably bots upvoating

Reply on news.ycombinator.com