The Evolution of Container Usage at Netflix
techblog.netflix.com
techblog.netflix.com
In regard to the analytics, they have 100 million subscribers. Let's say each subscriber watches an average of 100 episodes/movies a day. For each watch you record subscriber ID, movie ID, start time, stop time and get 32 bytes * 100 * 100,000,000 = 320 gigabytes of data per day total. I am pretty sure that you could get a commercial database and and business intelligence package that could support the type of analytics you need (mainly clustering analysis) at that scale. A national grocery chain, probably has a similar amount of data ingestion and a similar analytics need. In addition, I have subscribed to multiple Netflix type services and I have never weighed the quality of suggestions very high, giving much more weight to the functionality of the client, lack of ads, and large catalog of good movies.
As evidence that this is a solved engineering problem, just look at the number of similar movie services: Amazon, Google play, Hulu, PlayStation Vue, Pureflix, Crackle, etc.
Google, Facebook, Baidu, Amazon, the self driving car companies are doing cutting edge stuff in terms of scalability and analysis, but not Netflix. The complexity of their operations, seems mainly to be one of their own doing and not intrinsic to the service they provide.
So I look at stuff like the article here and see a bunch of very smart engineers who are bored with the (solved) core problem and spend their time making cool stuff which is actually a pretty good thing.
Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing to shuttle 5GB of data with no hiccups for an hour. Detecting and routing around machine or network issues fast enough that real-time video playback is not impeded sounds to me like a difficult challenge.
I am sure a large part Netflix's success is how seamless its products are: there's minimal to no loading time and the video stream is virtually always smooth - including during times of action/high frame rate.
Creating a video streaming platform is one thing, but creating a content delivery platform that is effectively as reliable as TV that is also cost efficient is no small task if you ask me...
At this point it's a bit like arguing that Disney's great success in Bluray sales comes from their encoding/Bluray mastering expertise.
There is however something about that last 5-10% where moving a few bits around the right way gives you business advantage, over places like Disney.
That said, it does seem like some of this Netflix stuff is overbuilt, although I don't need to run tens of 1000's of ec2, so maybe it's worth it.
Speaking of which ...
I notice that while youtube continues to buffer video while paused, netflix (and many, many other video players online) do not.
So while netflix will auto-adjust quality for you in response to a bad net connection, you can't just pause it, go brush your teeth, and come back with a big enough buffer to avoid skips through the entire video.
When did video players stop buffering-on-pause ? Why was that choice made ?
On the other end of the spectrum is whatever video player showtime online uses - it does not adjust quality and it does not buffer on pause. Basically it was built for perfect Internet connections and nothing else.
Not anymore, or at least not for long periods of time.
I would imagine there is not. When you master something for 1%, you do 20% without too much hassle. (Let's remember that netflix is only serving a couple of videos to a lot of customers, that is embarrassingly parallel and cachable).
About reliability. Video is one of the thing that has the least need for reliability. You can have thousands of hiccup during the hours, doesn't matter, nooone can see any of them.
I expect most of the tech challenges to be in cost optimizations, not in delivering the service.
Also if you've ever tried a TV channel's streaming service you'll realize that recording the start and end time won't tell you much about the quality of the experience.
Classic HN hubris.
For example, Netflix dynamically adjusts video quality on demand. Pretty easy feature. The top HN comment will explain how they would implement that in five minutes -- hypothetically, of course!
It is entirely possible and reasonable to prototype a streaming service serving 20% of the us traffic with a single machine and mongodb.
Well, a single machine, mongodb AND akamai.
What's important is really the distribution and caching layer, and Akamai has been offering that for a very long time for all the big internet companies.
Those other guys may also be offering streaming, but how quickly can they adapt?
If we make a scale from 1 to 4 out of those four panels, I'd say Google is around a 2.5 and Netflix is somewhere above 3. It's tough to assign numbers much past 3 because the technology becomes indistinguishable from magic pretty quickly.
- Applications for all devices Netflix is on (TVs, Phones, Displays, Roku, Apple TV, etc.) and all the support around it.
- Recommendation and analytical software within Netflix. Not even accounting for other areas of the company where ML can be applied.
- Billing and financial software for Netflix users and partners
- Proprietary/internal customer support software
- Content management system, along with all the international and legal challenges around that.
- Any/all software used by their marketing teams
* Complexity multiplier (variations of contracts, catalogues, infrastructure, language, currency, payment methods, taxation, government regulation, corporate governance, market structure, customer preferences) of offering service in >100 countries.
The suggestion that building a service like Netflix is a "solved problem" is naive to the point of idiocy.
You can still watch netflix on an original Wii. That hasn't gotten any OS or software patches since like 2011. That's a long tail of legacy clients to have to deal with.
Sure, you don't want to publicize things which could be a competitive advantage to your company, but I bet these banks of yours wish weren't sitting on huge heavy custom-rolled COBOL codebases right now...
I think you're missing a quantifier of some sort. Perhaps "consumer-facing"? Of any AWS customer, Netflix is far and away the leader in understanding and making use of AWS. It pushes the limits more often than not.
Their scalability, failure/recovery handling and (assumed) cost-effective deployments are _the_ standard in AWS and cloud computing. They even have an open-source toolset[0] for testing your infrastructure against failures and recovery that's quickly being picked up by several other companies to harden their infrastructure.
Then you realize that regularly, throughout the day, they're intentionally bringing their own infrastructure down and they can still operate with nary a hiccup. I wouldn't call that over-engineered.
It's off-topic, but having interacted with national-level grocery chains before, I think you grossly over-estimate their technological capacity and savvy. They're probably one of the slowest industries to adopt new technological advancements, for better or worse.
This is exactly what they do. The video serving part isn't that complicated, it's the rest of the site and APIs.
Although those are a tad over engineered considering the final output is often a slow and unwieldy UX.
I can't wait until this becomes buzzword de jour and startups start using it in their product descriptions. Then someone needs to start talking about products to "enable developer innovation acceleration" to outpace these crufty companies stuck at 25 kph.
...The scary thing is I think that makes sense, actually. I can understand that sentence as a thing I would want - Twiddle with your company culture so that individuals come up with (and make) new ideas, in such a way that as time goes on, their ability to do that grows...?
Makes sense to go at it from a systems view (holistic), rather than components view.
How about -
Homeopathic synergies for creative empowerment and evolution?
That sounds properly almost, but not actually, sensical.
I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing fast IO speed of containers on bare metal.
Doing a little research: GKE runs on https://cloud.google.com/container-optimized-os/docs/ which is designed for GCE, which runs on KVM: https://cloud.google.com/compute/docs/faq
So it looks like just Triton and Red Shift.
Disclaimer: I used to work at Red Hat, which means I like the people behind Red Shift, but that also means I hate the people behind Triton as part of the early 2000s Linux/Solaris wars.
I'm 99% sure that Google runs one VM per container because that's the only way to make it safe.
Anything else would be insane.
Hypervisors are much, much harder to break out of than a Linux container.
I'm 100% sure you are wrong. You might as well just use VMs.
Containers are not only about safety, you know.
Not on AWS.
If you are running a webapp and a database with a known workload, not so much.
That one piece of lazy coding held back Javascript on their platform for years, and I'd argue that it as the most significant cause of their browser's market share plummeting (or at last tied with lack of adblockers).
With containers on top of VMs, you now have to manage which containers run on which VMs (and the cloud provider worries about physical boxes), with a pure container solution you just spawn containers (and the cloud provider worries about physical boxes)
They might not be deploying containers on VMs on bare metal because they want to, but because they /have/ to.
I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.
It might add another 1% overhead for most tasks to run containers on HVM virtual machines.
The one giant exception is network performance. The network is usually virtualized at the VM level, unless you have an "enhanced networking" VM with SR-IOV enabled. For containers it's virtualized a second time.
This makes the combo potentially terrible if you're trying to run high bandwidth stuff on low performance VM's.
I still like the combo because it allows you to give the big ole finger to AWS if they try to lock you in one day. Since your containers are isolated from the VM you can easily spin them up pretty much anywhere else.
- Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC.
- There is no overhead for running processes in containers.
- There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host's network capabilities. Otherwise it is typically a combination of bridges and overlay networks for which the benefits outweigh the performance concerns for most workloads.
The overhead for running containers is usually very low but real. The OS needs to partition low level resources that are normally shared and the scheduling introduces some overhead.
I disagree about network performance. The virtualization adds a somewhat small but non-trivial overhead here (the overhead for other stuff could probably be considered trivial)
Here is a paper I dug up on that gives results to back up my ranting. It's a bit old now but probably still holds mostly true. http://domino.research.ibm.com/library/cyberdig.nsf/papers/0...
My point about network virtualization is that it is not required to use linux containers. Yes, some container tools do create network abstractions that add overhead, but they aren't required and most tools allow you to optionally bypass the abstraction and sit directly on the host's network stack.
But I don't see how that would be any worse on EC2 compared to bare metal or any other hypervisor/provider.
Maybe I'm just interpreting too much into the OPs wording and he did not mean that it's a specific EC2 issue.
Now, Netflix, being a priority customer, may get higher limits and such. But average joe public cloud user should keep that in mind before trying to use EC2 for running containers.
Similarly, do you have any data showing that a container system has such incredible overhead compared to the actual application workload? I mean, if that was true you'd think the entire Kubernetes team would be staying up nights figuring out how to reduce overhead.
See https://blog.openshift.com/securing-dockers-future-with-seli...
But people keep selling SELinux or AppArmor as a solution for multi-tenant container environments, which is just plain false.
The real solution are efforts like like Intel's Clear Containers and Hyper's runV.
Tech has too many edgy code names.
BTW, their opening positions are always prefixed with "senior" title but I guess that makes sense; Netflix builds pretty much everything from scratch under time constraint.
I think this reflects poorly on them, just as much as companies that use OSS but don't contribute anything back.
Seriously, if a company only wants to hire experienced folk, then so be it as long as they don't exercise REAL discrimination.
It's also interesting considering a quote I remember from Google about their preference for junior engineers..."fewer bad habits to break."
It feels kinda weird to say that, but then I see what our DNS servers are doing, or I learn about some customers that struggle to consume their own logs due to the speed at which we produce them, and things that you think are not an area of concern become that... when you have enough traffic flowing through your systems.
Pretty much the only thing that has really stood out is Kafka. Kafka does work at scale.
So, making an off the shelf product that actually does scale to the top 5% rarely makes sense, since, at best, the customer base will be limited. In the average case, though, the customer base will be almost nil, because of the per-site quirks and customization that are always present.
I experienced this on a weekly basis in the late 1990s and early 2000s at WalMart, where we were centrally managing a network with well over 10 million nodes.
Every vendor ever constantly tried to get in with us, so we had to come up with a triage system or an enormous portion of our time would be spent evaluating.
Even those best of breed would almost never work for us, because of our unprecedented scale (at the time).
In the rare case where an off the shelf product was selected and successful, it had to be heavily customized.
I am of course talking more about 'framework' kinds of tech, or, I guess, things that work at scale. A lot of off the shelf tech was and is used that isn't scale related.
If you don't mind me asking, what kind of position are you looking at in the UK next week?
Yes Teradata were awesome at the time. Their secret sauce was the hashed index stuff,which I think was the source of their major patents.
Our PM was not amused when I pointed out that, although hashed indexes were awesome, they did nothing when you were doing a wildcard search at which point you're doing a table space scan. Nobody has thought about this wrinkle...
Re: Teradata: I sat next to that group for half a year, but I had no other direct exposure to it. I did what one might call 'devops' for Network Engineering.
The other thing is 'early optimization' when you know millions of users are going to use something, you have to build things right to begin with.
Is this the core Netflix API ? have they moved from java - previously, their entire open source contributions were around java (https://netflix.github.io/). Hystrix repo was updated barely a day ago.
For me, this is more interesting than the VM part.
Hystrix is important and won't be going away any time soon.
More info: https://www.slideshare.net/mobile/KatharinaProbst/the-new-ne...
Disclaimer: I'm on paternity leave and not on those teams, but we've talked publicly about this stuff recently..
Curious what their 'own isolation technology' does that docker doesn't.
Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top of marathon and sends it some sort of recommendations for scheduling. I need to find a good example of what this is actually doing.
Probably related to that "and security" part. That isn't currently docker's strong suit.
"Apache Mesos frameworks match and assign resources to pending tasks. Fenzo presents a plugin-based, Java library that facilitates scheduling resources to tasks by using a variety of possible scheduling objectives, such as bin packing, balancing across resource abstractions (such as AWS availability zones or data center racks), resource affinity, and task locality." [1]
Marathon is a Mesos framework for scheduling long running applications (like rest services) and keeping them running. If what you want to do is serve http traffic then Marathon does the job (although the stand-alone UI is now deprecated and will only bug-fixed for "the next few months", so you'd better like the full DC/OS offering).
Titus appears to combine the functionality of Marathon, plus the ability to run batch jobs. I wondered if Titus was a fork of Marathon with new bits, but that doesn't appear to be the case. I believe it deals with one glaring flaw in Mesos, which is that frameworks all independently calculate the best use of the offered resources. When compute becomes available, Mesos makes offers to frameworks, with some basic logic such as making offers to frameworks currently consuming the least. But that means that there is no way to customize the resource allocation across different use cases (e.g. between REST APIs, one-off tasks and Spark clusters). It'd be great if Fenzo did "sit on top of Marathon", so I could customize how it schedules based on the "bigger picture". Titus avoids the problem because it schedules everything.
This space is getting busy in the last few years..
Overlay network technologies (flannel, weave, calico, etc) are popular but they aren't mandatory. You can implement it using hardware switches and VLANs if you wish.
This. So much this.
-------------------
We run a peak of 500 r3.8xl instances in support of our batch users. That represents 16,000 cores of compute with 120 TB of memory. We also added support for GPUs as a resource type using p2.8xl instances to power deep learning with neural nets and mini-batch.
In the early part of 2017, our stream-processing-as-a-service team decided to leverage Titus to enable simpler and faster cluster management for their Flink based system. This usage has resulted in over 10,000 service job containers that are long running and re-deployed as stream processing jobs are changed. These and other services use thousands of m4.4xl instances.
While the above use cases are critical to our business, issues with these containers do not impact Netflix customers immediately. That has changed as Titus containers recently started running services that satisfy Netflix customer requests.
I get that they are bragging about their implementation, but what number for peak batch processing instances would they be embarrassed to divulge?
A container is just a process. Really no different than any other. So cache and shared libraries, etc all the same with just a little care.
$ curl http://techblog.netflix.com/2017/04/the-evolution-of-container-usage-at.html
<!DOCTYPE html>
<html><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
<title>Access Denied</title>
<style type="text/css">body {margin:0;font-family:verdana,sans-serif;} h1 {margin:0;padding:12px 25px;background-color:#343434;color:#ddd} p {margin:12px 25px;} strong {color:#E0042D;}</style>
</head>
<body>
<h1>Access Denied</h1>
<p>
<strong>You are attempting to access a forbidden site.</strong><br/><br/>
Consult your system administrator for details.
</p>
</body>There are also kubernetes and mesosphere to do similar thing, but they are harder to use, you can't learn them over a week end.
Netflix runs their OpenConnect appliances at every big ISP that will let them put one there:
https://openconnect.netflix.com/en/
They run Freebsd, and all of the web services (thousands of them) run in AWS on Linux.
https://www.rapidtvnews.com/2016031942170/netflix-moves-all-...
They use AWS for video transcoding) (see http://techblog.netflix.com/2015/12/high-quality-video-encod...) and probably other activities (analytics, business)