Netflix: What Happens When You Press Play?
highscalability.com
highscalability.com
You know old time journalists had a solution to this. Interviewing subjects with first hand knowledge.
I know I rag on highscalability a lot, but I feel like they deserve it. They've written multiple articles about topics in which I would be a first person resource (Reddit and Netflix), and not once have they ever contacted me or anyone I worked with for an interview. They've quoted me from presentations, but the quotes were out of context and wrong they way they quoted it.
All of this could have been cleared up with a single interview of anyone who is a primary source expert.
I have to start by interviewing my old co-workers! :)
In case I need to spend money, I prefer to own the thing I pay for. So Blu-ray/DVD/CDs discs for me. The gathered video/music can be played on all my devices from car to phone, even in flight-mode offline. I trade a bit of less convince upfront to really own it, but then I can play it as often and exactly when and where I want.
Anyway every few years one could get a higher resolution version, unfortunately the films aren't sold in source versions, but downscaled, but the video-on-demand at always lagging behind and stream worse bit rates.
Netflix in particular seems to have some really interesting ideas hidden behind unclear blogs(at least for me personally) and long presentations I rarely have time to sit through, even though I often appreciate them when I do.
How many failures in journalism are attributed to "lazy journalism" where nobody will even leave the building these days to even ask someone involved in the story or topic. How the internet made it "easier" but they've become so over-reliant on it that many journalists fail to include information that isn't already online.
Then again, Netflix is probably a phone call away, so that may be a whole 'nother level of lack-of-followup.
Netflix seems to be similar. Their open source record speaks for itself, and their technical blog and presence at conferences and meetups is outstanding.
Source: though these opinions are my own, I work on Infrastructure at Reddit.
I'm very sorry if I've misquoted you or quoted you out of context. I'd certainly correct them instantly if contacted. I've searched my email and I don't see any corrections, so I'm not sure how I can make amends.
I pull a lot of quotes from a lot of sources. Many thousands a year. My goal is always to highlight ideas that I think might help readers do their job better. Highlight is they key word here. That you say those sort of things often is not surprising. My hope is always readers will be made curious enough to follow the link and learn more on their own.
And unfortunately there is no "they" at HighScalability.com. There's only me and there's only ever been me. So all mistakes are mine and only mine. There's never been much money or time to be as comprehensive as I'd like.
I do the best I can, but I often fall short, no doubt. I'm not a journalist. I never said I was. I'm a programmer. Have been all my life. I started this blog a long time ago. It was just stuff I found interesting. For whatever reason it got popular for awhile so I tried to make something out of it. That, like most of the industry is petering out now, so we'll see what the future holds.
The point about interviews is a good one and one I've wrestled with. Over the years I've conducted a lot of interviews. Much less so lately, and that's for a reason. I've learned that presentations at major conferences are usually the best resources. Presenters put a huge amount of effort into making those talks. And everything they say is open and approved for public consumption by definition.
Interviews with someone like me are often a bother, a time sync, so I don't usually get as much out of them. Interviews have a high overhead, taking a lot of time to setup and there's often a lot of red tape and legal wrangling that must go on to get access and content approval. Which is again, why, presentations are such awesome resources.
This article was derived mostly from people working on a project making a major presentation or from primary sources writing an article. I couldn't do better than that.
And regardless, I would still make mistakes. Believe me. It's surprisingly hard to make sense of an interview and turn something from thread mode into document mode. That process is an error prone one because I have to fill in the gaps and there are always gaps.
It's rare that I've ever just published an interview. I don't think they are that useful in general. I want people to read a highly condensed document they can get something concrete out of with minimal effort or just ignore easily if it's not relevant to them. The strange style I've developed reflects that goal. This article was of course different because it has a different audience.
So again, I apologize. But I guarantee my mistakes are never intentional, never for money, and never for lack of caring or effort.
What industry are you referring to here?
At a previous employer, we met with AWS engineers who told us that this sort of thinking wasn't correct. Availability zones are for failure tolerance. Regions are conceptually different and for things like moving compute closer to clients, mitigating political instability or ensuring that data is held in a certain jurisdiction. To back up that claim, they told us that amazon.com ran out of a single region, us-west-2.
As an aside, another interesting point from that meeting was that autoscaling wasn't a good strategy for scaling up in the event of the loss of an availability zone. The reason why is that in those scenarios, a lot of other customers will be trying to fail over as well and autoscaling can get very slow. Instead of autoscaling, they recommended that we run 150% of our per-AZ capacity in 3 AZs so that if an AZ went offline, the remaining two would already have the necessary capacity. We still used autoscaling, but it was for responding to spikes and troughs in our own load, not failure scenarios.
I'm curious whether these two observations are correct since it seems like a lot of people view regions and autoscaling through a failure-recovery lens.
Disclaimer: while I work at Netflix, these views are not necessarily those of my employer
For your first point, I think it depends on your usage scenario. With a big company that requires good performance to satisfy their users, like Netflix, having multiple regions is a given. But what the article probably missed is that they also most likely use AZs within the same region to mitigate failures, but since they have multiple regions at hand why not use them to mitigate failures as well? And their monthly tests are a simulation of a worst case scenario if an entire region goes down.
As for autoscaling I also think that people think of it as a magic solution. More users? Autoscale. Zone down? Autoscale. But the problem arises when they think that it's instant because it isn't. Things have to happen behind the scenes, but for many people, they only care about what happens on stage, so for them configuring autoscale is just better because it's easier than load balancing on 3 different AZs.
The Netflix client/SDK was configured to contact 3 different regions. I'm guessing somehow, on the backend, it was replicating the state of your request to the other three regions. If one regions were to fail, the client would query the other two regions and regain its state.
Having 3 regions available - the other two should be able to pick up the load of losing one without hitting capacity.
Unless propagation to their database takes two years, I'm really interested in how they actually arrive at match ratings.
So your click turns into an event on a Kafka topic which is eventually consumed by Flink, joined with whatever other data sources they need and then written out to a store somewhere. IIRC the various machine learning models exist inside their Flink architecture.
https://ci.apache.org/projects/flink/flink-docs-release-1.4/...
After trying to get Spinnaker running, which is like 10 services for a fairly straightforward application, and reading about how Netflix has Cron jobs running to 'clean up' data inconsistencies... I started to think Google has a better approach to scaling the code.
Netflix is strong on eventual consistency, and making services as small as possible. Having tried to build something in that image, holy shit the issue of distributed transactions becomes a nightmare.
We had to build error handling in everything to handle the failure of everything else. Maybe half our code was 'just in case' to deal with exploding failure scenario complexity during service call fanout. If your service calls out to two or more services that also do data updates, God help you. There's no call ordering or way to make sure both succeed or fail, so if one does you could easily end up in an invalid state.
Contrast this to Google's approach. Maps is one service, Docs is one service. Their service boundaries are much larger so they can shove the complexity of consistency and rollbacks back into the DB where it belongs. And they avoid eventual consistency as much as possible.
If Google can make these big 'monoliths' work at huge scale I don't see any advantage to 'microservices', it's just a bunch of pointless overhead. I think Netflix has had a little too much industry koolaid ...
Distributed transactions are a sure sign that your service boundaries have been incorrectly drawn.
Imagine all the data Google stores with your user profile across their hundreds of application boundaries. To keep things consistent you would either need to store all profile data in one massive service or support distributed transactions.
The Netflix model is fault tolerance, with every service supporting various failed profile update scenarios. Google just decided to add distributed transactions support to spare all that overhead.
From their website - https://help.netflix.com/en/node/23742
Resolution:
Google Chrome Up to 720p on Windows, Mac, and Linux. Up to 1080p on Chrome OS. Internet Explorer up to 1080p. Microsoft Edge up to 4K. Mozilla Firefox up to 720p. Opera up to 720p. Safari up to 1080p on Mac OS X 10.10.3 or later. Streaming in 4K requires an HDCP 2.2 compliant connection to a 4K capable display, Intel's 7th generation Core CPU, and the latest Windows updates.
P.S Interesting to note that Chrome OS plays 1080p, I assume because it has HW acceleration enabled by default (in browser alone, not in android apps though).
https://caniuse.com/#feat=hevc
Not sure why Chrome doesn't support HW for it though albeit Chrome OS. My Celeron N3060 Chromebook can decode it.
I read Netflix has started using VP9 for streaming, newer CPU's have VP9 decoders in them.
[EDIT] in fact, now that I think about it, I almost never just browse the catalogue anymore, while I used to a lot. I open it and go to search ASAP, then close it if I didn't find what I want or see something else interesting in the related results. The auto-playing is annoying enough that it's almost totally driven me away from that part of the interface. I knew I didn't like it but hadn't thought about how it'd affected my behavior, but there it is. Time browsing catalogue from "lots" to "nearly none".
[EDIT EDIT] I might add that full-screen ads you have to scroll past (with no indication that's what you need to do) and other general mucking with the plain grid-layout interface make it wicked confusing for my parents. They just got a Roku solely for Netflix and if it keeps doing that I bet they won't use it very much.
On our Roku, if you sit on a movie or show for just a few seconds, Netflix starts auto-playing a video. But it's not the real show anymore, it's a weird custom-Netflix-made background advertisement for the show using random clips set to odd royalty-free stock music that is specifically not from the show/movie you are actually on and doesn't match. It kind of looks like they set some machine learning robot, or some poorly-paid Amazon Turk people, to generate these clips for much of the titles in their library.
Why? Why does this exist? What user testing or data could have happened for Netflix to decide to spend time/money/bandwidth/resources on this?
Not an earth shattering issue or anything. But it's a personal pet peeve to encounter this every time I turn on Netflix. It feels like they are trying to video-clickbait me, even after I've already paid them for everything.
I use the Netflix homepage primarily, rarely searching. I've watched those ads they play and then started watching the TV show/ movie they're for.
90% of the Google traffic through these devices are youtube videos.
They mention it in the article like Netflix pioneered this idea with open-connect. If anything it's harder to get OC devices from Netflix then it is from Google.
Even Akamai does have a similar program where they let you put their CDN on ISP NOC.
It sounds like 10:1 is a decent saving but still objectively a lot of data.
The best solution is to peer with their DC - let them do the heavy lifting and you can pull from their cache, then you can chuck the GGC nodes. Google is building a small DC close to our NOC - which means lots of saving for us and a better experience for our customers. We can see us chucking the nodes or in this case, we most likely have to give it back to them or something - they are very expensive and large supermicro servers.
10:1 is not a lot if you are serving a lot of traffic. For 50gbps, thats 5gbps bw - that's a lot of bw for free service - we don't cap youtube traffic (users likely to watch high res 1080p, 4k videos nowadays), we can't sell youtube traffic (license agreement from google) - so as far as ISPs are concerned its pissing away money. To be honest services like youtube, facebook and Netflix, even with their on-premise cache solution are a huge headache and expense for ISPs - it's not just the BW, rack-space, distribution switches, power, they all add up.
There is a reason why Google with all their expertise, knowledge, money and influence can't even make a dent in their ISP business - because even with all their money - it insanely expensive when you try to cover most of the country.
Believe me, you can. Deutsche Telekom, Germany's AT&T and Europe's biggest ISP, does this and in the evening YouTube videos regularly stutter on FullHD. But the customers just don't care enough.
Many of the things Netflix does here, Google does too... but VDNs did before YouTube was a twinkle in Chad, Steve, and Jawed’s eye.
The custom appliance proactive multi-tier video placement process outlined here, along with more techniques not mentioned (some of the most significant are only just now being rediscovered by non-CDNs such as Netflix or even AWS), were generally invented and deployed between 1998 and 2003 - 2005.
The Point is the global scale of this on-premise network and the amount of data they move. I think Netflix is a blip compare to youtube/google/Akamai/facebook (I recently learned facebook also has a similar program, but I think they also use Akamai), on a global scale. I didn't like that glossed on over this as if Netflix is doing something exceptional or even unique. Obviously, it takes some serious engineering chops scale this amount of traffic globally and routes the traffic through on-premise nodes. But at this point, it's not exceptional - if anything its commodity.
For instance: https://fortunedotcom.files.wordpress.com/2015/10/apps-by-in...
Still true in 2017: https://techcrunch.com/2017/07/05/u-s-cord-cutters-watch-mor...
Regardless, long form video streamed globally without buffering is incredibly difficult. Short form clips are drastically easier. Do the math on how much content at what quality can fit in a given cache, or how long it takes to swap a piece of content in a cache, you'll start to see why.
It's surprising to me that streaming well is still not a commodity. Even less so if you consider transactional (paid) video, which is what Netflix does.
For example, just recently, tech press talked about cable companies losing $10B a year from account sharing.
This was a problem VDNs solved in late 90s / early 00s for streaming protocols, but the technique for handling this across a global footprint was sidelined when YouTube's "http" progressive downloads / pseudo streaming and 'ad supported' video became vogue.
Now that streams need to act more like streams again, and people want to charge for the content itself again, the solution is again needed, and no http-based "commodity" CDN seems to have it sorted out. Netflix does.
PS. A couple refs from before Netflix/Google/YouTube:
- https://www.google.com/patents/US20010027491
- https://www.google.com/patents/US20010047275
- https://news.microsoft.com/2000/06/12/microsofts-new-digital...
Those were first iterations. Later iterations were more sophisticated, and not patented in order to not disclose.
Most importantly, outside the North-American/Europe echo chamber Netflix is a blip (a large blip nonetheless) compare to google/facebook/Akamai (probably even Cloudflare) traffic. I don't say this to undermine what Netflix does both as a service and a Network Engineering standpoint - I am personally a big fan of their service and their tech blog - I was trying to make a point on some info mentioned in the blog.
I am sorry if I wasn't able to express my point properly.
I am too.
For example, even back then we did live streaming events of millions of simultaneous viewers and billions of on-demand streams a month, from petabytes of content in 14 datacenters globally (including Southeast Asia). All before Netflix, Google, Facebook. Akamai and most of the top CDNs were clients.
> Outside NA/EU echo chamber Netflix is a blip...
Netflix is very much playing outside ‘the echo chamber’: http://www.popularmechanics.com/technology/a20138/netflix-in...
And growing: http://variety.com/2017/digital/news/netflix-250-million-hou...
If you have stats that show they are ‘a blip’ in traffic volume globally, please share.
> I was trying to make a point on some info in the blog
By continuing to insist Netflix is “a blip”, and offering no definition of what you mean by large scale, or why you’re so sure Google is doing video delivery networking better, I’m finding it difficult to have a substantive discussion.
PS. You started an ISP in Southeast Asia, and looks like you had Google/Akamai footprint for your ISP. So maybe your ‘blip’ assertion is your ISP data? From the provider point of view, different ISPs (even in US) had very different viewing patterns. Without looking into individual ISPs, we guessed this related to median connection speeds, peering headroom, and localized content/viewing preferences.
What... No. An HD ProRes will be 150-200GB for a 1.5-2 hour film. That is a 1920x1080 Apple ProRes 422 HQ @ ~190Mbps which is the 'gold standard' accepted Mezz file for Netflix, Google Play, iTunes, AIV (Microsoft, Sony, TalkTalk etc). There are different codecs for broadcasters but this is the standard for VoD. If you deliver uncompressed (uncommon, few relative gains) then it might run into the hundreds of GB but never a multi-terabyte file for a feature.
Although most of this article has been informative, I am dismayed by the outright BS. It basically ruins any trust I would have of this otherwise well written piece. Can your exaggeration be explained or justified please?
Don't be afraid to drop things, even if they're important.
You've got a title, right? "What happens when you press play". Answer that question. There's a whole load there that is interesting but doesn't answer the question of "Given Netflix have a bunch of video ready to go, how does that reach me?". Don't get me wrong, the rest is interesting, and important when thinking about Netflix as a whole, but ultimately I'm unsure they fit in the article.
You might be worried about missing important bits, I personally worry about losing readers. It doesn't matter how important something is, if my target audience quits 15% of the way through the article.
Anyway, no sweat really, blogging is hard enough. It's been a long time since I've written regularly, and I just started a personal blog, and yeah, a few weeks later im sitting on a mound of drafts I need to take a 'minimum viable post' eye to. :)
Merry Christmas!
The normal expression is "explain like I'm 5", but in the cloud, you need to double everything, in case an AZ goes down.
How much of it due to special discount that netflix gets from AWS. Would a company thats not highly visible as netflix save the same amount of money ?
> Recommendations.
I find netflix recommendations pretty bad. Does it really work for other people?
They used to, the old system I found the recommendation to be pretty good and discovered some new things as a result
The new, like/dislike system is garbage and most of the stuff is recommends are not something I would enjoy... I wonder how much of that is even based on my personal rating anymore or it is just stuff netflix wants to promote...
Anti-lock Braking System
Both are exactly what they sound like
Buuuuuuuuut no.. it was a silly short-lived comment about piracy (not even an educating one)
Even recently Microsoft tried it for updates, and people were quite angry.
I honestly don't see a situation where someone would prefer to use their own bandwidth for a paid service rather than having a server do the heavy lifting. Either as the provider, or the consumer.
Maybe Netflix or a streaming service could pay for bytes uploaded. Abuse would be difficult to mitigate but in principle this would encourage peers to provide the company's service to the network.
I don't think this is true. Some enterprises were concerned about the security aspects and asked for greater controls, which Microsoft provided, but Windows 10 has not removed P2P updates, AFAIK.
I think it was the metered connection bit people were annoyed about, especially those who were tethering to their phones and then it downloaded/shared a huge amount of data.
This is easy enough to check with a simple Google search. (It's still enabled by default).
The five largest public torrent trackers track 30-50 million peers at any given time, serving over 10 million torrents. Torrents are estimated to have over a quarter of a billion active users. Torrents can be streamed, and in the end you have a DRM free copy of the file that you can rewatch at your pleasure, on any device(adding custom subtitles if you need them). It amazes me how much effort has been expended on pushing malware to almost everybody(binary, unauditable blobs in every browser) solely so that massive cartels can make money on something that has been solved since 2001.
Subscribing to Netflix, or any other proprietary streaming (dis)service is not only foolish(in the sense of surrendering your own freedom, not to mention your hard earned money to nefarious corporate interests), but deeply unethical.
Yes, this is the basis of our society. I work for money to buy things, mostly from corporations. There's nothing nefarious about that.
I find it really hard to believe that paying for content is unethical.
Your money isn't going to the people who created the content. It is going to cartels that have no interests in common with you, and have massive lobbying power in order to force you to download malware and restrict the access of information from you.
I wasn't 'lobbied' or forced to install 'malware'. I'm aware of what's going on (I too have heard of R.M.S.!) and I made a conscious decision to subscribe to such services.
To elaborate - we absolutely have something in common. These 'cartels' (businesses) don't give a flying fuck about DRM, they just want to be able to sell things. Often, they make good things that I want to buy. Good set of interests to have in common.
At the CS reception he told a retired professor to switch banks and just not tell his wife because the bank didn't support a FOSS browser—while eating the dandruff in his hair.
Later, at the wider talk, he called our IT department fascists for requiring authentication to join the Wifi network, then put a halo on his head, proclaimed himself the Jesus Christ of software and tried to auction off a stuffed gnu to a dead silent crowd.
Good times.
Patently false and easily verifiable.
Media is an investment, which is why were getting lots of "low creativity" movies, re-boots, extended franchises, sequels, etc.. People will risk money for art, but investors don't want to risk anything: some franchises can basically churn out anything and be sure to make a huge profit (which Hollywood accountants can spin as a loss by charging millions for IPR).
I think the spirit of the former comment is "when a movie makes $$$ profit how much of that is going to the colourist, or the make-up artist, the IT support, the animation assistants, ... all people essential to the movie. Versus how much goes to investors whose only part is being already rich?".
The creators get wages, but how many of the creators get profits, and how often is it more than simple investors get?
There are certainly reasons for not liking DRM, but "freedom to pirate" seems like a pretty poor one.
You are right about investors retreating to "safe low creativity", especially in times of downturn. Traditional movie theaters are in decline, so of course traditional cinema is playing it safe for now with reboots and sequels. Netflix, on the other hand, is more awash in investor cash... so at the moment, they are able to take greater risks.
As far as cutting out the "investor" side goes, it would be interesting to see a content host that is explicitly non-profit. Beyond major content hosts, I think the Soundclouds / Bandcamps / Patreons etc. definitely have their place, and have been helpful for some not-so-commercial / hobbyist artists. Even these platforms however seem to struggle with the conflict between making a profit vs serving the artistic community sometimes.
https://www.washingtonpost.com/news/the-switch/wp/2013/10/25...
A small example of the nefarious behaviour you support by paying for Netflix.
> Yes, this is the basis of our society. I work for money to buy things, mostly from corporations
Intellectual Property is by definition "non-excludable", which means my having access to some information does not exclude you from having access to the same information, unlike land or food or clothing. It is very different from these, and using your Government to violate the sovereignty of other nations and individuals to impose your warped up notions of property is deeply unethical.
Both are obviously false, but I think that was the intend (argument by confusion?).
How much money do the "cartels" generate for the content creators?
I believe the tens, if not hundreds, of thousands of people involved in the production of the thousands of TV shows and movies earned much more money from those "cartels" than those torrent trackers
It checks DRM settings in my browser and gets me lowest quality because my browser is open source. That is what happens.