Netflix’s Media Production Suite
netflixtechblog.com
netflixtechblog.com
- I wonder what the UI looks like compared to tools I use now
- I wonder if there will be a free tier, since my video needs are modest
Creating massive amounts of high quality content efficiently, on a global scale, with seamless global distribution is an incredible competitive advantage.
I don’t see why they would provide it to anyone outside of their ecosystem.
It’ll be interesting to see if they translate this to games as well.
They will never tell you anything that is real-world relevant for you.
At best, you might get some kind of _theoretical_ insight. It's because they're operating at a scale that just isn't realistic for hobby developers.
But they're still engineers just the same as you and me. So they write blogs like this. And it's interesting! I love to read them.
Netflix was famous for that, too -- no RSUs, just straight cash, and we'll fire you if we think you can't deliver.
100s of devs would essentially be their entire, company-wide, operating budget; it's gotta be like 10-15 people tops on these things.
The culture was and still is as you have described, with massively high pressure, "radical candor" taken to arguably very unhealthy levels, and with no hesitation in firing you. This is a major reason why, despite the fact that I am a video engineer with a film background who lives walking distance to 2 of their campuses and has a great amount of respect for their technical achievements, I never apply there.
It’s a company that prioritizes micro services and enterprise style crud apps internally. I’ve seen so many of their presentations and it’s like an IBM demo.
It’s data , data, data. That’s their approach to everything.
why wouldn't data data data be their approach?
that and churning out design-by-keyword visual media
CRUD apps are great for what they are. Exciting they are not.
Studios at weedonandscott dot com
If you’re unable to appreciate a behind-the-scenes look at their engineering because the technology isn't for you or available to you, that's totally valid! But it's a you're not interested thing, not a Netflix is boasting about something that doesn't matter thing. Only a few thousand teams in the world need most of what they do over there, but that doesn't mean they aren't massive technical achievements. Most of them are. The scale, complexity, and cadence of modern production has given rise to some of the biggest technical challenges I’ve ever seen. And for anyone close to that world, this kind of content is of great interest — if not genuinely valuable.
One of the people I worked with that is now at Netflix on this stuff was so violently opposed to not owning his own in office render farm and drive array it verged on ridiculous.
Why?
The resistance to cloud services is based on preventing leaks, not opposition to technology.
And in specific response to your comment: Netflix's "technology" is just a content management system. They're just reinventing a wheel that many of their competitors already use and bragging about something that Disney, Paramount, etc., did over a decade ago when they began embracing digital-first production.
“Digital-first production” can mean lots of things, so when you say Disney, Paramount, etc did this a decade ago, you’ll have to be more specific. Do you mean an end to end digital process? That’s not what this is about. Have you worked on a Netflix production? It’s night and day different from the studios you mentioned.
I think it will remain fine for Netflix in any case keep or replace. But companies who keep using Netflix OSS, or architecture ideas only because Netflix is so cool are going to have worse outcomes. Case in point is Micro services revolution which is almost invented and promoted by Netflix.
I feel Disney is up there too they just don't blog about it
For as long as broadcasting has been a thing, major broadcasters were involved in pushing the technology forward. For most of it's history the American network NBC was a subsidiary of the Radio Corporation of America. But NBC's brand is not tech, they want to consumer to associate the gliz of the picture.
There are plenty of people who have worked on Netflix and non-Netflix shows and would would argue that Netflix's workflow and high standards are difficult if you're not used to it yet, or more stringent than they'd like, but very few would deny the end results or technical superiority
This is objectively not true. Netflix has put almost no tech into the basic tooling of modern day TV/film (i.e., the cameras or audio equipment) or the software used to produce the content, or even the tech used to create the sets, makeup, CGI, or any of the other actual work that goes into producing the content.
The only place where Netflix has put in more work is on the non-linear distribution side.
Netfix is way behind the big dogs in the live streaming space. Peacock...the smallest major streaming service... livestreamed dozens of Olympic sports simultaneously at HD and 4K resolutions to over a hundred million simultaneous viewers without issue. Netflix couldn't handle half of that traffic for a single boxing match without crashing or degrading the streams to CRT-era resolution. The biggest player in the live streaming space is Disney Streaming (fka BAMTech before its acquisition) which was created to create the technology to stream MLB games and now currently provides the technology for ESPN streaming, NHL, MLB, Blaze Media, and Hulu's live streams.
The difference is that Netflix's competitors don't brag about their technology.
I've been a big fan of peaock! The Olympics coverage was massively impressive. Like much of what Peacock does, their success wasn't just about comprehensively covering it (which they did do) but also with how cleverly they packaged it, and all sorts of cool features it had that nobody else does/did, like the "Gold Zone" Red Zone-esque whip-around coverage, the constantly-updating highlights and key moments, etc. My impression of Peacock from the beginning was very good because their design and interface pretty much blows everyone else's away, and then I continued to be impressed after discovering a lot of these "cleverness" features, like when I noticed while watching soccer matches that the key moments/highlights were tagged and timestamped for easy access in real-time as they were occurring. I just wish they had a better catalog of shows to go along with. It is worth noting that while they are the smallest streamer, they do have one of the largest budgets and are probably the least burdened by existential risk because they've got Comcast behind them
As others have pointed out on this thread: Netflix's solutions don't scale up or down for others. This means that the technical solution is sophisticated because their unnecessarily complicated internal business process requires it. Netflix could save billions a year if it streamlined their production side like the studios have. Case in point: almost every film Netflix has made in the past few years has gone way over budget, and it's not because of backend buy-outs. (The Electric State cost $320 million, meaning that the base budget before backend buyouts was around $225 million. It doesn't even look like an $50 million movie.)
Netflix can get away with inefficient business processes because it's making enough money right now to paper over those costs, but eventually they'll have to streamline their processes. (For an indirect example, look up the story of Carolco Pictures, the company behind T2, Rambo, and other major hits.)
Vulture: Netflix's The Electric State Is a $320 Million Piece of Junk
The Atlantic: How to Make an Instantly Forgettable, Very Expensive Movie
The Guardian: Why are the most expensive Netflix movies also the worst?
EDIT:
Regarding this:
> Netflix can get away with inefficient business processes because it's making enough money right now to paper over those costs, but eventually they'll have to streamline their processes.
Light on details: the article is almost 3000 words, filled with vague and low-effort content: a lot more "We're so big and global!" and not nearly so much "Here's the problem we faced because we're so big and global, and here's how we solved it."
Misleading: they use the word "democratizes" twice: "we have crafted a scalable solution that ... democratizes access to advanced production tools across the globe" and "we’ve taken a bold step forward in enabling a suite of tools inside Netflix Content Hub that democratizes technology: the Media Production Suite" -- do you really get to say "democratizes" when you're describing an in-house system?
> “We did massive testing with the hard drives, and everything was great, and then we had an experience where we shot, and when we sent in the material, they couldn’t get the information off the hard drive,” said Cameron. “So the studio went ballistic and was like, ‘There’s just no way we can we can let you guys do this.’”
> The compromise was the production would record to hard drives as well as SRW tape. And unlike today, verifying the digital footage was equally cumbersome and tension-filled.
> “We recorded everything two or three times on decks that we carried with us,” said Beebe. “So we were backing up, two or three times.”
https://www.indiewire.com/features/interviews/michael-mann-c...
So they just rediscovered what IT world knew for decades, or what am I missing?
Even hobby level DSLRs have two card slots with option to write to both.
Professional cameras have tons of gear strapped to them, a second drive or some link to external storage is a no-brainer.
I remember when hard drives started getting big that it took a long time to get data on and off them. They got bigger faster than interfaces could keep up.
I think about 2004, a "big machine" would be an aluminum powermac G5 with an 80gb sata hard drive. Or a powerbook G4 with a 60gb ATA drive.
But you are right that Collateral did do something very new/unusual at the time, and that was shooting scenes in higher frame-rates than 24, and mixing multiple frame rates in a film. (This might not sound like much, but until this time, pretty much every film was 24fps for the previous eighty years and it had a very specific look that everyone's eyes/brains were conditioned for, unbeknownst to them.)
And the other thing that was very interesting thing about it (though not something very visible to a viewer) was that it was shot on the Thomson Viper FilmStream camera[1], which was the first major attempt at shooting not just digital, but very close to "raw". It was also a huge pain in the ass. The camera itself was massive, but due to the bandwidth, it recorded to an external storage array that had to be pushed alongside it at all times, and that was itself about the size of a shopping cart. (This device was hilariously referred to as the "Director's Friend.")
In 2002, my friend and I, both cinema nerds in high school, drove an hour away to the nearest theater showing a film called Russian Ark[2]. Why were journeying to to see a strange little Russian film where a never-named character walks the viewer through Russian history? Because just like each episode of the recently-released Netflix show Adolescence, this entire film was a single, very long, very complicated, unbroken shot. One shot. No trickery, no cuts that were just hidden to the audience, one shot, through streets, buildings, snow, ballrooms with a couple hundred choreographed actors, it was crazy. This is easy now compared to how it was back then.
As we've now established with Collateral (and this film predates it by 2 years), digital cinematography existed, but the storage was a real problem, the power was a real problem. Since this film was one shot, it needed almost 100 minutes of both, unbroken. And since it was a very complex moving shot, it had to be operated handheld. So essentially they had an incredibly ripped director of photography who operated the camera on a steadicam the whole time while a giant array of daisychained batteries and hard drives were lugged behind him. And they did it something like 100 times until they had a few takes where there were no mistakes.
None of this really means anything to anyone anymore, but at the time, to cinematography nerds at least, this stuff was all absolutely insane!
[1] https://web.archive.org/web/20040103133953/http://www.thomso...
I'm wondering why people would have chosen to do early digital if it was so inconvenient. When did the cost and flexibility advantages start to really kick in?
Simply shooting a feature film digitally was not that complicated by this time, or at least it didn't have to be. The Sony CineAlta F900 was the camera developed to shoot the Star Wars prequels, and was revolutionary at the time, became the gold standard for years, and very convenient relative to film. Tons of things started to be shot in 1080p around that time, and it was very nice to work with. Collateral was insane because they wanted to shoot raw and at high frame rates. Russian Ark needed a single unbroken shot in a form factor that one human would be capable of holding for that long. Aside from very specific and/or boundary-pushing needs, the arrival of the F900 in 2000 was effectively when digital was more convenient than shooting film while also meeting the technical requirements of high-end production (though it was many more years before most cinematographers agreed that the image quality was comparable)
In the years after that I worked with them to write a custom application based on a Bluefish444 card combined with some ATTO fiber channel storage just to get the frames to disk fast enough. A lot of custom code, overlapped I/O, that kind of thing. We had a beast of a JBOD RAID setup, must have been about 12 spinning disks.
The only alternative in those days were systems that stored to tape, but could only do so in a compressed format (I think Sony had a solution that did 4:2:0 instead of the 4:4:4 coming out of the Viper). People were scrambling for these storage solutions so much that we even got Arri to lend us their prototype D-20 camera (which turned into D-21 which turned into Alexa) just so we could make sure our storage system worked with their camera. We just had this amazing prototype camera sitting around our office for what must have been a year. They just lent it to us. Wild. I think our only main competitor at the time was Codex, which admittedly had a much slicker system.
We visited the CINEC trade show and got a ton of interest. I think I still have a business card of the DoP that did all the miniature work in Lord Of The Rings. He loved the fact that we would store things uncompressed, which would make things like compositing a lot easier.
Unfortunately, mismanagement caused the whole thing to collapse. Oh well. Nowadays you just use a CompactFlash card :)
While working on set between 2012-2014, we were shooting all RED, and I had a 12-disk ATTO FC RAID10 rig on set at all times borrowed from RED. Not needed for speed of frames by this time of course, but for the ridiculous total storage required shooting 6.5k raw and the time needed to copy it all. On paper this system should've been good/safe enough. In practice, we almost lost it a handful of times within a month, each in a unique way, including the time a stunt driver messed up, veered off course, and plowed directly into video village, striking the RAID and killing exactly the maximum number of disks in one mirror it could tolerate, but thankfully no more. (Needless to say the shoot was a massive learning experience and I have never managed data the same ever since.) By the time the shoot was over, the RAID was alive, but it was absolutely beat to shit, and I was afraid of how the guy from RED would react when he came to pick it up. When he did, he was completely unphased. He chuckled and said, "You should've seen how messed it was after Ridley Scott's crew borrowed it!"
Very cool background though, I was not quite old enough to get into it all quite that early! When the Viper came out, I was still in high school, just exceedingly nerdy. I believe to this day I have PCs with ATTO Disk Benchmark on them
Why was certain scenes in Collateral filmed in other frame rates than 24 fps (unless you are doing slow motion of course)? AFAIK it was never projected/shown in anything else than 24 fps.
The other major part was the shutter speed. They of course could not actually shoot/project 48/60fps, but they a shot a lot scenes at the high shutter speeds one typically uses when shooting those frame rates, a lot of it had that "ultratrealistic" look that people had weren't used to in films, resembling more the look of video, TV soap operas in 60i, etc.
I feel slightly absurd even writing about this considering how little of this really applies today, and how inconsequential changing the shutter speed on a camera is now. "I hit the '+ shutter' button a couple of times, revolutionary!" But it's crazy how conditioned everyone was to these looks at the time due to how little variety there was. I taught this film class where I would demonstrate to everyone, with nearly 100% success, that they all were influenced by and conditioned for these frame rates, even if they didn't know what a frame rate was. We'd shoot a scene with multiple cameras side-by-side shooting at different frames rates, play it back to the class later, and ask which one looked "more like a movie" to them. Invariably, even if they couldn't explain why, everyone always picked the 24p version
Isn't this the most expected result? How could it possibly be anything else?
But would this experiment yield the same results today? For people in certain demographics or below a certain age, I think almost certainly not.
I also think that a lot of the people who would've easily identified the 24p clip back in 2004ish when this class took place may no longer do so now. I never would've thought this before 2018ish, but I had experiences with people around and since that time that surprised me and turned me completely around on this view.
Though there is one other alternative that that I wouldn't have difficulty believing. I think it's possible that more people than I just predicted would identify the 24p clip as "cinematic", but that they may not view it as a desirable look, inherently more dramatic, having much impact on the viewer, etc.
Wouldn't this just lead to judder? Maybe that was something that had not been seen before in a movie shown at cinemas.
I've heard that sometimes scenes like kung fu fights were filmed in a lower frame rate (maybe 20 fps) and then it got faster when it was projected in 24 fps. If you do it the other way around movement just get slower (which is what you want for slow motion).
What does this mean?
Bandwidth was of 100KB/sec at most, I suggested to do that fly over things if the systems team didn't want to raise the priority of that transfer, after prod tried 3 times over the weekend, sadly, they changed the priority of that flow, it took still like 40h? For the initial load.
I’d often thought (critically) about the lack of visual diversity in Netflix output - and this is something I often see stereotypical film-enjoyers complain about.
I’d never considered it as a consequence of Netflix’s sheer scale. It’s always really interesting when I discover that something I’d previously put down as an (unimaginably unimaginative) aesthetic choice might in fact be an operational choice. It makes me check myself!
It sounds an incredibly complex and clever system; I can’t help but feel that applying such a strong vertical to the more creative aspects of film and tv production - such as colour grading - will ultimately prove short-sighted.
The actual content is invariably a result of the infrastructure behind it. And it’s not just the image, it’s also what kind of scripts are written, what kind of audience insights are passed on to creative producers, what kind of creative teams are selected.
Is it proving short-sighted? If you’re optimizing for cinematic art, then yeah. But they’re optimizing for subscriptions and global reach. That vertical will likely move to live-streaming, sports and other forms that retain subscribers. And on multiple global markets at the same time (not just U.S.)
It’s a weird vertical, they’re quite sophisticated in their approach, but it’s surprising how they sometimes contract entire chunks out. I’ve read academic papers talking about how Netflix is a very strange disjointed thing.
I do wonder if an in-house aesthetic can become ‘tacky’ in the age of global media - can trends ‘die’ when there are still billions more people to reach? And will a creative org structure like this be able to move fast enough should that happen? I don’t think we know the answer to that yet.
I personally believe (maybe optimistically!) that this will be an important question even though Netflix’s natural conclusion is to move towards the subscription-retaining, low-creative products like sports that you mentioned.
The problem with those entertainment products is that they have intrinsic value: if the provider is adding little value besides distribution, some (or lots) of users will pirate that content. Super apparent in sports media.
Maybe it’s a naive hope, rather than a belief, but I hope / believe that because of this, companies like Netflix will be ultimately forced by users to have more idiosyncrasy in their production pipeline and output. It’ll be really interesting to find out!
https://aws.amazon.com/blogs/aws/new-physical-aws-data-trans...
There's one that uses Gmail to exchange documents (not financial, but important nonetheless) and uses the read receipt to determine if it has ingested the data. Replaying ingestion is marking unread.
It was not allowed and they had reliable corporate tools to exchange patient data, but the UX probably felt so cumbersome they'd rather face legal risks than doing it the right way.
Netflix is looking to sell shovels to as many people as possible, it seems.
The problem is, storytelling is risky. Either you stick to something bland (adaptations of popular books, cartoons or videogames) and it usually gives decent returns, or you go for something completely original - at the risk of it either going boom or going bust.
As your average cinema movie is a triple-digit million dollar business these days just in pure production and actor cost and double that in promo cost, it's hard to find banks to finance the production, and so the banks prefer to go with something "proven and bland" over something risky.
They make "content", and the distinction is super important.
Yes, it's pretty vertical, but essentially every big streaming platform or production company is. There's a very specific Netflix way that governs how all Netflix shows are produced. There is a very different but still meticulously standardized way that governs how all Hulu shows are produced, one for all Warner Bros. shows, etc. This is an area where being vertically integrated is totally fine. It not only makes enormous sense for these studios, but nobody outside of those environments wants or needs these workflows. Netflix's workflows are there to aid Netflix even more than their shows, and while there's a lot of excellent stuff in their workflows that most productions should utilize for efficiency/safety/whatever, there's also a ton of stuff that would make no sense to use independently.
This workflow and tool set is tuned for Netflix and is probably opinionated in a variety of ways to conform with Netflix production standards and requirements. If your production is being funded by Netflix then you're incentivized to learn and use their provided tools.
Wasn't this how AWS started at Amazon as well?
I remember when being made by Netflix was a unique and cool thing, it didn’t last long until it meant probably-slop.
https://www.youtube.com/watch?v=h-vWRLGnTGg
A filmmaker discusses their experience with Amazon, who bought the rights to their movie and decided to release it on their streaming service instead of in theaters. The filmmaker explains why they feel this is unfair and why they are fighting for their movie to be seen on the big screen.
My personal experience with netflix has been that a good filter for 'quality' is what specific TV series and documentaries various 'scene' groups in the warez/torrent community consider worth ripping and properly encoding.
There's a certain amount of manual effort that's required to properly encode a ripped netflix or amazon prime series. People who do this strictly for street cred in the piracy community generally don't waste their time on schlock.
Maid? The Queen's Gambit? Baby Reindeer? The Crown? Ripley? BoJack Horseman?
Sure they make a lot of schlock too, because they're a business and that's what most of their audience wants.
But I don't see how you could possibly criticize them for that when they continue to put out some pretty astonishingly artistic and soulful stuff.
But OK yes, Netflix produces a lot of volume because that's also what its viewers want. Are you saying that's a bad thing?
Sometimes people get home from an exhausting stressful day at work and they just need to relax with some mindless entertainment. And that's OK. Not everything has to be art.
1. Consumers frequently don't know what they want, and will consume whatever is placed in front of them. Implying the consumption of Netflix shows (or other media) is a conscious choice they are making seems disingenuous to me. You see the same with doom-scrolling TikTok or Twitter (or even HN). Frequently it is less of a choice and more of a compulsion.
2. Even when they are making conscious choices, sometimes the things people choose to consume are objectively bad. e.g. sometimes people get home from an exhausting stressful day at work and just need to shoot up heroin. While you can respect the agency of those people to make those choices, most societies do various things to discourage such behavior. A drug dealer whose entire defense is "selling crack isn't bad because my customers want it, actually" is not gonna get exonerated.
And no, TV isn't heroin or crack. And who on earth are you, or me, to judge what other people want to watch on TV? That's paternalistic and insulting. Next do you want to censor which books I can read?
And I never suggested preventing people from watching what they want to watch.
If our ability (as a society or as individuals) to filter out the slop from the rest increases in lock-step, this is a non-issue. But it seems that this not what has happened, and instead we are inundated with mind-numbing content that absorbs our time and does little to impact us in any positive way.
Effort of course went down that's clearly a good thing. As for quality, before you needed to get enough return to pay for the expensive equipment and process, so likely you only do it for very few project. So on some abstract sense maybe 'quality' did go down, but that isn't bad as the total amount of high quality goes up far more.
See the massive amount of great TV that have happened since digital cameras.
> slop
What's slop for you is somebody else's favorite show.
Art doesn't have to 'impact us in a positive way' whatever that means. You are not a better person for having watched "Lawrence of Arabia". And in the past most people didn't watch "Lawrence of Arabia" but generic TV shows.
Personally I can easily filter the 'slop' (ie thinks I don't care about) so given how much better the ability is to select what to watch. On demand media libraries, recommendation systems, digital word of mouth and so on. In the past there were few TV channels and only a few movies in theaters at the same time.
So the total access to high quality content has gone up exponentially.
That is, it used to be (80s/90s) a lot more obvious what the prestige/not prestige boundary was. Cheap TV content (soaps etc) was shot on video, expensive content shot on film. Now everything looks the same. Perhaps the one remaining effort signal was lighting, but Netflix seem to have chosen very flat bright lighting styles for everything now. Bad news for us chiaroscuro lovers. And even when directors do try to do that, they've often over-estimated the HDR so you get the opposite: an entire series which is too dark.