Hacker News Reader

At one point, Netflix video streaming accounted for something absurd like 20% of all Internet traffic in the country. Regardless of whether it "scaled incredibly well," I would imagine there are still novel issues with that much scaling.

Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing to shuttle 5GB of data with no hiccups for an hour. Detecting and routing around machine or network issues fast enough that real-time video playback is not impeded sounds to me like a difficult challenge.

"Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing to shuttle 5GB of data with no hiccups for an hour."

Speaking of which ...

I notice that while youtube continues to buffer video while paused, netflix (and many, many other video players online) do not.

So while netflix will auto-adjust quality for you in response to a bad net connection, you can't just pause it, go brush your teeth, and come back with a big enough buffer to avoid skips through the entire video.

When did video players stop buffering-on-pause ? Why was that choice made ?

On the other end of the spectrum is whatever video player showtime online uses - it does not adjust quality and it does not buffer on pause. Basically it was built for perfect Internet connections and nothing else.

Probably too many cases where they deliver a bunch of content while it's paused and then the content never gets played. More than half the people who pause it probably end up closing the tab without finishing the video, so one can save a lot of bandwidth by only delivering the data when it's needed. A totally unsupported hypothesis.

I arrived at the same conclusion when I realised that YouTube no longer buffers the video all the way through to the end on mobile anymore. I miss being able to buffer a (long) video for offline playback when riding public transport.

Ditto, for this case I youtube-dl prior and watch on mobile VLC

> I notice that while youtube continues to buffer video while paused

Not anymore, or at least not for long periods of time.

Beat me to it.

I am sure a large part Netflix's success is how seamless its products are: there's minimal to no loading time and the video stream is virtually always smooth - including during times of action/high frame rate.

Creating a video streaming platform is one thing, but creating a content delivery platform that is effectively as reliable as TV that is also cost efficient is no small task if you ask me...

Bullcrap. The tech is good enough (but not that much better beyond that level). >90% of the success comes from the content creation/licensing strategy/execution and marketing. I.e. non-tech.

At this point it's a bit like arguing that Disney's great success in Bluray sales comes from their encoding/Bluray mastering expertise.

You seem to stumbling into the distinction between functional and non-functional requirements. Netflix spends massively more on content then tech so that's consistent with what you're saying. To use you're Disney example though, yes it would effect their success if they weren't able to deliver their content because their Bluray mastering expertise was lacking.

Exactly. The tech division, while overfunded and overengineered is fulfilling its role in a good enough way.

Tech part was "good enough" for last 20+ years - it's why most companies call their tech teams "IT department".

There is however something about that last 5-10% where moving a few bits around the right way gives you business advantage, over places like Disney.

That said, it does seem like some of this Netflix stuff is overbuilt, although I don't need to run tens of 1000's of ec2, so maybe it's worth it.

Tens of thousands of ec2 instances, none of which served any customer, if you red the article carefully.

> Regardless of whether it "scaled incredibly well," I would imagine there are still novel issues with that much scaling.

I would imagine there is not. When you master something for 1%, you do 20% without too much hassle. (Let's remember that netflix is only serving a couple of videos to a lot of customers, that is embarrassingly parallel and cachable).

About reliability. Video is one of the thing that has the least need for reliability. You can have thousands of hiccup during the hours, doesn't matter, nooone can see any of them.

I expect most of the tech challenges to be in cost optimizations, not in delivering the service.

That is the core of their product. However, there are a lot of by-products and back office software needed to run the company. As a consumer, you most likely are only perceiving a small percentage of the technology running a company.

- Applications for all devices Netflix is on (TVs, Phones, Displays, Roku, Apple TV, etc.) and all the support around it.

- Recommendation and analytical software within Netflix. Not even accounting for other areas of the company where ML can be applied.

- Billing and financial software for Netflix users and partners

- Proprietary/internal customer support software

- Content management system, along with all the international and legal challenges around that.

- Any/all software used by their marketing teams

* Operations and control plane for all of the above that is robust enough for SREs to deal with million-to-one-chance occurrences happening every few minutes.

* Complexity multiplier (variations of contracts, catalogues, infrastructure, language, currency, payment methods, taxation, government regulation, corporate governance, market structure, customer preferences) of offering service in >100 countries.

The suggestion that building a service like Netflix is a "solved problem" is naive to the point of idiocy.

Another thing worth noting is that they still support basically every piece of hardware they ever did. Think built in apps on old smart TVs or early blu-ray decks that haven't been patched in YEARS, devices from before 2009~ even.

You can still watch netflix on an original Wii. That hasn't gotten any OS or software patches since like 2011. That's a long tail of legacy clients to have to deal with.

For the record, the official Netflix client for PS2 (only legitimately released in some South American countries) was discontinued and is unsupported. I'm guessing they moved exclusively to H.264 at some point (no idea what codec they served prior to that point, but to be playable on the PS2 it would have either had to be MPEG-2 or MPEG-4 ASP/Xvid)

Given that streaming is basically competing with discs, it makes sense not to drop any devices ever; a DVD produced today is expected to work on a DVD player from however long ago, provided it's still operational.

Most of these stuff are also done by large corporate IT departments such as banks. They all consider it pretty routine and are not writing engineering blog posts.

Maybe the engineering community as a whole be better for it if so much duplication of work wasn't going on behind closed doors.

Sure, you don't want to publicize things which could be a competitive advantage to your company, but I bet these banks of yours wish weren't sitting on huge heavy custom-rolled COBOL codebases right now...

This sounds like one of those developer estimates that takes 10x longer than stated.

"Netflix is over-engineered, it's just videos lol"

Classic HN hubris.

For example, Netflix dynamically adjusts video quality on demand. Pretty easy feature. The top HN comment will explain how they would implement that in five minutes -- hypothetically, of course!

"Responsible 20% of the entire us internet traffic? Solved problem mate, i could do it on a single machine with mongodb. Overengineered!"

Forget about the joke for a minute and let's be realistic.

It is entirely possible and reasonable to prototype a streaming service serving 20% of the us traffic with a single machine and mongodb.

Well, a single machine, mongodb AND akamai.

Well of course, mongodb is web scale don't ya know

MongoDB is shit. No need to use it.

What's important is really the distribution and caching layer, and Akamai has been offering that for a very long time for all the big internet companies.

Prove it.

Just think how much more they could do if they stopped using MySQL. /s

Anyone up for the "make Netflix in a weekend challenge"?

Rails + Devise + Paperclip + JWPlayer = who's got Bob Iger's phone number?

How does the fact that other companies have streaming video services lead to Netflix not having to solve that problem? This seems a bit like saying that Walmart doesn't need to worry about managing their supply chain because Cosco has already solved that problem.

Also if you've ever tried a TV channel's streaming service you'll realize that recording the start and end time won't tell you much about the quality of the experience.

In the end Netflix is one of the places people would like to work because of stuff like this. The sheer scale of operations at netflix combined with modern technology is very interesting to many engineers, so it also kind of work in favour of their HR.

I get the impression they're running analytics not just on the users but also the content (which would take more horsepower). Guessing they use the analysis to model scripts and generate ideas for their Original Series. Thinking of what is needed to do this, you need to analyze themes, characters, story lines, visual effects, music, many components, and then tie them to user types and viewing patterns. Then they create series with either wide appeal or strong appeal to a very small niche. On top of that, these shows load almost instantly while you're still determining whether to watch or not.

http://www.smbc-comics.com/comic/2011-12-13

If we make a scale from 1 to 4 out of those four panels, I'd say Google is around a 2.5 and Netflix is somewhere above 3. It's tough to assign numbers much past 3 because the technology becomes indistinguishable from magic pretty quickly.

> Google, Facebook, Baidu, Amazon, the self driving car companies are doing cutting edge stuff in terms of scalability and analysis, but not Netflix.

I think you're missing a quantifier of some sort. Perhaps "consumer-facing"? Of any AWS customer, Netflix is far and away the leader in understanding and making use of AWS. It pushes the limits more often than not.

Their scalability, failure/recovery handling and (assumed) cost-effective deployments are _the_ standard in AWS and cloud computing. They even have an open-source toolset[0] for testing your infrastructure against failures and recovery that's quickly being picked up by several other companies to harden their infrastructure.

Then you realize that regularly, throughout the day, they're intentionally bringing their own infrastructure down and they can still operate with nary a hiccup. I wouldn't call that over-engineered.

[0] https://github.com/Netflix/SimianArmy

I can see this kind of over engineering in all the companies that hit the VC/IPO lottery. They all hire like crazy the best engineers and have to keep them busy doing challenging engineering stuff. And managers want to protect their turf so they join in. So you have ever more complex designs in anticipation of growth, side projects, detailed analysis, blogs with pretty graphs, etc. A combination of Parkinson's Law and Second System Effect.

For sure there is a bit of self-flagellation. However, what it doesn't tell you is how cost effective or as they elude to: how improves developer velocity (a stand in for productivity) is.

Those other guys may also be offering streaming, but how quickly can they adapt?

Don't they also have some kind of recommendation engine? I bet that takes a bit more data and processing (who watched what, did they finish, maybe even more like what kind of scenes did they stop watching, or watched again, etc)

> Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box.

This is exactly what they do. The video serving part isn't that complicated, it's the rest of the site and APIs.

Although those are a tad over engineered considering the final output is often a slow and unwieldy UX.

> A national grocery chain, probably has a similar amount of data ingestion and a similar analytics need.

It's off-topic, but having interacted with national-level grocery chains before, I think you grossly over-estimate their technological capacity and savvy. They're probably one of the slowest industries to adopt new technological advancements, for better or worse.

While the contracts may be negotiated months in advance the videos aren't delivered until very close to the release date, the same for Google and amazon. In some cases it may be a few days early but production schedules are tight and even if the post house has it done early contracts and controls forbid them from sending it to Netflix, etc too early.