A View of the Entire Netflix Stack | Hacker News Reader

A View of the Entire Netflix Stack

highscalability.com

407 pointshepha1979152 comments

@monsky: (Somehow this got disconnected from the thread below.)

The headline of the slide you pulled that from is literally "Hard Work—Not Relevant."

Netflix doesn't care if you work 100 hours a week, they care how well you perform. If you can do an amazing job in 30 hours a week, they're one of the few places which would actually embrace you.

> I'm not sure how Netflix grades things... but in school B is "Above average".

So what? Why on Earth should Netflix be targeting "average" workers when they deliberately pay twice the average.

That's about as ridiculous as saying that Harvard should stop admitting A students because "B" is average.

Netflix has a deliberate strategy of keeping their engineering bar very high and paying accordingly. That doesn't mean they want you to sacrifice your life to the company.

Also, Netflix is in no way a startup.

The problem about the idea of "doing great things in 30 hours" is that people who are not directly involved with implementation and deployment usually think that each and every 2 day task takes at most 2 hours, because we humans are unable to expect the complexity that most tasks have on this planet. (every time you think about a college, super market employee etc "just do your job man" you are experiencing that fallacy)

And then you as the 30 hours employee are faced with not delivering 40 hours of work in 30 hours (which is possible if you are smarter, more involved, have more experience etc) but no you are expected to do 160 hours of work in 30 hours, because you don't compete with the other engineers but you compete with the expectation of your manager.

We need to accept that most people (colleagues, employees, customers, bosses) will always be disappointed by the results we achieved because they had unrealistic expectations to begin with. And we must excuse them because we do the same the other way around. That is why "40 hour week" is so relevant. Because then you can say after working hard for 40 hours, really doing all you can, maybe even achieving 30% more than your colleagues, that time is up and you need to take care of your children, wife, parents, friends and hobbies you may have.

Now let's assume for a moment that all I said was BS. Let's assume I'm super untalented and for most people it's actually possible to achieve in 30 hours enough to impress their leaders. Now you do it once. You do it twice. Maybe thrice. But at that point your leader will adjust in his head, maybe without realising, what 100% means for you. So the fourth time he doesn't think it's awesome anymore. The tenth time he thinks you are lazy. And now you have to achieve 50 hours of work in 30 hours. It still doesn't work out.

> And now you have to achieve 50 hours of work in 30 hours.

Could this argument not be made about the 40hr week?

Sorry, I wasn't able to express myself clearly. If you fix a 40 hour week your boss is disappointed that you stop working at 40 hours. If you do not have a fixed amount of hours your boss will still be disappointed, but you have no good reason to ever stop working. Without the 40 hour limit you work more and more and more and still disappoint your boss. The disappointment can't be avoided. But the unlimited overtime can be avoided, by limiting the amount of hours you work.

Only if everybody else was working 50 hour weeks.

It has nothing to do with everyone else. If you impressed your boss while sticking to a 40hr week, they'll only be impressed so long before it becomes your norm. In which case you bump up your hours to maintain your prestige. I don't buy that reasoning, just saying it goes both ways.

This is deeply true

I think the general idea that results should be the only relevant metric is good. I'm a bit worried that the mantra could have bad long term implications though. Iirc. I read a study from the domain of education science that basically said "always praise work ethic, rarely praise results" and it seemed convincing at the time. [I'll see if I can find it later and edit it in]

That's probably true when it comes to education, but the primary purpose of an employer isn't education.

Netflix doesn't necessarily pay the best, but they pay well. Read this:

http://www.slideshare.net/reed2001/culture-1798664

They pay their senior software engineers $350k+ per year.

Not according to Glassdoor. Average Senior Software Engineer is $212k per year in total compensation. Directors of engineering are at $328.

Glassdoor is wrong. The pay at Netflix is much higher than it shows there. GP is much closer to the truth, but still low.

Source: I was a senior engineer at Netflix.

> GP is much closer to the truth, but still low > Source: I was a senior engineer at Netflix.

OK, suppose it was $400K+. How do you describe your life (-work) balance? Also, may I ask you - was it your decision to leave Netflix? (and what you didn't like there?)

Another thing: as you can see one can have wrong expectations - e.g. on salary negotiations one can ask for 250-300K while as you say, the norm is much higher... So do Netflix propose alternative much higher salary compared to what you've asked?

Work life balance was great. Sure there were crunch times, but for the most part it was 40 to 50 hours a week. If you were at the office at 7pm, you were there pretty much alone.

When I started at Netflix, I asked for what I thought was a crazy salary, almost double what I was making previously. My boss said, "No, that's not high enough, lets give you 10K more". Every subsequent raise after that, my reaction was generally, "oh, that's more than I expected".

Netflix knows that there is no point in low balling you, because you will soon find out and then be upset/leave. They never want to lose someone over a matter of money. They only feel good about losing someone because they no longer enjoyed their work.

Thanks, really interesting. One more thing to clarify - on top of that "$400K" salary, does Netflix also pays for social/health insurance (including family)? If not, how much you normally pay for such insurance?

I mean, I've heard many times that while NET base salary e.g. in Microsoft or Adobe might be lower, when you add up all the benefits, bonuses, stocks - it becomes quite competitive to Netflix compensation. Also factor in the job security and family-friendliness and it becomes much more attractive... What do you think about this?

Basically they either cover your health insurance, or you can opt to just get paid extra and not take insurance (if say you have it from a spouse's job or something).

And yes, the other big companies total comp does come pretty close to Netflix. The difference is Netflix gives you cash to spend however you want, and those other companies force you to consume benefits that you may not want to get your "full comp".

If you're worried about job security then Netflix is not the place for you.

But no, .NET jobs do not compare to a job at Netflix. Not even remotely close. Unless you are being paid 300k and get 100k in bonus/stock every year.

He's talking about net salary, not .net jobs.

I would describe 40-50 hours a week as "ok", not "great". Great is 35-40 IMO.

40-50 hours for a salary over $300k is great. It's practically unheard of in any field.

Usually you have to make a severe tradeoff between good pay and good culture. Not so with Netflix.

I don't think you'll find a company in the bay area where people work less than 40 hours a week as an engineer.

No it's $350k+. First off, I got this information first hand from someone I know who got a job offer there. Second, go check the H1B salaries from 2015, that shows the real salaries, not glassdoor.

The problem with only having "A" talent is that you become very vulnerable of bias in your own criteria. People often primer more concrete skills like in-depth knowledge. While sometimes you just need people who are more motivated to do whatever. It's a classic problem of successful sports teams for instance. They buy a lot of high profile players that end up not working very well together, won't adapt to new situations and are sensitive to failure.

Yes, that's true but it's also part of why Netflix also has a "fire fast" culture (unlikely sports teams). So people who can't adapt their skills to performing well in a team aren't kept long.

Efficiency pay (which Netflix heavily practices) does a great job of attracting people who are "motivated to do whatever."

I meant to say there will be an average at Netflix. When you're being compared to others. Not everone can be an "A" level individual.

That doesn't make any sense and is a deliberate misreading of the slide.

It's well-accepted that when you say "our team is made of A-players" you're grading against the overall industry, not within your individual team. Grading within teams leads to perverse systems like stack ranking.

> Sustained B-level performance, despite "A for effort", generates a generous severance package, with respect.

You've got to love these startups who basically say "if you're not giving us everything beyond what we expect... go screw yourself. I'm not sure how Netflix grades things... but in school B is "Above average".

Is Netflix looking to employee and develop people or are they looking to extract life from people?

Probably the latter. In my view, Netflix (IBM, Amazon, Bloomberg etc) should consider what would happen if they were completely honest to their employees:

"You are a cog in our machine. We do not care for you. We only care for your work. We will pay you insofar as you can execute on our vision. Further, you will be paid according to how hard it is to replace you. If you can be replaced by someone who executes significantly better, relative to your pay, we will tell you to leave."

Needless to say, stating that would make many of their employees, especially the weak-minded and insecure (those who defend their terrible culture), unhappy. I imagine even those in charge of the culture, speaking those words, would feel bad.

And they should feel bad. While such overwhelmingly negative words should almost never be spoken with authority, these decision makers must at least be able to answer the question "What are we really saying to our people?" truthfully. And if the answer is something along the lines of "you're a cog in the machine", then something has gone very wrong[1]

In my non-expert opinion, if a company truly wishes to be a "good" place to work, it should know the distance between their words and the truth, and then focus on minimizing that distance.

Spoken more simply: the more dishonest a company is to its employees, the worse the culture.

[1] Or right, if the company requires joylessness to turn a profit.

>"You are a cog in our machine. We do not care for you. We only care for your work. We will pay you insofar as you can execute on our vision. Further, you will be paid according to how hard it is to replace you. If you can be replaced by someone who executes significantly better, relative to your pay, we will tell you to leave."

Isn't this the ideal of wage labor and the reality of 98% of jobs? I've never even suspected that my corporate employer cares about me as a person aside from how I produce, and have never suspected that I'm being paid by them for any other reason that it would cost them a lot more than they pay me to find somebody to replace me. I don't mean to say I haven't had great friendships and working relationships with management, but to the business, I'm a line item. When I notice a business making decisions contrary to the bottom line, I suspect corruption, not corporate affection.

They're a cog in my machine, too. I can only depend on that cog if it behaves rationally.

I agree it's the reality for a majority if not the super majority of jobs. I also think it could be the ideal of hourly wage labor: which is to say that the company wants nothing more than what they can motivate you to do with money, and that you want nothing more from the company than their money.

Looking back, I see I made my comment in the context of Silicon Valley behemoths and startups: companies that can both pretend to, and potentially change some fundamental way of doing things.

I think the difference in my mind between "creative" companies and normal ones is the "potential", achievements outside of a strictly defined expectations of contract, for both the business and the employee. Maybe a great insight, a new tool... anything that greatly benefits both the business and the employee, especially if the employee uses the product(s) and believes in the mission.

When you take away that potential (by saying employees are just cogs in a machine), you might take with it the creative spark that these hopeful companies ostensibly rely on. In my mind, that "potential" is the real joy of a job, which is my reason for the (slightly snarky) comment that "cog in the machine" is only acceptable "if the company requires joylessness to turn a profit". If I could, I would edit that post to be more clear/to take into account businesses which do not require any specialness.

At least for my central point, "the more dishonest, the worst the culture"... I believe that stands, especially in circumstances where a person is (somehow) manufacturing toilet paper and their supervisor is telling them they are (somehow) on their way to curing cancer.

> If I could, I would edit that post to be more clear/to take into account businesses which do not require any specialness.

That I'm creative (amongst other things) is why they pay me well. My successful experiments have a record of adding value well above my pay even if aggregated with my many unsuccessful experiments, although I try my best not to keep it that way for long. I'm not sure I've had any technical job with completely "strictly defined expectations of contract", I think that's a red herring. It sounds nice that the most creative and innovative projects/companies give less of a shit about the bottom line; I personally think that the bottom line is the fuel for creativity (didn't have much money when I was younger.) Anybody can sell your product with unlimited money to spill, but doing it with the best profit margin possible is creativity.

I like to think I'm a special snowflake...

In my experience, most employers think this way, but they don't like to acknowledge it. I've had bosses that talk a lot about "win-win scenarios" and how we're all working together to this common goal, all the while creating a very worker-hostile environment where people are pushed to work much more than they feel comfortable with, with regular emails praising the workaholic who works so much he never gets to see his kids. The unspoken message is, this is what we expect of everyone.

These same people get profoundly uncomfortable if you respond to their platitudes with reality. They prefer to keep up this illusion of cooperation, because it makes them feel better.

I've said this before, but how I wish software companies based on the cooperative model[1] were more of a thing. It's not healthy for people to be treated like worker bees. Corporations should be a collaborative effort, and the wealth and decision-making should be distributed proportionally.

In a way, the structure of our corporations both reflects and influences our society. A recent study has shown that the US is now more of an oligarchy than a representative democracy. Would we be happier and better off if "working for" was replaced by "working with"?

[1]: https://en.wikipedia.org/wiki/Cooperative

We are in the very, very early stages, but my startup[1] is working on an open source toolkit for building cooperative companies.

[1]: https://infinite.ai

That is awesome! I'll be keeping an eye on this.

I was with you until that employees are deserving of the proportional wealth. The people who took the risk to found the company are deserving of that.

I meant proportional to their participation. Naturally, the founders should have more of a stake, but certainly not 1000x the stake of Joe Engineer.

Bit offtopic perhaps, but Netflix CEO pay is something like 15-25 times the salary of Joe Engineer.

Worth noting, but the 1000x figure that gets bandied about so often when referring to CEO pay is typically limited only to the Fortune 50 or so, and is in no way the mean pay for the top guy. S&P 500 companies lower the average to less than 400x worker pay, and most businesses have a much smaller gap than that, and across all companies, average CEO pay is something like $157k.

I guess I'm thinking of the traditional startup IPO, where the first handful of employees get "fuck you money" while everyone else only gets a small piece of the pie (despite possibly doing just as much work).

But also, more importantly, people outside the C-level rarely get a say in how the company gets run.

That's kind of a different thing altogether though, really. The "fuck you money" generally comes only in the case of a liquidity event, such as IPO, sale, etc.

To my ear, there are compelling arguments both ways. Assuming a large company, employees 1-x should probably enjoy some of that fuck you money, but clearly, at some point there's a falloff. If my company sold tomorrow, I would not have in any way been involved in its sale. Same with an IPO. Our company has been around for years, sells expensive software, has millions in revenue, and I honestly couldn't tell you whether or not we've turned a profit yet. Assuming we have, it was probably done only last year, which meant that somebody was footing the bills of a multi million dollar company for a decade or so, and that person wasn't me. If somebody was spending millions on an idea that took a decade to pay off, then absolutely they deserve more money in a liquidity event than I do, and I'd be a fool to begrudge them that.

That said, not all companies are created equally, and many get to market / revenue / profitability at different timetables with different employee counts, so YMMV. That said, the notion of 'fair' is so abstract in this scenario then I think it's safe to assume that it is universally convoluted. Hard to pin down exactly what is or isn't fair without internal knowledge I don't have.

It's not about the amount of effort that you've contributed, it's about the amount of risk you've taken on. The larger the company gets, the less likely it is in general to fail. So of course those who join sooner get more money - they took on far more risk than the guy who joined the month before the liquidity event.

As you up the chain salaries are a multiple while stock options go up by orders of magnitude.

In the US CEO/lowest paid worker is ~10x. Meanwhile in Denmark Norway is is on average something like 4x.

I think that is economic nonsense. Risk taking does not merit any rewards otherwise it would be a sane strategy to just maximize the risk of any endeavor.

Despite what some might consider common wisdom, capital and labor are equally valuable. A company will get nowhere without its workers to move it there, and it can't get started without somebody taking a financial risk.

Netflix does this far more than most. They're very clear that they're paying you to perform and that if you don't you will be kicked out ASAP.

You might prefer a company filled with B-players doing B-work for B-pay, but don't hold it against Netflix that they're honest about what they expect.

Which just translates, as many of these Netflix threads do, to constant terror that you're going to be out of a job. I'll take my game, A or B, somewhere else, thanks.

Good point on the terror. On the individual level, A/B/C players is your average self-justifying dichotomy: I'm better than those people because I work hard, I work smart etc. Yet on the system level, it's the perfect rationalization for companies too lazy to hire correctly and too impotent to own up to mistakes.

"Why couldn't these folks go above and beyond like these other teams? It must be that they are inherently not good enough. Fire them and get a new team in."

I would even go so far to say that it's the CEO's and senior management, not the employees, who push this myth the most. In somehow all formulations I've seen, the persons's boss, who is amazing (regardless of context), and her or his bosses who are also amazing, are the judges of all performers in the company. Because the system is perfect or filled with perfect people (read: the top part is), any point of failure must be the employee, by process of elimination.

Further reading: https://en.wikipedia.org/wiki/Vitality_curve#Criticisms https://en.wikipedia.org/wiki/Jack_Welch (the

Nobody good enough to get hired at Netflix is "terrified" that they'll be unemployed for longer than a week, come on.

https://en.wikipedia.org/wiki/Impostor_syndrome

I think the problem is more about sustainable effort.

If there was a /HN/bestof -- I'd submit this.

Might be a little egotistical for me to say, but I do like it when my writing is appreciated.

Thanks!

Read some stuff from Michael O. Church

I'd rather not.

Netflix hires very few entry level engineers. I just checked their jobs page[1] and there are almost no postings for non-senior engineers.

I'd argue that a senior engineer should be highly effective while still developing in her role. Also, expectations for different "levels" of engineer vary. Typically an "exceeds expectations" performance review for "Senior SDE" would be neutral for e.g. a "Staff SDE."

I have no idea how Netflix actually treats its employees, but applying the charity principle leads to other possible interpretations:

- B was chosen arbitrarily to drive the point home and could well mean a D on a school grade scale.

- They don't want to be too hard on fired employees or hurt their future employability (e.g. even "above average" people get fired over there).

I don't know anyone who works at Netflix and I could very well be wrong. Their "performance > effort" policy does however resonate with me.

The difference is that they back that statement up with A-level compensation. Most startups don't.

Also, is Netflix still a startup?

In terms of total compensation are they really that much different from other big companies? I know a fair amount of folks making Netflix level money when you factor in stock. They just give it in cash. shrug

I mention above Netflix pays Senior Software Engineers over $350k/yr. This is far above anything any other company does by a wide margin. I'm not sure how much Google or Facebook employees make, but are they really making upwards of $400k/yr total comp?

An argument I've heard against this is that stock (etc.) is a way to give delayed money; it depends on you being locked in, the company not tanking, etc. The "equivalent" money now is more valuable. I'd love to hear a refutation of that, along with what these salaries are.

Yes, principle of time value of money says you need to discount future money. But deferred compensation is also deferred tax and it can be very useful to be able to reduce/avoid a big tax bill.

Actual dollars? Sr Engineer base pay at $175k+. Mid level at 135k. There are plenty of formerly middle class engineers making over $300k a year including equity. People often discount startup equity while overlooking the very lucrative large tech company equity. (Fb, goog, etc). My source for this is both friends and hiring in this market.

To be sure not every eng in sfba is paid this amount. But they probably could be if they were willing to optimize for comp.

I want to second this about netflix. They pay the same total, but have different requirements.

That's the startup mentality: "we can't develop our employees" because they're being cheap and naïve.

Netflix is anything but cheap when it comes to employees.

> "if you're not giving us everything beyond what we expect... go screw yourself.

I'm not sure how that makes sense. Netflix can't expect employees to surpass their expectations, by definition. I don't see the problem if they're clear about their expectations and send people on their way who either can't or don't wish to fulfill them. I don't know if I have the ability to fulfill their expectations, but I know I have no desire to try and thus no desire to work for them, but I'm fine with other people having different preferences.

It's fairly easy. Business asks for something. Business gets mad when what they asked for was wrong, technology failed, bad luck, or they felt like you didn't do more than what they asked.

I mean, what do you expect them to say?

"We look for a SOLID B- out of everyone here."

I know a handful of people who work for Netflix, and they all love it, although some have commented it's a hire slow / fire fast environment, but once you're out of that initial trial period, it's a great place to work.

Even a b- is above average. Wither they like it or not, there will be average.

Not if they pay twice what the average is (which they do, by most accounts).

You can have the best of the best, you just have to pay for it. Netflix does.

I've always found it amusing when companies say they hire "the best of the best", but offer only market rates or (usually) lower.

When I saw this the first thing I thought of is this Planet Money episode about it "Hard Work Is Irrelevant" http://www.npr.org/sections/money/2015/08/28/435583328/episo...

The levels are not seniority, they are an approach to learning and programming. Junior A level people are not the same as B level people. Hiring B level people to "just finish" what the A level people have done is an invitation to delays and failure.

Can someone clarify what : - A level - B level - C level

are?

The analogy apparently switched from "A" and "B" the grades/marks in school (hence "A for effort", where A is better than B), to something like the "A team" vs "B team", where the A-team is the elite performers and the B is not as good. Also "C-level" was not brought up until you, but it usually means executives. :)

Although, I see there is at least one person out there who has a concept of a "B-level manager" http://smallbusiness.chron.com/blevel-manager-35848.html so I am less sure about my reading of my grandparent comment.

C-level, I meant executives :)

Sounds like the right way to make sure you only attract great team players.

There are really a few outcomes on that: you're going to hire people that will rip each other apart to get the A ranking, your going to get a group of people that are so afraid of screwing up that they're basically neutered, out your going to get everyone cooking the books.

I can't see this requirement on the individual actually doing anything for the team.

This reads less like a technical overview of Netflix's stack and more like an advertisement for working there.

I'll give you my two cents on the matter. Netflix spins a great narrative and ideal -- the rose-colored "meritocratic" dream of Silicon Valley.

I'd suspect the reality of working there is a lot more mundane than the beautiful narrative that they (and by extension, the author of the post) are trying to sell you.

There are probably a wide array of experiences that current employees have, ranging from pure joy to misery, which are contingent upon team, co-workers, management, compensation, etc.

An interesting read nonetheless.

Adrian Cockcroft gave a much better overview of their open source projects here: https://www.youtube.com/watch?v=R2kKmMyqTfc

It's two hours long but the first hour is sufficient to get an overview of their open source projects.

Good call. Thanks for posting. We'll add this to the comments of the original article.

You might want to link to the OSS site too - https://netflix.github.io/

faizshah - posted (and gave you credit) on the original ScaleScale post. Thanks. BinaryRage - The link is there to netflix.github.io on the original post.

Only the UI layer of Netflix is in Node.

Most of the cloud services are JVM-based (as is much of the NetflixOSS ecosystem [1]), and then the Node services connect to the JVM-based services via Prana. [2]

[1]: http://netflix.github.io/

[2]: http://ispyker.blogspot.com/2015/10/towards-being-better-abo...

This isn't a very good article. Judging from the first table, netflix is written in node.js using python and java. :-/

Thanks for the feedback. We heard this a little bit through twitter. Can you reference any links? Happy to make the change. It was a little tricky to put together with out interviewing someone. We want it to be accurate and useful. Please post any reference links here and we'll follow up with an edit.

Well, node.js implies that at least part of Netflix is written in JavaScript, yet the only listed languages are Python and Java.

Someone at Netflix did a talk perhaps it was about Falcor. They use React embedded in browser apps across all devices. So they use Javascript on the front end. They use Node.js as the proxy server / router with a framework called Falcor which is written in Javascript. Falcor queries all the multitudes of services and databases then caches that data in a graph. Falcor reaches out to all the servers and databases, scores or services, that are written in Python and Java.

Picking a nit, they don't use React in the browser across all devices. They actually use an internal rendering engine called Gibbon on some devices with client code written JavaScript with React but not rendering to HTML or a browser instead using their internal engine. Interestingly this has become easier with React breaking out React and ReactDOM.

Interesting to note that Netflix must see Gibbon as a competitive advantage, I've been told its not likely to see an open source release.

https://www.youtube.com/watch?v=eNC0mRYGWgc

OK - updated it on scalescale.com (the original post) and gave you credit in the comments. Good find guys. Appreciate it.

Fair enough. Will modify shortly and good point.

I know they use Scala:

https://skillsmatter.com/skillscasts/5424-netflix-scala http://hanselminutes.com/394/demystifying-scala-with-netflix... https://github.com/Netflix/atlas

thanks for the links! Atlas is on the post.

> It was a little tricky to put together with out interviewing someone.

Why didn't you just interview someone then? There are plenty of folks there who would happily grant you an interview. If I still worked there I would have.

right on. PS cloudnative.io looks cool! Looking forward to playing around with it.

for example, there was reference to the Spring framework - any reference links are appreciated.

Spring even integrated some OSS projects from Netflix into Spring Cloud: https://github.com/spring-cloud/spring-cloud-netflix

This AWS re:Invent 2015 talk by Dave Hahn is also really illuminating: A Day in the Life of a Netflix Engineer (Using 37% of the Internet)

https://www.youtube.com/watch?v=-mL3zT1iIKw

Interesting to me that they use Atlassian Confluence for team stuff. Thinking of testing it out as well...

Confluence is pretty cool, but they suck at bullet lists and indenting code blocks... so basically they suck at a couple of basic UI stuff. Apart from that it seems pretty great.

Are you aware of the {code} macro which formats code?

I am. It does not indent properly though - there is only one level of stable code indentation (the entire block that is; it is possible to indent the code within the block, but I'd like to indent the code frame).

great link! thanks for posting fitzwatermellow. Adding to the ScaleScale original post and giving you credit.

The A (promoted) vs B grade (fired) perf means:

- No "regularly well performing" employees are kept. You're either promoted or fired. Ensure workforce moves.

- Doing the shitty necessary work such as fixing tech-debt just gets you fired. But I bet people do it anyway, else Netflix would be in troubles. It's just that they get fired after that. Not really nice.

It also means you've to be only doing stuff that has direct visible impact, and refuse any work that does not/place yourself visually above others.

Not saying this is a bad strategy to get stuff to work, not sure if very ethical though.

That's all I noticed was actually interesting ;-)

Exactly--in a work environment when everyone is fearing for their livelihood, succeeding at performance theatre becomes the primary concern. You need to be a top performer, in both senses of the word--but it's human nature that over time the acting sense will take priority.

Not sure if it was left out on purpose, but I'd love to see more details on their transcoding stack (they only mention that it is done in EC2)

Hi Kylek - thanks for the feedback. It was a little hard on first try to find information on this. The only thing that seemed clear was that adaptive bit rate was done on the edge and transcoding was done and stored. Feel free to post links here if anyone has any.

This is a bit confusing. I don't suspect Netflix is built using primarily node.js, and I don't see how the Netflix Prize (algorithm for predicting what a person is most likely to want to watch) falls under "scaling".

hey, thanks for the feedback lappa. We've updated the original article to display node.js better and added a comment.

The idea behind mentioning the Netflix prize is to show the openness and willingness to get a lift on their existing recommendation algorithm. There was a 10+% lift in accuracy. Their ability to open up the data and implement really gives them a competitive edge IMHO. It's a shame there seems to be some regulatory reasons they don't do it any more.

This is nitpicking.

Honestly, there are enough problems with the article to warrant a critique. However, someone may not want to take the time to outline every single one. It's incredibly confusing to the reader that the article states Netflix relies on frameworks for one language, and then states that all of their code is written in another language.

Hey, we got this fixed up on the original post- waiting for Todd to fix the repost on HighScalability shortly. If there are some more problems let us know. We want it to be accurate. http://www.scalescale.com/the-stack-behind-netflix-scaling/

Among the things I had knowledge of, a lot seemed incorrect, so I wasn't quite sure how to take the things I didn't have knowledge of.

I am a bit surprised. Why host Netflix application on Amazon EC2 and not have your own data centers? A startup hosting at Amazon EC2 makes sense

Why Netflix chose AWS: http://techblog.netflix.com/2010/12/four-reasons-we-choose-a...

Also Quora answer: https://www.quora.com/Why-does-Netflix-use-Amazon-EC2

The reason that strikes me as most important is #2 "Letting Amazon focus on data center infrastructure allows our engineers to focus on building and improving our business."

Too many companies roll their own solutions when they could take advantage of economies of scale like this and let another company dominate the problem.

Obviously there are privacy reasons to own your own data but I don't think they outweigh the performance and efficiency implications.

> Too many companies roll their own solutions when they could take advantage of economies of scale like this and let another company dominate the problem.

Too many companies build their business completely reliant on a single service provider that can/will change focus, increase costs, terminate/change service offerings, or have unreliable service availability.

aws and performance :) Crappy performance and poor uptime is precisely why people move off aws.

So, um, why does Netflix always seem to be up/performant?

Because they actually know how to implement fault tolerant scale on AWS. Most companies don't have simian army randomly wrecking infrastructure. Many companies/engineers think they can spin up a few (or few hundred) ec2 instances and think they have solved resiliency.

AWS is a tool, not a bullet made of silver.

They are shifting capex to opex. The complexity of solution that can perform OK on aws is significantly higher then complexity of solution that runs on metal. There are use cases for which aws will never be a viable option. AWS in general is a bad option for anyone but Netflix (Netflix consumes over 30% of aws resources if they make a mistake that causes even 30-40% spike all AWS customers are f#$ed)

Thanks.

Why is that surprising? If you're not in the infrastructure business, why would you build infrastructure? It's like criticizing an auto manufacturer for not manufacturing their own tires.

If you mean that AWS prices are too high, I think you're thinking in terms of retail prices. If you have enough volume, AWS will negotiate a volume discount.

Netflix accounts for a huge chunk of the national bandwidth. They have specialized needs when it comes to hosting that no other company on the planet has.

I imagine Netflix has sweetheart deals with AWS that net them significantly cheaper costs and dedicated teams of support staffers, given their size.

I really like that Netflix is constantly open about how their systems are run. In terms of raw throughput not a lot of companies can hold a candle to them, so its cool to see how the big players do things.

I would wager because, looking at that technology stack, most of it is commercial paid services? "We use EC2" just pushes the question to someone else.

So which part of this stack is to blame for their slow clunky website? (NB their website used to work well enough on my devices the newer one is the problem.)

Over the years, I've heard a few podcasts stating that Netflix relies heavily on Grails/Groovy for infrastructure management tools (https://github.com/Netflix/asgard, etc). Wonder why that isn't mentioned?

They do say: "We are sharing the things we think are most interesting." Maybe a minor use of Groovy and Grails is only interesting to people trying to promote those products.

Good call - we'll dive deeper into it and add it.

Is this reliance on a variety of external services why Netflix fails so often compared to e.g. youtube.com? I assume so.

http://mashable.com/2015/11/10/netflix-site-goes-down/

Where is the Chaos Monkey?

(http://techblog.netflix.com/2012/07/chaos-monkey-released-in...)

Simian Army Suite https://github.com/Netflix/SimianArmy

As the comment at the end of the article mentioned, they forgot to add Spring. I've met a few Netflix engineers, and they said their Java implementation is done using Spring.

Indeed the love has been returned. The Spring team have been wrapping and integrating a lot of Netflix OSS into Spring Cloud:

http://projects.spring.io/spring-cloud/

http://cloud.spring.io/spring-cloud-netflix/

No, it's not. It's primarily Google Guice via Governator.

hey Steven! Thanks. Do you have any links that reference this? We want it to be as accurate as possible.

No links, I got this from actually talking with some engineers. Have you not contacted a single Netflix engineer to verify your article? If not, I find that bizarre.

It must be really expensive tho (even with discounts). But I guess they don't care.

This view is missing some things. For example, Curator (now at Apache).

http://curator.apache.org/ "Guava is to Java what Curator is to ZooKeeper" Patrick Hunt, ZooKeeper committer

Source for the bit on Mesos?

check: http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-... and post anything else

A word to the author: Please proof read your writing.

OMG! Node.js in frameworks folks xD

Funny timing given that Netflix is currently down... [0]

[0] http://www.isitdownrightnow.com/netflix.com.html

Why there's so much brag about Netflix while they can't fix even simple bugs and UX issues? After all, it's one of the simplest apps there are. Whole stack could be handled by one guy.

Curious what you mean by this, as it's overall one of the more polished products I use on a regular basis. Hasn't always been the case, and the Android app still does some weird UI things, but generally the product is solid and a pleasure to use.

"Whole stack could be handled by one guy" - comments like like this and similar others ("I could build Twitter in a weekend", etc.) strike me as unproductive.

For example, Netflix consistently forgets what I have seen and if I don't watch the ending credits it thinks I haven't watched an item and suggests it in the continue watching section. There should be visual indicator on what I have watched, and if there are unwatched episodes in series. Now I have to open it and go to the episode list, which is unreliable. Also, starting video playback is too slow. It's not possible to open items in new tabs. There are no keyboard controls.

> Whole stack could be handled by one guy.

wat

Reply on news.ycombinator.com