Adopting Microservices at Netflix
nginx.com
nginx.com
Christ on a bike, that sounds horrific.
All this approach can do is replace codified guidelines with unknown, arbitrary ones. Who defines what "NetFlix's best interest" is? In an organisation of that size, conflicts of interest must happen all the time.
Is this just a fancy way of saying "If we don't like you, you're out of the door?"
"The more talent density u have the less process you need. The more process u create the less talent you retain.
-- Reed Hastings (@netflix)"
An 'untalented' developer is one who is perhaps slow to complete features, needs work on their design patterns, produces buggy code, etc.
Process isn't needed for these people, rather it is for people who do or try to get away with things like: expense excessive purchases, frequently lie about being sick to get out of working, speak or act in a lewd or unprofessional manner, etc.
Nobody is under the impression that mandating wearing a business suit (process) makes one suddenly better at programming (talent), the converse is equally absurd.
edit: punctuation
I read it like I would want to work there in a heart beat. Because it seems that Netflix trust you and doesn't (micro)mange you on bullshit KPI's or something.
It seems to me that at Netflix, you are trusted. You are trusted that your work improves the environment. Your work is evaluated afterwards. There is a risk of things going wrong, but with good, responsbile, knowledgable people, this is not so high.
Part of this is that you take the initiative to set expectations and clearly discuss what you are planning to do and what you'll be working on so there is no misunderstanding. It's all about setting expectations and communication.
I'm actually quite curious how life would be inside this culture. Since you have so much responsibility and control, life should be quite bussy, but manageable.
The ideal would be that you just have a bunch of adult, professional people who manage themselves just by talking to each other, agreeing on how they want to work with each other to get things done and they manage / steer this process themselves just with open communication. No (micro)managers needed.
The key is the people and their values / culture. Very simple things may make all the difference. Being on time. Being prepared for meetings (reading the short memo upfront, maybe lookup some things or make sure that you have done the tasks required of you). Starting the meeting on time. Respect each others time.
A willingness to to share and help people understand how tings are setup and why (reasoning), document the stuff that is important. And you know what is important or are capable of just asking what is important for other people to know so what you document is relevant.
I can only dream.
http://www.brendangregg.com/blog/2015-01-20/working-at-netfl...
If I can ever get my own business off of the ground, I hope to have it work similarly to what you described.
I've never worked for Netflix or any other company willing to make such bold statements so can't speak to the reality. It would be interesting to read a first-hand account from someone who has been at Netflix since the time such non-policies were enacted. At the time, it sounded like a fantastic experiment. I would love to see the results.
There are two major ramifications of unlimited vacation policies:
1) It changes how you feel about taking time off. Rather than paid time off being something that you have "earned", you evaluate your PTO in comparison to your teammates - specifically most people try to make sure they are not taking off more than the average. The end result being everyone is subconsciously competing with everyone else to consume less PTO than average.
2) Since you have not accrued vacation time, your company does not owe you financial compensation for vacation time not spent when you leave. As someone that changes jobs every 3 or 4 years, this is more painful than it sounds.
If I ever decide to be in a position that affords me the opportunity, I think I would enact a "minimum vacation policy". It's still unlimited, for whatever that's worth, but employees will be required to take a minimum amount of vacation every year.
For #2, I've worked for several companies, only one of which had a compensation package for unused vacation. All the rest have been use-it-or-lose-it by the end of each year. Getting paid for unused time sounds like a luxury.
But I also recommend negotiating for more time off when you sign on, especially if you have several years of experience in the industry. Even if a company says first-year employees get X days of vacation you can still negotiate. Especially for cash-strapped companies, vacation may be the only part that can change.
So, having a policy of "take only what you need when you need it" would be akin to giving everyone access to the same bank account that runs the company and trusting them to use only what they need and/or feel is fair.
It's worth noting that in some sectors (including the one I work in), minimum chunks of leave are considered an anti-fraud measure. A non-trivial amount of internal fraud is uncovered when someone is required to take 1-2 weeks off in a block, which is enough to require someone else to stand in for them. At that point, oddities get called out.
I've previously asked co-workers and even bosses who work too hard to take time off for exactly this reason.
> A single specialist in Java distributed systems is managing the entire configuration without any commercial storage tools or help from engineers specializing in storage, SAN, or backup.
So this person is either a genius who has specialized knowledge in these areas, or it's been abstracted by the platforms people to the point where it's only Java knowledge. Either way, it sounds like this person never gets to take a vacation or be not on-call. Hopefully there's not just 1 person responsible for all of the configuration, it just takes 1 person to run it.
> Process prevents problems. At many companies, the standard response to something going wrong is to add a preventative step to the relevant procedure.
The linked section goes on to talk about HR process with the "Act in NetFlix's best interest". It totally avoids the concept of technical processes. I'm betting "best interest" is a bit more formal on the tech side. Bad process is process for process's sake. Good process serves a very important role for offloading what people have to remember, as well as knowledge transfer. From time to time, steps of a process become irrelevant because of other changes and it has to be pruned. Sometimes companies forget about this step.
If Netflix has ever seen a problem with their builds/deploys and added a test to check against that problem, guess what--they followed the standard response to update the procedure. It's not necessarily a bad thing.
Now, what you have to take into consideration is that it's written as a contrast to all kinds of other places that run a command structure where it's your boss' job to figure out what is good for the company.
I don't know about you, but I have worked in those environments before. They attract bad people, as doing what seems better for the company, as opposed to what is better for your boss, is discouraged. I think of one example in my past, where the team worked for two months on a feature we knew our in house customer did not want, and would refuse to use. However, it helped our boss' internal political status, so the team tolled, the feature was unused, and in the end, half the team quit. Our former manager, however, is still doing well, because what is good for the company was not what was good for his boss either.
I often get recruiting calls from that company. They have trouble retaining good talent. Imagine that.
So the Netflix approach brings in people that are more dedicated to the business than the chain of command. There's a lot of very good engineers there, and it's a pool that is often untapped outside of startups. It's not a bad strategic decision, IMHO.
I can easily see just a couple rotten apples spooling that who could damage that whole system very quickly.
I'm sure it is, but is there any company where that isn't the policy?
Source https://hbr.org/2014/01/how-netflix-reinvented-hr/ar/1:
"Hire, Reward, and Tolerate Only Fully Formed Adults
Over the years we learned that if we asked people to rely on logic and common sense instead of on formal policies, most of the time we would get better results, and at lower cost. If you’re careful to hire people who will put the company’s interests first, who understand and support the desire for a high-performance workplace, 97% of your employees will do the right thing. Most companies spend endless time and money writing and enforcing HR policies to deal with problems the other 3% might cause. Instead, we tried really hard to not hire those people, and we let them go if it turned out we’d made a hiring mistake."
I trust myself, and everyone on my team to act in Netflix's best interest. Throughout my career, I've worked with many professional engineers who could have been trusted to work in the company's best interest. But there have been many times when we've been prevented to do so, due to process. So I've been fascinated at how Netflix is operating, and it's been great working here.
It's important to know that a key role of management is to provide context to employees: what problems exist, what challenges we are facing, what opportunities might exist, what's important to Netflix right now. So that we know what to do, and how to exercise judgement.
I've written about working at Netflix earlier this year on my blog (someone else posted a link).
http://en.wikipedia.org/wiki/The_Tyranny_of_Structurelessnes...
We'd all like to work with people who operate with the same understanding, ethos, sense of humor, etc. It makes the day better, right? No office drama, no awkward meetings. But I would contend doing this tends to put you at a genetic disadvantage, and that some types of confrontation are essential for growth. Introducing only like-minded people will produce linear results.
What codified HR rules do is put some workplace guidelines around humanity - that allows a dissonant group of varied people to, you know, be themselves without necessarily worrying about conforming to avoid ruffling feathers.
Perhaps it's more nuanced than that, but it sounds more like "hire people you like" expressed somewhat condescendingly as it is.
I hate to be the one who says this, but that's not what those guidelines do. They don't allow varied people to be themselves. They make everyone be a self-similar group of other people. Other people who are not themselves.
Expressing competing professional views is indeed healthy, and it's part of our culture deck: http://www.slideshare.net/reed2001/culture-1798664
slide 13: "You challenge prevailing assumptions when warranted, and suggest better approaches"
slide 14: "You say what you think even if it is controversial"
Fully formed adults have confrontations all the time. The difference is that the result of the confrontation is not threats, backstabbing or whining later. We've all worked with the engineer who thinks they are the smartest thing around and acts like a 2 year old every time they are challenged. That's the person I assume Netflix says they will not tolerate, and I agree with them.
I have heated discussions with co-workers all the time and at the end of the day we are all still respectful of each other and our ability to get the job done.
Sure. But without some HR codifications the line between "fully-formed" and "childish jerk nymph" becomes wholly subjective, and controlled by the majority. The end result is an organic uniformity.
Nobody gets along all the time. But how you define "fully-formed adult" can vary from one person to the next. Essentially you're defining "good and bad" or "right or wrong" by a feeling. I can understand the appeal of that, but it would worry me quite a bit as even a loose policy for behavior and interaction.
"Fully-formed adults" are people who I would expect to know how to mediate themselves. If people aren't able to successfully mediate their own disputes, then your failure in hiring came long before your latest addition.
Examples of people likely to be fully-formed adults: a mother with multiple children. A schoolteacher. A military sergeant. A nurse. A bartender. In general, people who have been exposed to enough pointless complaining and dispute that the object-level arguments don't matter to them any more, relative to the issue of figuring out what will allow everyone to continue working together optimally.
Note how most of those jobs are service jobs. People with natural talents rarely have to grow up in this way, and the people with the most natural talent are frequently the least grown-up (rock stars, career academics, startup founders). You only see this "fully-formed adult" trait re-emerge at the highest levels of accomplishment: astronauts are frequently fully-formed adults, for example.
An obvious example is about how you use your computer. In my old job, it was highly regulated. Want to install Firefox on your machine? Sorry, that's against IT policy, we only support IE. Your machine is locked down to prevent unauthorised installs, and even if it weren't, you'd be up for a warning for violating the policy.
In my current company, the sysops still limit what they support - for example, they're not willing to support OS X Yosemite users yet, and recommend users stay on Mavericks. But, if you want to upgrade to Yosemite, you can, just don't ask the sysops for help fixing it. And if your broken Yosemite install is impeding your work, well, we trust that you'll resolve it one way or another.
Or another - alcohol. In the government, alcohol at work was banned. A beer at lunch could get you fired. When we had Christmas parties, alcohol was provided, but on a token-per-drink basis. Everyone got two tokens. My current company, the beer fridge is a treasured perk. There are no rules about how much beer you can drink, or when you can drink it. So far, no raging alcoholism has impeded work.
What I really notice is that when you treat people like adults, they respond like adults, and when you treat them like children, they respond like children.
Sorry, but as someone who works in IT security, this sounds like an absolute nightmare. Even if you have a small company comprised only of intelligent developers, those developers do not necessarily understand the latest malware threats or what sorts of software can introduce risks. Wide-open FTP servers and Tomcat servers with default passwords are a major issue. I would actually say developers probably introduce more threats into our environment than any other demographic.
Whitelisting software installs from specific domains (google.com, mozilla.org) is okay, but a carte blanche policy is usually a very bad idea.
Even when we account for HR being an organ doing the bidding for the parent org.
The issue here is that some (many) people need more guidance, because it's not their company, and they have no idea what it considers "the right thing". Should you hire such people, you can't just dump them like thrash because you can't be bothered to manage them properly.
Netflix is going to get into a truckload of legal issues if it tries to scale this practice internationally in countries with decent employee protection laws, which is most of the Western world outside the US.
Netflix sounds to me like a Randian cult.
I don't think they are saying "we have zero policies whatsoever", but exaggerating the level of trust. For instance I am sure that new parents are given a fixed amount of paid leave.
I think it sounds great to not have a lot of pretend-policy that does no good other than pretend to cover the ass of inept management.
IOW, I'd rather work for a structured hierarchical firm, where politicians brown nose managers and the managers firewall these people from the rest of the team.
edit: Hum, your name seemed familiar. I attended your LISA talk. You're brilliant and well known (witty to boot, as I recall. :) ). You, sir, will be listened to by default, so you will tend not to encounter the failure modes I described above. But do think about them, okay? Tyranny of Structurelessness really is a great read.
It helps to have technical management, who are able to understand all sides of a technical argument, regardless of how those sides were presented. It also helps to have staff who all can communicate effectively, at least at some basic level.
Tyranny of Structurelessness is great. But I'd like to see a more focused analysis of tech culture and its failings.
"Best interest" is hardly a measurable or even agreeable term.
It's an ideological wet dream. It's like saying, you can make any choice, you have the freedom, just make sure you make the "right choice".
Not to mention what happens when netflix's best interest conflicts with outside interests.
The company is Nordstrom, a retail store that has been around for more than 100 years and comes up often in "Best Places To Work" articles. It's pretty great.
Every functioning organization requires faith in the people you work with.
[1] http://www.slideshare.net/HubSpot/the-hubspot-culture-code-c...
I've never known a developer not to speak up when the development cycle is too fast or a user who is not averse to the resultant buggy software.
Pushing code too fast is clearly a problem that causes bugs, and neither developers or users are a fan of.
The 'development cycle' includes the total time to implement something, including time taken to fix resultant bugs.
Where you can speed up the development cycle without introducing more bugs (eg. by implementing comprehensive testing and QA practices), then no one is going to complain.
From the talks I kind of inferred that sharing is a good example of micro service, but that's also a good example of something that one could outsource to addthis, sharethis and similar (at least, I'm speaking from the perspective of a relatively young startup).