Engineering management at Facebook
algeri-wong.com
algeri-wong.com
When a company does its own support with internal personnel, it has a vested interest in making the customers happy and also serving them as efficiently as possible.
When a company outsources its technical support, there is no real incentive for the outsource partner to let the company know how the processes and tools can be improved and automated. In fact, quite the opposite. If doubling the business means double the support costs, that works out great for the outsourcing partner.
Perhaps it's still possible to outsource the support work, but still care deeply about the details of how that work is done, and how the process can be improved/automated with systems. I haven't seen it work that way, however.
IMHO, this is the way to have an effect on a company completely disproportionate to any other activity I know of. The tools I wrote at Apple have enabled projects which, AFAIK, are completely unheard-of at any other tech company.
That said, don't expect a payoff proportionate to the effect of the tool.
From a company owner's perspective, however, excellent tools can provide an advantage that is difficult for competitors to match. That's worth an awful lot.
It's great to invent your own projects at work.
This hubris has certainly been a powerful ally to Facebook's founder, but like so many other powerful people, companies, governments and organizations that came before them, I can't help but think it will ultimately lead to demise (unless tempered).
I think giving clear statements of your opinion and being proven wrong is much better than obscuring your point with "maybe" and "personally" and "IMHO". That's depending on the context of course, as you don't want to unnecessarily offend people (which is bad in general, but also counterproductive). But in a general analysis like this, not attacking particular people's shortcomings? Go for it.
You also included some conditional statements with, "depending on the context" and "But in a general analysis". I see no problem with acknowledging that things aren't always black & white.
I have to single this out: must we all prepare our rhetoric for the remedial education of those who were never made self-sufficient in the processing of external knowledge into internal beliefs?
I'm sorry, I just sensitive to how many times I've seen "top priority" in somebody's management presentation, as if it solves something.
That said, I mostly agree with the gist of what he's saying here.
In particular, it is damaging for a leader to declare a top priority and then go on the same as usual, not making extra efforts to invest in that priority, because of course the reality is that so many things are important, and they lack the will to focus at the expense of something else.
If something is that important, you'll direct significant resources to the goal and I imagine the author of the article is sincere in doing that. Many organizations however are resource-constrained and must make tough choices. It's just mismanagement to declare lots of top priorities - heck, you don't have to say anything, just show us how you are allocating the money and people, and adjusting your expectations of everybody's schedules and deadlines.
Also, I'm not sure if you used the Facebook API when it was new, but I remember in 2007 being distinctly blown away by how functional it was and how far ahead of the curve they were on web APIs. You can argue that "the best" engineers wouldn't churn out something so ephemeral, but that's subjective.
If "it's [everyone's] job to say no-hire" when they're "not sure" about a candidate, I'd like to know what is done to avoid systemic bias in hiring decisions.
Anyone here from FB able to comment? How diverse is the work force, especially compared to applicants?
I believe my statement might be clear. What I mean by that is "it's [everyone's] job to say no-hire," as opposed to anyone feeling like "well, I'm not sure about this candidate, but I'll let someone else reject them because I don't want to be perceived as mean." One problem with hiring is that everyone must be willing to uphold the standard - if you have too many people who are never willing to be the one to say no-hire, the standard will fall - everyone has to be willing to take the responsibility for saying no.
That said, systemic bias is an orthogonal (and real) issue. Being "not sure" should be applied to "does this candidate measure up" not "is this candidate a lot like me?"
Although it appears orthogonal, if everyone starts making hiring decisions without any checks and balances that an HR person would put in, I'd assume more bias to creep in. At the very least, I'd want the "standard" to be clearly defined.
Does anyone consider the possibility that there might be really capable people who don't fit the model that N developers have of what a good candidate should be ?
I guess it's also true that the cost of a false negative is unknown and not felt. It would be a really extreme case to be forced to say, "Hey, remember that woman we didn't hire two years ago? She went off and founded Company Y that's now eating our lunch." With a false positive, you feel the pain of cleaning up their messes until after they're gone.
Finally, sad to say, the majority of job applicants are not competent to do serious engineering work. A negative bias is right most of the time.
I've often heard this claim. I have no data for denying it.
However I have a suspicion that when people make this claim the observation that it is actually based on is that "the majority of job applicants aren't very good at answering programmer interview questions.". They are hence making an implicit assumption that not being good at interview questions implies not being competent to do serious engineering work. I think this assumption is highly suspect, because of the extreme differences in stress levels, time scales and general context between interviews and actual work situations.
And if they haven't contributed any open source code, then whack them with stupid programming questions.
http://news.ycombinator.com/item?id=1966689
The responses were interesting, ranging from "that would never work" to "that's what we do and it works great".
1.) Employees were agreeing to hire people who weren't as skilled as they are because they were afraid of newer and younger competition. As each generation of new hires were brought in they were successively worse than the previous. The most senior employees saw the degradation of overall talent level and eventually left for greener pastures. The company slowly ended up becoming composed of people who were barely qualified for their positions and several years after I had left they were bought out for pennies.
2.) They were sued for discrimination, it was a ridiculous accusation, but the company had to settle. The reason was that without consciously trying we had all hired people who were of similar backgrounds to ourselves in many regards. Apparently "One of our team members wasn't sure about her" doesn't cut it in court.
I have a very strong background in low level (embedded as well as device driver) software development. I have a long and visible track record in U-Boot development, including being a sub-repository "lieutenant".
Canonical advertised for a "Ubuntu Mobile Developer - ARM" developer for which I felt (and still feel) I was a very strong candidate. I submitted my resume and got no response. I did some searching and found the contact within Canonical who was responsible for the position and got an immediate response when I contacted him.
I did a phone interview with a Canonical employee, not the person hiring but another who was a technical expert. I thought the interview went well overall, but he seemed disturbed that I had not actively used Bazaar even though I had used other DVCSes, including evaluating several of them for U-Boot - I ultimately recommended git for U-Boot. I've also used Mercurial extensively in my job, so I would say I have more than passing familiarity with DVCSes.
Anyway, I didn't "make the cut" with Canonical. I've since gotten a new job and am no longer in the market, but Canonical is now advertising for three software engineers for that position http://webapps.ubuntu.com/employment/canonical_SE%20PS%20HB1....
The kicker? Canonical did not have "familiarly with uBoot" [sic] as a desirable qualification for candidates before I applied. They added it immediately after I entered their interview process. In other words, Canonical did not even know they were looking for me before I applied, discovered that at least one of my qualifications were important because I applied, and yet did not hire me. A year and a half later, not only have they not filled the position, they now have three copies of the unfilled position.
Go figure.
But Facebook is still young, they're still learning. Unfortunately they don't seem to be learning from the past, which means they get to repeat everyone else's mistakes.
This is backwards from many technology companies where internal tooling is often given a lower priority and the work assigned to less skilled engineers. This is often done with noble intent, like to try to create the best value possible for customers via improving customer facing features. But I think that proves a short term strategy rather than focusing on enabling your organization as a whole. You'd be surprised how many resources become free for feature work when your tools and infrastructure are so good that you don't wast effort on what could be automated or prevented.
This comment is correct. My intention is not to prioritize tools over thinking about anything else. If you read the essay linked from it, I said "your most talented engineers should be working on your tools." So when I say "highest priority," I mean development priority, i.e. you shouldn't have your best people working on features and your second-tier people doing tool work, like many organizations do.
If you want a starting place Robert corams book on boyd is excellent, while Frans Osinga's thesis Science, strategy and war is essential reading for the hard subject matter.
Here's destruction and creation. http://www.goalsys.com/books/documents/DESTRUCTION_AND_CREAT...
One tool to implement that is a mission order (No civilian counterpart), which is where you define an intent to be achieved and operating parameters to be met but not how that intent is to be carried out. parameters are mutually agreed on - and the subordinate has the right to refuse. Usually you leave most of the definition of the intent up to the subordinate. Like "build a database useful for storing graph data" or "Find a way to improve ads customer acquisition".
EFAS culture - An implementation of blitzkrieg for business gleaned from interviews done in the 70's implements this nicely.
Making the organization more efficient is a big win, we can probably all agree. Making the people more efficient through improved systems is a big win, we can probably all agree. Letting the people who perform the process figure out how to make it better is a vitally important part of that, though, as you point out.
But making tools and systems improvement a priority doesn't necessarily mean that we're going to elevate the systems to the point that they become sacred cows even when they get in the way of getting things done.
At most of them it just meant that HR ended up wasting time dealing with crappy tools, which is no big loss, since HR is a usually a waste of oxygen anyway. At Disney and Amazon, it meant that the people producing consumer-facing content wasted a lot of time fighting the tools that were supposed to enable their jobs, and a lot of developers had to waste a lot of time supporting the tools that weren't doing their job.
For example, just two and a half years ago, one of the buyers at Amazon showed me how she set up the relationships so that when you look at an iPod, you get a collection of compatible accessories. She had to create relationships for EVERY permutation between each accessory and for each accessory to the iPod. By hand. In a spreadsheet (Excel, I think.) Then she uploaded it to the site, and every once in a while a relationship would mysteriously fail, because of another relationship managed in another tool that she had no access to conflicted with it.
In order to efficiently address a disruptive innovation, a corporations need to change their processes and since tools are connected to processes, these good tools make that change even harder.
Yes, it good to have good tools for developing the current product, but organization needs to be ready to eliminate internal tools without hesitations.
To get maximal value out of the tools, everyone has to be willing to toss them when they start to becoming a big enough impediment. (Deciding on that isn't always easy, of course.)
Too often the environment is hostile to what might be a risky investigation. I've spent a week investigating a tool to potentially save four weeks. I failed, as I hadn't sufficiently understood the complexity of the problem. But ultimately that failure led to a better understanding. I changed the way we carried out the task, saving us a week anyway.
My project manager at the time thought that week was a total waste of time. (Thankfully he was later "let go" when 360 degrees of people who had contact were saying the same things). If I do that task again, I will have a go at v2 of the tool, and probably get it right.
Too many people think tools are a waste of developer time. It appears that the best tools where I've worked have been created in developer's free time, and then used widely by the team, almost to the shock of the management. Only then do the developers get a little work time to maintain or even improve them.
But yes, you can go the other way. In the largest organisation I have worked in, there was the "infra" team. Their job was to provide the environments, from source control to db instances (Oracle ones), test environments (multi VM set-ups), etc. However their removal from the actual development team resulted in tools and processes which took about as long as for us to do these things with our own tools... the same ones they'd taken to improve.
It's scary to how many project managers I've had to explain why I have allocated 20% of my project planning to writing a new tool. They see it as wasted time because they don't think that we will be redoing the task again... when it's something sales try and sell with every project?!
The biggest thing that struck me were the strong statements about technical managers. I think the sentiment is correct, technically experienced managers are great. But it reads like you expect all technical managers to be up to date on their coding? Or just be competent at writing pseudo-code? Hopefully it is the latter!