A Review of Airbnb's SRE Hiring Process, from someone who just interviewed there
redbluemagenta.com
redbluemagenta.com
I mean, I understand why companies like AirBnB would like them to exist. What I don't understand is why engineers participate in them.
I totally get selling yourself to the potential employer. Spending an hour on the phone talking through a technical problem. Having IRL conversations where you see if you "gel" with the team in a bunch of hard-to-quantify ways.
But why would any engineer in today's market take two days to respond to asinine pet/trick questions that random engineers lob at them in order for a shot to spend additional time for people higher up the food chain to lob hypothetically less-asinine questions in a way that makes them directly comparable with the other 19 people jockeying for the spot?
Do companies like AirBnB believe that there are top-tier people with nothing better to do? Or do they believe they have some perk that is different than the perks of every other startup currently hiring that would motivate top-tier people to create such time?
Maybe my perception of market demand leads me to overestimate it. I get so many "please work for us" emails (from actual founders, even, let alone recruiters) that my inbox is out of control. I've gotten a job offer while writing this comment. I can't even reply at that scale. Perhaps one offer a month is lucky enough to receive an actual e-mailed decline; I never even consider doing a phone interview. Am I a precious snowflake, is my experience non-representative?
While I'm very happy for you receiving job offers all the time (congratulations!), there are definitely reasons for an engineer to take an interview; in particular, falling in love with what a company stands for. The idea that all engineers are mercenaries ready to work for anyone and sell themselves to the highest bidder is an aged one, and one I had hoped was no longer true. If I fell in love with a company (as has happened in the past), I'd certainly be willing to jump through some hoops to "prove" myself to them - if that means going through an interview process so they can fall in love with me as much as I have with them, so be it.
When I see a woman I like, I don't go up and tell her about all the other women that want to sleep with me. Instead, I flirt, I jump through some hoops, make her jump through some of mine, and ultimately establish a strong connection. Or not; that happens too. Same goes for finding work.
I love Square's product, but the way they write their tests is totally perpendicular to my philosophy, and would totally drive me up the wall.
I love Square Enix's products, but the life of a game developer would suck all of the fun out of it for me.
The whole thing is subjective, of course, but I don't understand why product fit would necessarily imply cultural fit, and particularly for my subjective criteria, the fit seems poor.
In fact, I can think of many products that I at least moderately dislike that by all accounts have really fantastic people that I would get along great with.
Apple does not return the love, hence they commoditize apps and make building a business off apps, a race to the bottom for $1.99 or free as a result, many indie developer are getting broke right now. Very sadly, the same commoditize apps, help make iphone and ipad appealing to consumers, which in turn gives Apple the leverage to sell them for $500 or more because it is a hardware or device company and software or your app is only a complement to the hardware.
Where is the returned love from Apple. Be pragmatic, even while you choose to love an entity like Apple, that lacks capacity to return the love and will always act in self interest. Always search to see if your love is reciprocated by the person or organization you are in love with, because that is your self interest. Self love gives us the capacity to love others. Self love encompasses self interest among other things.
If Apple acts only in self interest, why do developers choose to act out of love aka blind fan boy-ism that is detrimental to their business (indie app development)?
http://andrewchen.co/2012/08/15/mobile-app-startups-are-fail...
http://penny-arcade.com/report/editorial-article/going-broke...
Because your example with "woman I like" is not valid. There was no woman _you_ liked. There was woman who liked _you_ (recruiter e-mailed). And that _woman_ expressed desire to sleep with _you_. And after that, _you_ strongly advocated to start flirting and "jumping through some hoops" with that woman. May I ask why?
There wasn't a woman who expressed desire to sleep with me. There was a woman who made eyes at me from across the bar. That's it.
His immediate response was him walking over and starting the conversation. I don't know about you, but I'm not lucky enough that every time a woman makes eyes at me from across the room it is, by definition, in the context of "Let's go to my hotel room."
People can purport to stand for just about anything, but unless there is a track record there of standing up for [principle] in the face of [adversity], it's just a politically expedient claim. There are wonderful exceptions, but most ordinary people and most ordinary companies don't deduce their day-to-day behavior from first-order principles, and at any rate, companies the age of AirBnB don't have the track record to enable an observer to distinguish principles from marketing claims.
> The idea that all engineers are mercenaries ready to work for anyone and sell themselves to the highest bidder is an aged one, and one I had hoped was no longer true.
This is not my view; nobody is saying take the highest offer, and I have said absolutely nothing about money at all.
My question is: why do engineers participate in technical interviews? Insofar as I understand it, your response explains why an engineer might (I would say improperly) create a post-hoc rationalization and "fall in love" with the company after-the-fact, but under your own logic it doesn't explain why the engineer takes two days off to go in the first place. Assuming that "falling in love" is even something you can do in two days, the candidate hasn't done it at the point this decision is made.
I suppose my point boils down to the following:
1) If you already love a company pre-interview (which can happen, by the actions of the company), then it makes perfect sense to take an interview. Even in the short lifespan of most startups, there can be enough action and culture shown that you can truly desire to work there / "fall in love."
2) Sometimes it's worth taking an interview to find out more information on both sides - a phone interview is useful, but as you say, IRL conversations are far more useful for the things you can't see on paper or hear over the phone. I'm not saying AirBnB did a great job with this - they clearly didn't - but that doesn't make all interviews useless. Without them, a lot of "bad fit" hires would be made.
That is why even when you kill your self for a product launch or their overall success, they still don't hesitate to give you the sack if it is required for something like cutting cost not to talk of when major adversity comes.
My point is be pragmatic and not too emotional with loving a company especially if you are not a founder or major share holder. Many people have been burnt by loving someone or an organization that does not have capacity to return that love.
SpaceX comes to mind (I was contacted by a SpaceX recruiter after I performed well in Sebastian Thrun's original AI class, but it never went anywhere after that).
However, I completely agree in principle that a 2 day interview is just ridiculous.
I don't really think it's the presence of an interview process that the OP is complaining about. Rather it's that the interview process can take 2+ full days.
Do a phone interview, a 2-3 hour relevant work sample test and a 1-2 hour in person interview--then if you're still want more, pay me for a short project (that I can complete nights and weekends).
After the Nth email from someone at Google, I got curious and said, sure, we can start the process.
The first phone screening was like "Unix Trivia Time". I knew lots of things, but some others just weren't things that matter to me day to day, so while I know they're out there, I don't recall the specifics. Generally, I'd know what man page to go look up, or I might also considering a certain search engine to find information. The interviewer said that I wasn't supposed to do that. This went on for what I recall as an hour or two, and I did pass, and got invited to do some other interview.
But the more I thought about it, the more I realized I did not want to spend any more time doing stuff like that: they don't have offices anywhere near where I live, nor any offices for technical people in parts of Europe that I consider pleasant places to live, and I don't want to live in California either. And supposedly it's pretty difficult to land a job there anyway, so the odds would be to spend N evenings/hours doing more silly interviews just to get turned down, or even accepted and be faced with the prospect of moving somewhere I don't want to be.
So I politely let them know that it had been decided that Google was not a good fit for my current employment needs, and that I wished them luck in their candidate search.
Clearly, this isn't the case.
airbnb seems to be missing the mark. tools come and go. the 'right answer' doesn't matter as much as the reasoning.
the ability for one to assimilate him/herself quickly and productively into the environment is what they should be looking out for.
So I go in and first question I hear. "can you describe how is TCP/IP built". This is for web development position, you know code igniter, php, jquery, that kind of stuff. I looked at them for about 45 seconds, got up and said "thank you for your time" and left before they had a chance to say anything.
Dude, wtf, are you serious? Its not like i couldnt remaind something about tcpip from the times of my b.d. 10 years ago, but seriously unless we are going to invent new transportation protocol for the whole human civilization to use, why on earth would you ask me that, when you hiring me to be a web developer. What a joke!
It's really hard to know the right way to hire SRE's/sysadmins/systems engineers, from a startup's perspective. If your founders are highly technical, they're more likely to be developers, which is a similar profession but frankly different enough to note. In some cases they will be designers, or non-technical people entirely. As your company grows, you need to hire developers first, so at best, you find a (decent) solution to that problem. As we already know, though, nearly every company that interviews and hires developers still does a shitty job at it--not because it's their fault but because it's a genuinely hard problem. It's only after you've grown awhile and your developers get tired of doing all the ops shit that you think to hire SRE's. But at this point, your culture and hiring practices are already pretty set, and set by people who aren't actually SRE's. It can be fantastically difficult to rollback this kind of momentum even if you do manage to get a top-notch SRE or three onboard.
There's another side of this, though. It's both a root cause for the problem and good news for Airbnb: Airbnb's competitive advantage is not going to be having the best SRE's. They're not AWS, Heroku, or Google. So it's not really a problem if they're bad at hiring SRE's, because most companies get away with being bad at things that aren't central to their business. What Airbnb gets right, they get fantastically right. This isn't to say they shouldn't do a better job hiring SRE's. They should. But they're still a great business despite this.
I find it remarkable how many skilled engineers reach for the toolchain first and then find problems to solve with whatever tool they have chosen. Even worse, they often reshape the problem to fit a particular tool and then solve it badly.
One of the downsides of the Github/BitBucket open repository movement is that there are now zillions of tools for these engineers use to go looking for problems.
Kind of reminds me of the database as a hammer story:
When I first wanted to spend some time learning about relational databases, my then boss told me a story about database people, which I’ve found to be remarkably true. The story is, RDB people have found the most beautiful, wonderful, perfect hammer in the whole world. It’s perfectly balanced – not too heavy, not too light, and swings just right to pound in a nail just right every time. The grip is custom-made, fitted to the shape of the owners hand, so that they can use it all day without getting any blisters. It’s also beautifully decorated – encrusted with gemstones and gold filigree – but only in places that won’t detract from how well it works as a hammer. It really is the greatest hammer ever. Relational database guys love their hammer. It’s just such a wonderful tool! And when they make something with it, it really comes out great. In fact, they like it so much that they think it’s the only tool they need. If you give them a screw, they’ll just pound it in like it’s a nail. And when you point out to them that dammit, it’s a screw, not a nail, they’ll say “I know that. But you can’t expect me to use a crappy little screwdriver when I have a magnificent hammer like this!”
http://scienceblogs.com/goodmath/2008/01/22/databases-are-hammers-mapreduc/Did you mean to say "open source"? I am not familiar with the github movement.
I've seen many repo's that aren't open source in the GPL/BSD sense but are open source in the "here's the code now its your problem" sense. Sorry if this has resulted in confusion.
How many people have actually been hired after going to one of these? Every job I've gotten has been through networking, back channels, and a token interview.
2+ days of anyone's time is too much to ask without compensation.
I'm curious about this. I'm only vaguely aware of either system, but I thought they solved rather different problems. Chef handles system configuration (like things in /etc, or "configure memcached to listen on this IP and port"). Zookeeper is more of an API to deal with distributed locking/sync issues for an application itself.
Was this interview question less of a "bingo" and more of a "one-side really misunderstood something"? Or how do Zookeeper and Chef compete?
My point, however, was that tools should never be the focal point of solving a problem - if you recognize the problem first as a distributed locking problem, or as a configuration problem, then you can start deriving a solution (and maybe that solution might have Zookeeper in it, maybe you'll write a new piece of software, maybe you'll use Chef, etc.)
The issue here is software push and service orchestration. How do you do a rolling restart of a cluster in an automated fashion?
As for 'configuration management', I feel like puppet, chef, etc arent catch all solutions for 'i want my config to be the same on all machines' for the above mentioned 20 minute cycle.
I think it's important to share interviewing stories. Both for other startups to learn from, and for future employees to know what they're heading into.
Or is it not every manager's wet dream to have good guys right out of university who won't hide their true skills by being incompetent/nervous at interviewing?
Conversely, is it not every young guy's wet dream to be able to display their true skill in an interview, rather than mess it up by being bad at interviewing?
Where are they to learn if nobody shares the stories?
What conclusions would you draw yourself, if you were about to hire this person and that post came to your attention?
If I was hiring and I googled the persons name and found this, I would perhaps be slightly concerned. Not alarmed enough that it on its own would sway me either way, but I would probably pay a bit more attention to the persons attitude in general. To me it suggest either a lack of courtesy or an overblown ego.
If you really dig into it, you'll find huge variations in pay for similar skilled, experienced, and hard working employees just based on who did the hiring, how much was asked for, and when they were hired.
When I was in college I had a job where I made about 50% more than most people I was working with--I didn't really work harder (everyone worked hard) nor was I much more skilled. I just negotiated a bit better, and was hired by the right manager.
On a scale of misbehaviors, this is probably far on the "good" end, but still. Who should gain from this post? If Airbnb, why not mail them directly? If he is "warning" potential hires, aren't he in effect hurting Airbnb? If he wanted to discuss hiring practises in general, why not anonymise them?
And a substantial benefit to anyone else who reads this and can make a more informed decision before deciding to interview at airbnb.
Also the job == relationship metaphor breaks down fairly quickly. What applies to one doesn't apply to the other just because you say it does.
Perhaps they really were looking for someone who already thinks they way they do, and that is valid. they probably dont need you in their office for a few weeks saying "i could do this in chef if..." until they fired you for "poor fit".
Also dont forget that tech/aspy folks just are... blunt in a way that most people interpret as "insensitive". Maybe they're just being efficient? You write a blog, seem pretty eloquent, so I'm guessing you are NT, so perhaps not a good fit after all.
NT? (edit: you mean non-technical? I'm definitely not non-technical - I work as a systems engineer by trade.)
And, yes, you don't need to use Chef or Puppet to solve the software deployment problem. You use it to ensure system state, so you know you can throw software on top of it without any problems. (this can go on into a debate about golden images vs. configuration management, but I think this is a different discussion.)
That's probably true as well, though... I've been in shops without Chef or Puppet, and it's really not a big deal for me, either. The thing that stood out was the justification for not choosing Chef or Puppet, not really the fact that they weren't using it.
On a technical note, it's not clear to me how you KNOW that chef or puppet has solved your system state problem. Now if its limited to doing super basic stuff, ok great, but then you really arent adding enough value to justify the huge cost of these systems.
I'd also like to hear what you have to say about "throw software on top of it"... for example, I want all 40 webservers to have the same build revision of our codebase. But I need positive feedback - and a gradual rollout possibly. Eventually run puppet clients really dont tell us this. Oh wait, I mean I want to rollback to the previous deployed version, because it's broken horribly.
These practical software deployment issues I find just are not addressed by the system configuration management tools like chef, puppet.
And that is why they are so passionate about it, because a million people have told them 'just use chef' and yet chef doesnt solve their problem at all, nor does it solve any fraction of their problem either.
I'm a fan of both of those systems, because it's easier to express system state with them (and clearer, too), versus using shell scripts or what have you. I'm definitely not advocating using it as a way to deploy software, but it's definitely possible to do so (I'd likely build system packages, throw them in an internal repo, and use yum/apt to ensure software packages are up to date, or at a certain version.)
It's not the only solution, though. I've worked at shops where there have been exceptional software deployment pipelines that are absolutely not tied to a configuration management system. I'm also not a fan of using Chef or Puppet for doing a git/svn repository checkout of software.
Again, I want to reiterate my point: it's still not about the tools, but about identifying the problem you need to solve and discussing possible solutions to the problem. There's no right answer.
So you said "use yum/apt to ensure software packages are up to date, or at a certain version."
So how do you make a 100 node cluster get to the desired installed state? Do it by hand? Ssh loop? cssh or similar tools that attempt to do parallel ssh? This is the part where things really break down and then I realize there actually is no principled way to do it, and in fact it's done by hand, and the sysadmins are just creating more work for themselves ensuring job security.
It's also a bit insulting when you say that we 'in fact [do it] by hand', because sysadmins, as a whole, strive to not do that. Fine, I can see myself logging into a box and typing 'chef-client' or 'puppet' by hand - but this is too much work to be done by hand. That's why we automate. I don't know why you have this idea that we simply want to make things harder just for job security.
I have a destination in mind, and I'm always looking for new travelers. But I havent seen many lately.
May refer these guys http://labs.agriya.com/burrow