An inside look at Google’s data-driven job interview process
washingtonpost.com
washingtonpost.com
1) I graduated from college long time ago, and have worked in a BigCorp for 6+ years, where the toughest technical problem all center around CRUD, nothing that requires any deep algorithmic, machine learning, or NLP techniques (ala building a search engine).
2) During my free time, I hack on my startup, and although the problems are more challenging, I tend not to spend too much time trying to optimize things, or come up with the most efficient, elegant solutions because of time constraints. I just ask myself: is this good enough and move on. Because of this, I develop bad habits that would never be allowed in Google.
Can't win if you don't play :)
Actually, considering the horror stories we have heard about Google's hiring process, it might be more appropriate to say: "the only winning move is not to play."
For example we have a fairly "locally optimized" single column spine for something that stands upright, but if we were to design something that stands upright (such as a building) we would rarely use a single column.
Anyway, I wonder what Google does to mitigate such a threat from their specific systems described in the article.
Further I wonder if they collect any data that might tell them whether their process turns people away, because I've heard lots of horror stories (especially on HN) about the Google interview process. The article doesn't seem to mention whether or not the interviewees enjoy the process. I suspect nobody (at Washington Post or Google) bothered to ask them.
(Not that I can blame Google, I've never heard of a company asking recently hired employees about how they could make the interview process better. But I figured the WaPo would have interest in writing that story)
The article ends with:
> To make sure they don’t miss out on top talent, Google employs a team of full-time screeners to sift through applications. The company would not say how many people are employed in these roles, but it said the group is "sizeable."
I suspect at the end of the day their screeners and recruiters are about as "average" as other companies of their size. Numbers certainly wouldn't make up for quality. Just my anecdote, but here's the first part of the email a Google recruiter sent me:
> I'm a talent scout on the Engineering Staffing Team at Google. I came across your details and feel that you could be the sort of person we are looking for to work in a new role we're hiring for in our Mountain View HQ, called 'Performance Engineer' for which we're looking for a candidate with a combination of compiler, high performance software design and computer architecture experience.
I'm not known for much, but for what I am it's exclusively JavaScript.
> Further I wonder if they collect any data that might tell
> them whether their process turns people away, because
> I've heard lots of horror stories (especially on HN)
> about the Google interview process. The article doesn't
> seem to mention whether or not the interviewees enjoy the
> process. I suspect nobody (at Washington Post or Google)
> bothered to ask them.
Google sent me a survey after I turned down their job offer (back in 2006). Among other things, I wrote "this recruiter is nearly single-handedly responsible for me not coming to work for you".
A couple years later I heard that she had gotten promoted.
Google also hires contractors and interns, and some of those get 'converted' to full time employees, essentially with the contracting/intern period as one big job interview.
if we were to design something that stands upright (such
as a building) we would rarely use a single column.
> The firm recently generated buzz in the talent industry
> when it said it had done away with the notorious brain
> teaser component of its interviews after statistics
> showed the ability to ace them had no correlation with
> success at the company.
Journalists: "$COMPANY gives all its new employees wedgies!"
$COMPANY: "We do not think giving employees wedgies is useful."
Journalists: "$COMPANY generates buzz by doing away with wedgies!"
Typically, trick questions are things like "how many golf balls fit in a bus", "you've been shrunk and dropped in a blender" , or the infamous "why are manhole covers round". These supposedly test the candidate's creativity, but actually just test whether the candidate has read a book of riddles recently.
Adding features to your software can drastically reduce throughput per CPU core. Even simple architecture refactoring for better scalability can drastically reduce throughput per CPU core (just as you have to pick availability versus consistency, you also have to pick scalability versus throughput).
And hence, because it depends on what the software does and how it evolves, the question cannot be answered unless you stress test a CPU core. The question itself, as phrased, is even stupider than how many golf balls fit in a bus.
I participated in Google's interview once. After 2 phone screenings with algorithmic questions I was invited to a full day on-site interview involving meetings with 5 people, all of which asked me to solve algorithmic problems. In the end I was told that I did OK at the first 3 meetings, but not so well on my last 2 (actually I was able to answer all the problems given to me, but my performance dropped at some point due to getting tired and the last 2 people probably had other questions they wanted to ask, but couldn't due to lack of time).
Personally I recognize the importance of algorithms, data-structures and reasoning about asymptotic complexity. It's stuff that some are learning since high-school and should be common knowledge for all of us.
But what about other things of importance, like actually being able to build and deliver functional software? What about things like the quality of the code you write, or being experienced in building scalable systems, or being capable of working in a team, or having personal projects? I was never asked (maybe it was just my chance to end up being interviewed only by people that cared about algorithms).
Of course, it can be said that these practices worked well until now for Google. After all, there are plenty of really capable people working there. But maybe that happened in spite of the interview process, not because of it (e.g. one reason could simply be that resumes with internal recommendations have priority and people at Google are good at networking and recommending other good people).
Often, the specific result of an estimate is less interesting than the user's understanding of what drives that estimate, because it determines if they understand the levers they can pull to change the outcome in the real world.
Sometimes folks call these "puzzlers" but I think they're unfairly lumped in with true "puzzlers" - crap like "why is a manhole round" where there's a right answer that depends on either experience or some specific insight. A question where you're asking a candidate to walk through their thought process on a hypothetical (but reasonable) situation isn't "puzzling" unless you just can't handle the question.
http://en.wikipedia.org/wiki/Drake_equation
That's the mental template I have for these guestimation bullshitorama type questions, anyway.
I personally think manhole covers should be human-cross-section-from-above-shaped.
This struck me as exceedingly slow (I know that my friend got an offer in 10 days start to finish, but he's probably an anomaly since he had multiple offers already).
Is this in line with other software companies this size? I also wonder if this varies with function.
That fifth person is a “shadow interviewer” who is simply training to conduct interviews for future job seekers, and that person’s analysis isn’t included in the decision-making process.
This is hilarious! It's just like the ETS (education testing services) that runs all the US based standardized tests.
How on earth does any company take 45 days!?
The screening process takes a week or two (14 days) and then you have to schedule an on-site interview loop, which can often take a week or two lead time (since many interviewers have to travel to SF, and you have to schedule the interviewers as well) and that only leaves a week or two at best to issue a formal written offer. I've seen it take a lot longer than in total at many good companies.
Well, you'd want to call someone on the phone and get a verbal OK before sending them the offer letter, right? From verbal to written shouldn't be more than a 24 hour turnaround.
So that's the day of the interview, the day after the interview to make the decision, and three more days (so the rest of the week) to generate and send out a written offer that the candidate may or may not have already verbally okayed. That's 5 business days, or 5-7 calendar days.
> The screening process takes a week or two (14 days)
14 days to look at resumes and do a phone screen or two? Maybe at maximum, if you have to bounce the resume between teams to find the best fit.
> and then you have to schedule an on-site interview loop, which can often take a week or two lead time (since many interviewers have to travel to SF, and you have to schedule the interviewers as well)
Why would the interviewers have to travel? Don't you interview people on the same campus where most of the interviewers actually work? Why would it take two weeks to book a couple of hours on k different people's calendars (hopefully most of them selected from a pool of size n >> k)? Sorry, I think that timeline is way too padded.
Re travel: I meant that the candidate has to travel, and the interviews have to be scheduled with the pool of interviewers. The minimum time for this is generally a week, but usually it takes longer. Interviews are a significant / immovable commitment and interviewers are required to give notice if they can't make it several days ahead, which means they have to be scheduled well before that.
Large companies, yes. Not sure about software.
And the thing that struck me as Monty Python ludicrous is that no one from the actual team is interviewing you! Almost every large company and most smaller companies have you interview on site with your actual team members. Is Googlish really that strong a predictor of team fit?
What is "hilarious" about it? How would you propose that new-hires learn how to interview?
It is primarily meant to be a learning experience for the shadow interviewer. They will get to see the feedback written by the primary interviewer, and the primary interviewer will usually critique the feedback from the shadow interviewer.
The feedback from the shadow interviewer isn't actually dis-carded - it gets sent to the hiring committee, with the indication that it is from a shadow interviewer. It can be given less weight by the hiring committee members, but I've been on hiring committees where the shadow interviewer picked up on something that the primary interviewer missed that affected the hiring decision.
My first interview on-site was delayed due to room reservation issues, so it started late, and this continued as a theme throughout the day.
That's anecdata, but I have to wonder whether whatever schema and/or processing pipeline are allegedly in place to remove human bias would be able to understand issues like this or of other kinds - i.e. environmental and/or human failures outside the scope of the process. These systems don't tend to critique and monitor the environment, just the data that is entered into them afterwards.
This plus the admission from a few Googlers that personal recommendations do really make a big difference make me a little jaded when reading articles like this - I'd much prefer to see admissions of honest weaknesses alongside all the positives, but I guess we're still some way from treating anything except severe security breaches that way in the software/marketing industries (this article is in the latter industry).
Edit: I feel like I should add a bit of context about why I feel the article is misleading, given that my post was inspired by personal experience.
The misleading aspect is that the article tends to portray the process as striving towards an unbiased science, whereas my perception is that bias is still part of the decision-making process (arguably for good -- personal recommendations can be very positive indicators, as long as they're not from an old-boys-style network), and I feel that there is insufficient measurement and understanding of interview factors to make it a science (i.e. exhaustion/travel factors, cultural differences, personal factors, etc - which I don't think affected me, but are still a real part of interviewing).
NB: When I say old-boys network, I imply any kind of non-meritocracy which simply aims to get people 'in the door' without full vetting; I believe this is possible regardless of gender, but just that's the term I know to describe it
I like that they decided to stop asking brain teasers due to a lack of correlation between performance in them and performance once hired. Do they really think that cumulative GPA has a strong correlation on new-hire performance?
I certainly don't.
(I expect to get some push-back from you guys and I'm interested in the discussion to follow :))
Also, what was the exact wording of Google "straight up" declining to interview you?
these companies get so many graduate applicants each year they place an arbitrary requirement on GPA just so they can slightly reduce the number they take to the next stage of the interview process.
The data they collect is really only to make the process more efficient, not more effective. They reward employees that process the most phone screens in a month. The strangest thing, in my opinion, is that the interviewers usually don't make a decision at all, they just give a rating. Then, a group of people who've never even met the candidate decide whether to hire them based on forms that were filled out.
| Google does the same kind of software interviews as anywhere else. [...]
| You code on a whiteboard and it's supposed to be compilable in C or Java.
The reasoning is that my job is not to stand in front of a white board and write syntactically correct code without the aid of an editor or compiler, so maybe there is a better way to screen candidates that directly test the skills they will use on the job.
A good interviewer (and good hiring committee) wants to see that you can write "correct-ish" code: it looks like it would compile modulo a typo or niggling detail, but the algorithm, data structures, and control flow are clear and valid. A good interviewer will also tell you that up front, e.g. "I'm interested in your code, not your syntax". If they don't, ASK!
The problem is: not everyone is a good interviewer, and it's surprisingly hard to teach someone to BE a good interviewer.
At the company I'm leaving, we ask candidates to do a small project as a first pass. If the code isn't awful, we call them in to pair program with us (we're a pairing shop) on the code they wrote to improve it.
The company I'm joining just has you come in and pair (they're also a pairing shop) on projects their team is actually working on for most of a day.
I find both to be pretty good approaches. Better than having someone write code in an unfamiliar setting (the white board) to solve problems which are often, but not always contrived (write quick sort, etc.), at least.
We've had pretty good success with identifying good candidates since we switched over to this model and the place where I'm starting is a consultancy which is pretty well-regarded in the startup community, including HN, so they've probably had reasonably good success identifying talent.
If you're getting 1000 resumes per day, I think you would be able to weed out a significant number of them just by giving them a coding assignment to work on. Churning out a resume is easy, but sitting down for a few hours and writing well-designed code takes effort.
How would you grade/score ~1000 code submissions per day, though? You could conceivably do Coursera/Topcoder-type automated grading, but that can only get you so far and can't distinguish good code vs. bad code vs. "copied from Glassdoor" code. It might be useful as an initial filter, though. You'd have to constantly be implementing new questions w/ associated grading scripts, though, as the problems would inevitably leak.
Interview feedback is not really "filling out a form", either. I take about 6 hours to write feedback for a 45 minute interview. It's very detailed and really covers a chronology of what happened during the interview. Interviewers also submit a hire/no hire score for the candidate, so it's not only the hiring committee that makes the hiring decision. If an interviewer is willing to say "this is the best person I've ever interviewed in my life", that's a big deal.
That's a ridiculously long time to keep a candidate waiting.
Part of the challenge of a technical interview is to get at someone's coding ability without resorting to what are essentially brain teasers disguised as computer science questions - and I'd expect a lot of disagreement around where you draw that line.
Here are a few I've been asked:
- Print out the fibonacci sequence recursively.
- Print all permutations of a string (using recursion).
- Swap two integers without creating a third integer.
- Out of several million database entries, select a few at random to display to the user. Don't repeat any until they've all been displayed.
- Implement mergesort, code a singleton, print a binary tree in order, add a branch, find a cycle in a linked list...
Which of these would you consider brain teasers, if any? I'd say the "swap two integers" is the closest... but if you're including a lot of these questions, are you still in brain teaser land?
I'd love to see some google data on what types of technical questions correlate with job performance, rather than simple "brain teasers".
The classic example is sexism in orchestras. Orchestra interviews used to consist of the candidate sitting on a stage and performing their piece for a panel of judges. The gender ratio of performers was terribly skewed, even worse than the software development field is today. Orchestra managers excused this by saying that women were simply not as good -- after all, the judges scores don't lie!
But a funny thing happens if you put the performers behind a screen. Suddenly, all the factors of their gender and race and grooming go away, and new performers start being a lot more diverse. There actually was a bias, a serious one, and it caused orchestras to lose who knows how many excellent candidates.
The current state of the art for programming interviews at big companies is to have the candidate solve actual programming problems, including whiteboard coding. It's not practical to put the candidate behind a screen or modulate their voice, so splitting up the tasks of "interview candidate" and "review interview feedback" is the best that Google has figured out.
Compare this with what google is doing. It would be the equivalent of having orchestra candidates write an "original" tune on paper in 20 minutes and then talk about select topics in music theory and then submit that to a figure skating judge panel.