The "well-calibrated interviewer" curse
rachelbythebay.com
rachelbythebay.com
The best candidates light up when they have a chance to explain something cool, no matter who is listening and get even more excited when you ask a question about a detail. Good candidates are patient and thoughtful and ask YOU check-in questions ("so now we have X, right?") Mediocre candidates get a bit lost, sometimes overshoot or undershoot the right level of technical detail, but muddle through and sigh in relief at the end. Asshole candidates try to dump a bunch of smoke and mirrors on you, say shit like "well, if you haven't taken an algorithms class ...," wave off your questions, and pretend they don't have enough time to answer the question "seriously."
If you think in your engineering career you're never going to have to explain "how something works" in at least some detail to a semi- or non-technical person, you are not going to have much of an engineering career.
I agree this is true in a general human-conversation sense: many good engineers like discussing things with non-engineers if the latter show any interest, and the best can actually explain things in understandable ways. But I think in the context of Corporate Bureaucracy things can be a bit different: I can forgive somebody who's normally a patient explainer being frustrated if they feel their time is being wasted by endless HR bureaucracy. In particular, there's a difference between someone asking you about a detail because they're actually interested in it, and someone asking about it even though they don't care at all, because it's on a checklist and their job is to plow through the checklist.
Of course, that still wouldn't justify being a condescending ass, and certainly wouldn't justify giving wrong answers! If I felt the other person was uninterested and operating by rote, I would probably just recite correct, by-the-book answers.
I used this to my advantage at times in interviews, I allow the candidate to think I don't know much about the topic at hand. It's nice to see how they can articulate something and you can smell the BS quickly.
That said I am looking for Attitude, Aptitude and Fit. It would be hard for me to spend much time with someone who started the call like that, I would have most likely politely bowed out pretty quickly (i.e. you failed Attitude in the first 30 seconds).
I can only hope that her not-that-productive-in-interviewing colleagues don't do such shit.
Secondly, even if you don't like your interviewer, if you're asked technical questions in a phone interview, it is just common sense to give the best answers you can. If you're competent, before long you'll realize that the person at the other end actually does understand what they are talking about.
In this case I fully agree with her judgement and decision. The fact that the interviewee failed to realize that he was getting a technical interview when told he would get one says that he was incompetent. The fact that he was then unable to answer the technical questions he was asked them says that she discovered ample cause to fail him. (No surprise - most interviewees will fail the interview process.)
He has already sunk the interview in her eyes, but she doesn't want to tell her employer that this guy bombed due to his interactions in the first 30 seconds with her. Moreover, he is a sexist bastard.
He then tells her something she disagrees technically about load average, and this makes him a lying piece of trash.
Regardless, he has bombed and sunk the interview, and her only duty now is to spend another 44:30 seconds to bury him.
Of course, she is playing this for humor, so I am out of line for thinking Rachel sounds like a jackass who has done a disservice to the candidate as well as a disservice to her employer.
And at the end of the interview, we are left with a sexist man who has had his preconceived notions confirmed by encountering a female tech. That's something of an own-goal.
Given they have very few women interviewers, and the guy has to work in a team, any woman who has the unfortunate luck to work in the jerk's team is going to have their life made difficult. Rachel seems to have done a service to the company by flunking him, IMHO.
Rachel didn't 'ask him a number of questions', she purposefully played the role of a ditzy, clueless airhead to play to his stereotypes rather than just ask the questions straight. Sure, it might feel good at the time and we can all laugh at the guy, ha ha, but what actually happened was the guy got reinforcement for his shitty behaviour. If the guy was giving weak answers, then being a straight interviewer isn't going to suddenly make him Einstein.
I had to spend the next half hour talking with recruiting and hr about this guy's claims of me verbally abusing him. They probably gave him another phone screen just to CYA.
Also, it's useful to gather evidence, because some hiring manager who has three more headcount to fill in the next two months before his performance review is going to wonder loudly if it was really sexism, or if we threw out a sterling resume over a misunderstanding.
What if he has impressed everyone else in the process so far? Maybe he's very personable when talking to men, or other interviewers happened to not ask questions that would have revealed skill deficiencies.
If they want to hire him, saying he made a sexist comment might not prevent the hire. "Maybe he's not that bad." But if you have a couple of comments he made, and can show that he isn't technically proficient (or merely average) then you have evidence other people can use to make a good decision.
One is the corporate image that the company tries to project.
The other is that upset interviewees in the US occasionally develop theories about why they might have been discriminated against. And their mis-memories about what was said can figure into that. No matter how far off base, the lawsuits are expensive, and the truth does not always win you the case.
Therefore every competent HR person and anyone with experience in this part of the law will tell you that you never give anything away about why someone failed the interview.
Irrelevant even if it's true. If you're going to make that sort of assumption about people, knowing nothing more than their gender, I don't want to hire you.
Knowing why similar episodes happen in first place also helps. Some people are socially awkward. Some people are tired of and irritated by numerous phone screens without followups. Some people come from another culture. Some people are genuine assholes, though, no doubt about that.
Still, I am surprised that such a tenured interviewer doesn't know that much about interviewees.
Confusions like that would have easily surfaced if she had asked "What made you to believe this isn't the technical interview?", but instead she played the part of a less technical recruiter trying to give a technical interview.
If I went to interview with some small company and the technical interviewer didn't seem to know what they were doing, that would certainly be a mark against the company in my book. If they extended an offer, I might decide not to take the job.
But things can happen. What if the real technical interviewer had a sudden crisis come up and someone else without much experience was substituting in? If you do a good job they may be impressed with your communication skills on technical issues when talking to less technical employees. Maybe you'd get a further interview with the more technical person since you did a good job in that accidental circumstance.
There are quite a few people here saying she was unprofessional for handling things that way, that she should have cut the interview short. If he thought company was acting unprofessionally, he could have just cut the interview short, saying he didn't think it would be a fit.
For all we know, as soon as the interview started he made up his mind and stopped trying.
A "no hire" is probably appropriate in that case, though, as anyone who makes that many assumptions without verifying that they're correct probably would make a terrible engineer anyway.
I consistently do two-ish phone interviews or on-site interviews a week. I'd say 2 hours is a little on the high side for average time per candidate (probably around 1.5 hours), but it does happen. I am sometimes called in to do tiebreaker interviews (before or even after on-site interviews), or requested for particular candidates. I occasionally help other groups (capacity engineering, software engineering, operations engineering) with their interviews on top of my usual load. I've flown overseas for people who can't easily get US visitor visas. I'm in the top 5 interviewers by quantity for the role (which isn't too hard, admittedly), and in the top few percent of the company (I even got some ice cream as thanks!).
In addition to interviews, I also spend an hour a week in a recruiting meeting trying to improve our recruiting process (including giving feedback to our interviewers on how to improve their interviewing and feedback). And I've done non-interview recruiting at universities (manning booths at career fairs, mentoring at hackathons) and conferences (at Velocity, SCALE, Surge, PyCon just in the last year).
My feeling about spending 4-6 hours a week on interviewing and recruiting activities is that it is some of the most valuable work I can be doing. Avoiding a false positive - ensuring we maintain a consistent high bar - is going to save people in my group (and others in the company) potentially hundreds of hours. Making a great hire is going to add thousands of hours of productivity to my group and the company.
And we're a small enough group (and even a small enough company for the non-PE hires I've helped with) that I can easily keep track of the people I've helped hire and it is supremely fulfilling.
> In addition to interviews, I also spend an hour a week in a recruiting meeting trying to improve our recruiting process ...
Exactly.. If it's broke "let's all work to improve it". Not "oh no here comes the recruiter again with some horrible candidate".
I don't know about the team you work with but the recruiting team I'm working with is full of bright excited and passionate people who want to learn. They don't code.. SO WHAT.. They know people well, and they're willing to learn. That's all I ask.. from anyone... I can teach you anything.. I can't teach you to have a good attitude and I've yet to find a way to increase people's IQ.. :)
I might not be able to teach you as a recruiter to code, or "be smarter" but I can help you "work smarter" and help avoid burning people out further down the pipe.
> My feeling about spending 4-6 hours a week on interviewing and recruiting activities is that it is some of the most valuable work I can be doing.
4-6 hours and once in a while you get to add 2,000 hours of new capacity to the team! :-)
If you think about the ROI interviewing and getting the right people on the team can be amazing. Much better then those hours spent refactoring 20 lines of code...
Taking some time to understand what people do, the challenges they face, the work they head off without you seeing any of it, and that they're as interested in getting things done, and getting better at what they do, as you are is well worth the investment.
It sounds like it could be a fairly unreliable indicator. I'm not sure what the interview-to-hire rate is, but there are only a pretty small number of people that are offered positions - and even then sometimes you don't win against a competing offer, or a sudden realisation that they want to stay geographically where they are. So even for the top few percent of interviewers who interview ~100 a year, you might only have 2-6 hires, which isn't really a large enough population to evaluate.
In an interview, the best you can do is try probe talents, skills, and knowledge, but you'll also get a point-in-time view into their current attitudes and beliefs, as well as the possibility for swagger and misdirection (even self-misdirection).
We do look carefully at situations where one or sometimes two people get strongly negative or positive signals while the remainder get strongly in the other direction, in case there is a problem. Most times it has turned out to be valid concerns, and it also helps us give feedback to interviewers on how to write better feedback, how strongly to weight certain traits ("not knowing the name of a particular useful function/method under pressure is reasonable, so let them just use it even if they got the name wrong"), or how to evaluate a particular type of candidate (newly graduated candidates, for example).
It might be interesting to track the number of screens -vs- cut-offs on-site.. :)
Also I am measuring the number of screens that fail as a quality indicator for the recruiting team.
Think of it as a pipeline from the first contact all the way through new-hire and 1yr review. How are we doing at each step and what can we do to improve each step for everyone involved including the candidate.
One problem I've seen in big tech. companies (especially Google) is that most employees hate doing interviews, they think it's worthless and a waste of their time. They don't seem to realize that this is a necessary part of the job (especially with a company that went from 20 to 20000 in such a small amount of time). In the end, would you rather have your co-workers be interviewed by yourself (or another technical fellow) or by an uninformed HR department?
Having said that, I would've loved to work with Rachel. I have a deep fascination and respect for responsible and involved people who can pull the weight of an entire team when everyone else ain't doing shit.
Everyone in software management is familiar with the concept that good employees are worth Nx more than average developers. But a good chunk of them have also attached themselves to the idea that employee quality scales continuously from average to good, and so there's significant incremental value to sussing out everything you can about a candidate.
I don't believe this is true. I believe there are "average" (meaning, in reality, "bad") candidates and there are "good" candidates, and you need to find out which of those two buckets people are in, do some extra checking for culture/work habits/communications skills, and then stop.
Instead, the formal interview processes I've seen are gauntlets of trivia and case studies and successive whiteboarding exercises that exhaust everybody. They don't generate value proportional to their cost. And they're unsustainable, so that whatever marginal effectiveness they have is compromised totally within a year when everyone who participates gets sick of it and starts phoning things in.
Tangential point: some people just can't interview effectively, and you need to keep them away from your interviewing process. You're probably not going to be able to perfectly balance the recruiting load across your team.
There is a reason for the regimented process that you're talking about.
The challenge for any large company is to delay, for as long as possible, the inevitable reversion to hiring the mean. Under normal circumstances, different teams will have different standards, and by the time you realize this you'll have a lot of teams whose standards have slipped, and you will have a lot of marginal employees.
Therefore smart large companies try to create a minimal process that aims to create uniform process across the whole company. The process tries to be lightweight as possible on individual people, but fights the good fight against entropy.
Unfortunately for Rachel, Google's process centers around the idea of having well-calibrated interviewers whose interview characteristics had been so well tested against others that hiring committees know exactly what to make of their interview feedback. (What absolute scale you grade on does not matter much. How your scale compares to everyone else, and how well it predicts everyone else, does.) Once an employee winds up as a well-calibrated interviewer, HR would like to use that person as much as possible (and each use does a better job calibrating the employee, and increases their desirability as an interviewer).
This is inconvenient for the employee to whom it happens. But, on average across all developers, the process is quite light-weight.
Instead, the formal interview processes I've seen are gauntlets of trivia and case studies and successive whiteboarding exercises that exhaust everybody. They don't generate value proportional to their cost. And they're unsustainable, so that whatever marginal effectiveness they have is compromised totally within a year when everyone who participates gets sick of it and starts phoning things in.
Google's process has been sustained company-wide for a number of years now. There is much that is imperfect about it, but evidence suggests that it is sustainable.
And then: I want my point to stand: for a given role, there are good hires and there are bad hires (where "bad" includes "meh"). There is not a lot of value in grading past that. But lots of hiring processes are premised on doing exactly that; they take candidates who can demonstrably perform the task required for the role and then beat them up with trivia questions, with puzzles, with nerd dominance quizzes, and with subjective stuff.
I point that out because if you can find a way to accept that, you can streamline a lot of recruiting process and make interviews less painful for everyone. The way I finally managed to make that leap of faith was to reflect back on everyone I've had a hand in hiring that didn't work out, and then consider the process that hired them. Then I think back on everyone I've hired who has surprised me with their awesomeness (some of them are even people I'd negged). I am now sold on the fact that most tech interviews are voodoo rituals.
What I would do/am doing instead:
* Get people out of HR/phone screening as quickly as possible.
* Have an explicit up-front call with me or someone senior in the hiring process to level-set people, try to defuse some of the tension, explain the process, and answer questions. Things go faster and more smoothly when candidates are convinced they want the role. The amount of sales effort it takes to get a candidate into the pipeline is not enough; you need to engage them on what the job is like, let them ask what the day-to-day is, and get them revved up before they interview.
* Minimize phone screening, both in length and in depth. I am on a mission to get rid of them altogether but am not convinced I'll succeed.
In the interim, what I'd like to do is take all the effort and expense we'd sink into hour-long phone screens and redirect it to coming up with a small rotating battery of really, really good questions that we standardize on.
* Replace phone screens with challenge problems. Calibrate them for the length of time a series of elaborate phone screens would take. Try to make them fun. Here I admit I'm not sure what I'd do back in my pure dev jobs to make this work, but I'm sure I could come up with something.
* Give the on-site interview team a full dossier of objective facts about what our up-front process learned about the candidate, so that nobody is trying to read tea leaves out of resumes. Candidly: I barely glance at resumes anymore, and I wince when I read interview reports that go "I read XXX on the candidate's resume and so asked YYY".
The most important thing I think I've learned about recruiting in the last year --- apart from getting better at getting people in the door --- is to generate sets of facts that are easy to compare between candidates. I'm learning a lot by reviewing the history of how people do on challenges, and I really very much want to be in the same place by the end of the year with all person-to-person screening interviews. A nice side effect I'm hoping for is for prep time on phone screens to approach zero, and for actual phone time for interviews to approach ~20-30 minutes. We'll see.
From what I have read (mostly Kahneman) most interviews as performed are useless and suffer from the illusion of validity. Instead, structured interviews and scoring that are the same for all candidates are strongly recommended. Which is, draw up a short list ~= 6 independent factors which from experience are reflective of the job and whose answers you can easily judge objectively. For each factor come up with a small number of questions with a score, say from 1 - 5 and then simply add up the scores. No fancy weighting required - research shows that regression does not add much to this simple model. To avoid the "halo effect" do not jump question categories. This will vastly improve upon the more common unstructured interview and though still far from perfect you will be hard pressed to improve on that short of an IQ or equivalent test.
Interestingly, and reminiscent to bagging in machine learning, unstructured interviews can be brought nearly up to par by averaging multiple independently scoring interviewer opinions.
On standardization, there is one gotcha. It is much easier to calibrate a standardized interview process and compare this candidate to that one. However at some point you'll have a candidate who goes through your interview, and blogs the whole interview when they get home. At that point you'll find that every candidate who bothers to do a basic search is curiously well-prepared for your interview - at which point your interview's value drops sharply. At Google's size this happens constantly. For a random startup, not so often. But a hot startup should expect it.
Therefore I would recommend that to the rest of your list you add a monthly calendar entry to see what a search can turn up about your interview process so that you notice when it needs to change.
Similarly, self-deselection is something you want to avoid - ie, you need to prove yourself to them a lot more. Making sure people you are interested in see the sorts of information they might need to consider working for you is important - maybe it's talking about your product/opportunity, your development/product/management/business philosophy, or even just mentioning who works with you and what they've done (although focusing on the VC team, or just saying "A bunch of former employees at Google, Apple, Microsoft, Facebook, Dropbox, ..." is probably not enough).
One thing I've found is that certain people are better at asking certain types of questions. I love "exploring" conversational-style interviews that are highly interactive and where not knowing something is an opportunity for the candidate to figure it out themselves or to learn a new thing. One colleague does an awesome debugging scenario that I sometimes think could continue for hours if he wanted to. I assume that certain candidates like particular interview styles - finding some way to match those two together would be great. (A recent Facebook start and I spent an additional half-hour during his interview because we both enjoyed the subject we were discussing. Making each interview that enjoyable...)
I'm not entirely sure what you mean by challenge problems - is this a technical phone interview with a specific set of broader topics/challenges, or is this something that gets sent to the candidate and they get some time to return a result?
I prefer the term "phone interview" when a non-recruiter/sourcer is evaluating the quality of the response to some question or challenge, and "phone screen" for the initial recruiting discussions where wholly unsuitable candidates are weeded out due to skill or experience mismatch to the role based more on treating commentary provided by the candidate as fact (although it does go a lot further than that).
In this world, what happens when a person wants an internal transfer to another team with more rigorous requirements? Does he have to re-interview? Who controls the transfer process--HR or the other team? Will re-interviews be fair, given that the interviewee is being interviewed by people he already knows? What if it happens because of a re-org? Do you get fired if you fail the re-interview?
Companies like google interview to a higher standard that may be required for your initial job with the understanding that you may have to transfer, and firing/re-interviewing is much harder than just setting a higher bar to begin with.
Phone screenings can be very useful to the candidate. I've taken the trouble to travel (via public transportation, so it took quite a few hours out of my day) to a potential employer only to find out within 10 minutes that there is no fit, and the lack of fit would have been obvious within the same 10 minutes of a phone interview.
As a candidate, I found the Facebook and Google interviews to be fairly daunting by description, but even with 10 hour timezone difference in jet lag, I did not feel I was being overworked, that the interviews overlapped dramatically in their content, or that I was being asked anything unnecessary (well, there was one interviewer...). In both cases, I felt I had an opportunity to show off some depth in a number of areas, and that I may have been less likely to succeed if my best interview was discounted. That said, I had a decade of experience before going through those gauntlets, and I am well aware that my experience of these things will differ to others.
As an interviewer and candidate evaluator, sometimes it is fairly clear from just a subset of interviews that a candidate is one we should make an offer to. But sometimes there are one or two interviewers who just did not get enough signal to be comfortable to hire, but the remainder made up for that. Improving that situation is something I am trying (in my own, admittedly limited, way), so any insight would be appreciated.
In a closed-allocation company (which Google is) you don't work for the company. You work for a manager, who decides whether you get to have a career there. This means that citizenship is worthless from a career perspective, unless the manager makes it a priority, which is unlikely (because what does a middle manager gain from citizenship work?). With Google's blind matching in addition, the people you interview are unlikely to end up on your team.
Companies get exactly what they deserve when they create perverse incentives and then wonder why 9 in 10 people are playing politics instead of doing what's good for the company. If you let the manager unilaterally wreck a report's career, then people are going to work to play their managers rather than doing what's best for the company. So you get political behavior.
Having said that, I would've loved to work with Rachel. I have a deep fascination and respect for responsible and involved people who can pull the weight of an entire team when everyone else ain't doing shit.
I agree. Losing her was a huge loss for Google, and I hope her current venture does well.
From the superstar who you really want to start working with right away, to the bozo who can't code their way out of a paper bag, I can't say that a single interview was a waste of time.
The bad ones teach you how to ask questions, the good ones keep you on your toes (and during the "sales job" part of the interview, make you introspect a little on your current job, which is always good).
[I wouldn't have failed the guy immediately for the "I thought this was going to be a technical interview." Then again, maybe it was the tone of his voice? He'd probably been dealing with a bunch of HR / recruiter nonsense]
When you're growing staff by 100's of percent year over year recruiting has to be one of the most important tasks you can put energy behind. You want the right people on the right teams and you don't want a random non-tech person making the determination.
I see engineers over-worked, up to their ears and always complaining about interviewing. I often wonder, we're in a rational, problem solving industry but we can't see the simplest problems. You're over worked and up to your ears because the team doesn't have enough people on it or the "right people" in it and you're slammed. Does it make more sense to just keep working your backlog or take time to get quality people on the team so you can double your velocity on your next sprint? Why is this so hard for people to see?
You have to kiss a lot of frogs to find the right people for your team. It's a numbers game, perhaps it's hard for programmers to understand that the hiring process is a lot like debugging, most people don't like doing it but if you don't things will break and stay broken.
All this said I do think companies have a hard time figuring out how to balance this. I'm consulting right now with a company that has a very large recruiting team and crazy hiring goals. Meanwhile they were sending a lot of "questionable" candidates to the engineering teams. Engineers were bitching about this to me over lunch in the hallways etc. My response, simple, I swam up stream. I started meeting with the recruiting team, reviewing as many candidates as I can and helping them develop a better process. We're setting goals and measuring our rejection rates and getting feedback on each one even if it went bad. Recently I told them that if we're cutting someone on site to let me do it. I want to meet these people before we cut them to understand what went wrong, did we do something wrong? Did we cut the wrong candidate? I want to give everyone the benefit of doubt and figure out how to improve the process. Could we have avoided wasting their time and ours?
In the OP's case perhaps upstream should have been measured by how many she rejected. They should have been incentivized to improve their own "quality score" instead of burying people on the "well-calibrated interviewer" list.
You can sit there and bitch about something that is broken.. Or you can choose to fix it.. Which will it be!?
- Most people didn't get into programming so that they could interview people 10+ hours per week.
- Plenty of managers won't really count interviewing time in the project management tools (or give you credit however time is managed). You're still on the hook for whatever work you would have been responsible for with 0 interviews.
I agree that said I like working on really good teams with smart people so there is some balance, you either are willing to invest in improving your team or you can sort of hope it happens by accident. Hoping it happens by accent is like hoping you'll win the Lotto but never buying a ticket!
> Plenty of managers won't really count interviewing time in the project management tools (or give you credit however time is managed). You're still on the hook for whatever work you would have been responsible for with 0 interviews.
Then clearly for that manager this is not a priority and they should leave you alone. :-) I think the challenge then in that situation is the manager is making a big mistake. I'm a hands-on manager, I take stories off the board, I code, I do meetings and I know how hard it is to really deliver when you're doing lots of "distracting" things. If your manager is pressuring you to do interviews then you need to push back and take less work off the backlog. Employment should be a dialog, not a series of orders barked at you by someone who doesn't understand their impact.
Yet another reason why closed allocation and traditional management lead inevitably to decay. When there's a career penalty to time spent in recruiting, that's really bad for the future of the company.
I am always amazed at people who will let a PHB torture them and gladly bitch about it at every family function and with a round of friends at the bar.
Really.. So you're an amazing talented person and you're letting a PHB ruin you life.. What does that say about YOU.. ????
I suspect that recruiters are going off name-recognition: there're some interviewers that they know & like, and so they assign candidates preferentially to those interviewers with digging into the database to see if there'd be another equally-qualified engineer.
Many other things work like that too. If you do code reviews quickly, you'll get a reputation as a good reviewer, which means...more code reviews. If you always know the answers to people's questions, it means you'll get more people asking you questions. If you're a UI wizard, it means you'll always be assigned UI work, even if you want to do something else.
The usual way out of this is to learn to say "No". That's how I escaped the UI trap, and I've done it to get out of code review & question overload as well. I suspect Rachel's problem could've been solved by setting her interview constraints to 0/week, which (if done by enough people) would've also forced the recruiters to look further down the interviewer database and find untapped engineers for interviews.
My gut tells me that most likely the lack of quality coming out of the recruiting team was burning people out on teams. If an engineer thinks there is a 1:100 chance the person will get hired and they have to invest 2 hours in each person they'll never bother. If the recruiting focuses on quality and gets it down to less time and higher acceptance rates the engineers might (reluctantly) be more comfortable "spending 2 hours" on a screen rather then "wasting 2 hours".
Observing the process sourcers and recruiters go through to improve their quality has been very interesting to me. It takes time for them to calibrate to the role (just like interviewers), and while my Silicon Valley experience amounts to one company, my general impression is that many engineers at other companies just don't appreciate the effort they go through to calibrate to the role, and many engineering departments don't put in the effort to help them get to being great.
While there are occasional dips in quality as sourcers and recruiters try new strategies (maybe going deep on release engineering talent in the games industry in Alabama), I'm willing to accept this because I understand that they're trying to get better at what they're doing - it's just like they don't (seem to) get angry with me when I fail to get good signal on whether a candidate is good or not, possibly because I was trying out a new question.
Assuming that the problem was a lack of quality AND that this is the recruiting team's fault sounds unjustified - I assume that Google takes recruiting seriously and puts at least as much effort into calibrating their recruiting team as they do on calibrating their engineers for interviewing.
I see no evidence that they were doing a bad job on quality. They do as good a job as they can. But recruiters are fundamentally unable to evaluate technical skills. The OP would be the first person that interviewee talked to who actually could judge technical skills, and her job was to be a rigorous filter, to remove the egregious candidates.
The other team members should realize how important it is to do the screening instead of pretending working on code was more important.
That part of the process was not being criticized here, but I would agree with it.
If an engineer thinks there is a 1:100 chance the person will get hired and they have to invest 2 hours in each person they'll never bother.
Trust me, engineers are made aware of the statistics. But it is one thing to see them in abstract, and another to look through all of the candidates that you, personally, have talked to.
>> Does it make more sense to just keep working your backlog or
>> take time to get quality people on the team so you can double
>> your velocity on your next sprint?
Interview time has a very high ROI.
Interview time has a very high ROI.
Seems to me how much profit there is in panning a river for gold depends on whether a river has much gold in it. And it sounds like she was panning a river that didn't have much gold in it.
What would you say is your interview-to-hire ratio?
This was the result 200 hours of her work and some number of hours of work from others.
Each step of an interview is an INVESTMENT.
First investment is the Recruiter's time (sorry but they get paid and SHOULD get paid to take this first step)
Second investment is the screener (yup I'm looking at you Phone Screener and I do it all week long, suck it up and enjoy the ride)
Third and MOST EXPENSIVE investment is the on-site, team interview, disrupting many people's lives and productivity including the CANDIDATE
(blows my mind that NOBODY here seems to be on the side of the Candidate who is trying to find a new team and a incremental change in their career)
Fourth and even more expensive is ON BOARDING
Fifth and yet another expensive investment is ON GOING management and integration
So let's think about this.. Where in the process do you want to spend time?
Should you be upset that YOU PERSONALLY saved many hours your teams of wasted time?
Or upset you spent 2 hours of your time?
Answer this carefully.. Think slowly about this.. it's the deepest issue here.. And says a lot about who YOU are...
Unless you're working on a new project or have very long sprints then it's going to take more than one sprint to significantly increase velocity after increasing team size.
It's also really hard to double velocity by increasing team size unless you have a really small team that's not spending much of its time developing. Going from 7 to 14 developers will add a huge amount of communication overhead and give a slew of new problems.
Without trying to sound like I'm bragging, I consider myself a senior level, talented engineer. Recently, however, the startup I have been working at ran out of funds and I have been pounding the pavement looking for work. The problem is that I don't feel like any interview I have been doing, or for that matter, have ever done, is really a measure of how good I am as an engineer. It's more a stamina test to see how well I can be tortured and hazed and it makes me reluctant to take on more interviews. Most interviews are about being quizzed on the minutae of a language or about solving a programming quiz in 60 seconds while the interviewer critiques thought of every line. I just don't see how these interviews are actually filtering out bad candidates or measuring good candidates. They are measuring how someone performs in a very specific, non-realistic situation that does not emulate how every day work would occur. The worst example of this are the horrible SHL/Brainbench tests that seem to measure memorization more than skill. I do quite well on those silly tests, but I'm not sure they really measured anything accurately.
I was the happiest when I got homework assignments as my tech prescreen or was asked to walk through examples of my own code with their engineering team. I feel these are less stress on the interviewee because it is code you are familiar with and are a closer exemplar of what goes every day at work.
Or maybe I'm just a wimp.
I've also been on the receiving side of delightful trivia questions like "What are all the ways the character '*' is used in Python?" and puzzle questions that test "creativity"/"ingenuity", whose value are totally alien to me. (I've also done online quizzes/tests, including personality ones(!))
I would like to think there's a broad middle ground in between, but that there isn't a single process that is going to be effective for all companies or for all candidates.
In the past, I've used "homework" as a final step in a hiring process, which worked for some (it certainly washed out a few candidates who got passed less strenuous interviewers), but which others weren't interested in doing - including two that we hired anyway and who ended up being great coworkers.
I imagine I would feel a bit put out if a startup wanted me to spend 2-4 hours on something before they allowed me the opportunity to get to understand the position and the company (especially since I would hope the startup would look at my open source contributions in lieu of the homework). New grads might have a bunch of opportunities, and they might decide on skipping something with that sort of requirement.
I spent much of this past fall interviewing for an internship for the upcoming summer. It's one thing for a company to ask a candidate to devote a day to interviews, after flying them out and putting them up in a nice hotel, but it feels completely different to have the candidate complete some sort of homework.
The candidates that you want to hire are likely busy interviewing with several other companies as well, and probably don't have time to work on some random assignment for your company. So, unless your company really stands out, they're just going to pass.
I've found that interviews benefit both the company and the candidate, while I can only really imagine homework benefiting the company. I learn a lot about a company from the way they interview and the way developers talk about their jobs; the interaction is really important. Yet, homework assignments don't really have much interaction, and end up being a test of how much the candidate wants the job, instead of how well they fit with the job.
1. Multiplication (2 * 4 = 8)
2. Power of (2**4 = 16)
3. *args and **kargs in functions definitions and function calls.
4. Imports (import * from blah).
But that's the thing - it's a bad question. It should be asked as "what does this do?" or "how do you do this?", like:
"What does this code do: foo(1, 2, kw)?"
Or:
"How do you import all exported names from a module into your module's namespace?"
These show your code reading and code writing skills.
The original question is based on unreliable use of memory. Memory is great at slightly-fuzzy-key lookups. It is usually terrible for table scans applying a pattern filter.
Think of the challenge "Provide 5 common Unix command line utilities start with t"?
Most people who actually know 15 or so of them and regularly use 7 or so of them would have trouble providing all 5. Some will just blank for a minute (because there are just no pathways to work from), most will find 2 or 3. People who studied a Unix text book the night before would also know 2 or 3. A few might just totally blank and become affected by the fact they can't even remember something so easy that they will be unable to answer that question, and may affect their interview performance after that.
So it just is not in your interest to ask those types of questions.
1. Interviewing and evaluating a candidate has very, very little to do with engineering. Yes, you have to be able to evaluate code but this just scratches the surface of what needs to happen during an interview.
2. As a corollary to #1 - just because you are a good engineer doesn't mean that you will be a good interviewer. Never mind whether or not you want to interview. It is just a different skill set.
3. Interviewing can take a lot of time. But it also has high value for the company. If you are passionate about the people that work with and really want to shape the company, then get involved in interviewing. There are very few activities that will influence things more.
4. A lot of the time that I spend in 'interviewing' is actually time that I spend with co-workers to make them better interviewers. For in-house interviews the breakdown is 15 minutes preparing, 45 minutes interview, 30 minutes writing debrief feedback, 30 minutes attending the debrief, and then 60 or more minutes working with the other interviewers on the loop or helping the hiring manager understand what they should be looking for.
5. In some ways the interviewing process is no different than any other process employed by a company. In particular there is almost always room for improvement. As an employee of a company you can spend time complaining about the process ('it is broken', 'it is unfair', 'it is too heavy weight', etc) or you can spend time fixing it. If you are working at a company where you are not empowered to fix things like this then you should consider working for a new company.
6. I like lists!
That way it doesn't seem like waste of time and it gets people invested. If they don't pick the right person they could end up stuck with someone dragging the team down and they'll eventually learn the value of having a thorough interview and being interested in it.
The other side of the coin is that they should also be allowed to influence the type of offer and how aggressive the salary negotiation would be.
I have seen a cases were a good candidate technically hasn't returned back because of presumably a low offer.
Basically all software engineers (including some whose titles aren't Software Engineer, and including managers) go through this six week process - first four weeks is mostly getting people up to speed through two or so presentations a week and doing real tasks (which may mean meeting up with some other engineers to understand enough context on how to proceed) that go through the same review and push process as everyone else, and the last two weeks is mostly learning about specific teams and a bit of a sell exercise through a set of presentations.
All through this, you share a software engineer mentor with two other bootcampers who helps you navigate the code repos and also the company.
At the end, the bootcamper decides what team to work on, with some background information like company priorities and maybe with some teams specifically reaching out to them.
That's a fairly huge investment, so I'd love to know which other companies offer this sort of program.
I wonder how they deal with understanding/measuring performance with these short engagements.
(Facebook has a "hackamonth" program which people who have been at the company for 18 months are encouraged to do - work on a new team for a month, and then decide to return, stay, or join a third team.)
I've heard of this function being applied at the other end of the interview loop. Microsoft has its "as appropriate" addition to the loop at the end. (Senior person isn't bothered to interview you unless previous people liked you.) Amazon has their "bar raisers", seasoned interviewers added for quality control, at least one on each loop.
Doing this up front seems to make more sense, as you waste fewer employees' time. At the same time, as Rachel indicates, it's less satisfying to the valued 'well-adjusted' interviewer, as they see few of the people they interviewed get hired.
Are there other models that solve that problem?
Google's policy of separating the interview process from the job assignment breaks a lot of the incentives to interview that most companies have in place.
Most managers, if given a choice between hiring a "good enough" candidate and hiring no one, will choose to hire. Particularly in a growing company, where everyone is overworked and there's a lot of grunt work that people would love to have a new recruit do. Over time this results in a degradation of overall employee quality ("A players hire As, Bs hire Cs") and a bunch of deadweight that were needed to solve a temporary staffing crunch but later end up holding the group back.
It also tends toward fiefdoms and empire-building; when you're hired by the department, it can be very difficult to move between departments. One of Google's strengths is that transfers are pretty easy; I doubt that would be true if you were brought in by a department manager instead of assigned to the company as a whole.
Result: all rational incentive is to do a terrible job on those deliverables that are least pleasant and/or least rewarded.
It's a simple point. I'm surprised you missed it.
There're many other tasks like this - usually the reward for being good at your work is to get more work. And the way out isn't to be shitty about doing your work, it's to ask for what you want in return for doing well at your work. That could be a raise, or the freedom to work on interesting projects of your own choosing, or more personal time, or it could just be "I don't want to do interviews this quarter so I can focus on my professional goals".
I think there's a legit problem at Google in that interviewing isn't rewarded highly come performance review time, so it doesn't translate into bigger raises or better projects. But there're mechanisms in place for saying "Please don't send me interviews" or blocking off days where you won't be scheduled for one. If the best interviewers take advantage of them and then HR finds out that there's a shortage of good interviewers, that sends a strong signal to the company that they need to increase the rewards for interviewing.
It is the sort of ROI that can scale a company from 20 to 20000 employees in just a couple of years :)
If you're spending a lot of employee hours filtering candidates who could have been filtered more easily through other means, I can understand feeling a bit indifferent about it.
(or whatever the actual percentage was, I don't know the total)
That said it sounds like the quality of candidates sent to her were really low and the upstream recruiters should have been working to improver that.
Start with quick-screening questions online, then a phone interview, then more in-depth online testing (nothing too burdensome on the candidate, say an hour's worth), then several hours of interviews (perhaps all on one day) by various interviewers.
You should also gather candidate's expectations/requirements in compensation, corporate culture, etc as early in the process as possible, to avoid disappointment.
At any point in the process you should be willing to stop, politely but decisively, whether for jerkiness or for technical incompetence.
I work at a company with a similar policy on technical interviews (though I don't do any), and here I think it's uncommon to decline a particular interview unless you have some very pressing engagement.
Why? Job interviews aren't the only way to connect with the outside world. It's a failure of imagination on our part to believe that bureaucratic grotesque is the only way to build up a shared endeavor.
Open source work is one example of people coming together that has nothing to do with job interviews. Other disciplines have other approaches – e.g. auditions. Conferences are a way for people to meet who otherwise might not. There are many avenues.
I say we haven't noticed yet what an inhumane thing the job interview really is. We'd never inflict it on our friends. Why inflict it on prospective colleagues?
Last consulting gig I told I was willing to help them for a few months. I sent them a CV and they saw I meant business. Got an email saying: "you're starting next monday if it's okay for you".
Quite an efficient process ; )
(I don't know which part of my CV they did background check that said)
It's also sub-optimal, from a career perspective, to work on long-term projects. Those are the first projects to be cut, and in the modern asshole corporation, a project being cut usually means your job is over (because no one will take a transfer from a failed project, especially because managers tend to blame project failures on the people below them and have months of lead time before shit goes down).
Recruiting is the ultimate in long-term projects. It's extremely important for the company, but very hard for someone who's good at it to get personal credit. Most of your time is wasted on horrible candidates, and the good ones have a lot of great options, so it takes a long time to get a successful hire: 6 months to find the first, plus 12+ months for that person to be successful at the organization (at which point, it reflects on him, not on you for hiring).
Companies rely on good citizens like Rachel who keep the place going, but rarely reward them.
However, most of the traditional closed-allocation companies view (despite their claims to the contrary) non-executive recruiting as undignified grunt work (you're associating with people, most of whom the company won't hire) and don't reward it. It just leaves you with one less hour per day to serve your boss's political needs.