The anatomy of the perfect technical interview from a former Amazon VP
firstround.com
firstround.com
Here were the most frustrating aspects of interviewing for me:
1. Bar raisers don't ask questions that they themselves don't know the answer to. I believe this is what Roseman is referring to (as a desirable?!? attribute) when he says "One of the things that really pisses me off is people asking questions that they don’t even have a good handle on themselves". What happens in practice is that you miss hiring people who may not be able to answer a question to which every interviewer already knows the answer but whom might be uniquely qualified in some other area.
2. Interviewers still ask candidates to code answers to theoretical questions on the whiteboard. There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform.
3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript.
4. This isn't really part of the Amazon interview process but Amazon's job descriptions are (for the most part) written so generically and absent of personality that I think they miss out on lots of great candidates for whom startups simply seem more fun & interesting.
The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process.
I suspect the real problem is the implication that this is even a solvable problem.
Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with?
We work with what we have, and take the benefits it gives us.
Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.
My personal threshold attempts to minimize false positives even to the extent that I will pass on people that would probably have done a good job. I tune it this way because I think that bad hires are one of the worst things that can happen to a team. But, each individual has to calibrate this for themselves.
I'm not looking to solve the problem, I'm looking to make the best possible decision based on the information I have available.
1. You need large samples of hires because you must know what features actually correspond with good performance
2. You need to have a near perfect understanding of statistics
Once Google and friends collected a large enough sample they concluded their puzzles are not a good indicator. Similarly, people tend to have bias towards certain skills without any statistical evidence that they are correlated with performance. And statistics is all about removing the bias, which has proven time and time again is very hard. Otherwise you would be just gambling on instinct, which at best is just a sanity check.
Basing it on the interview questions posted online.
Gaps in my algo knowledge are there on purpose since to me they are a solved problem and easily found online.
When looking for work, I have no room for companies that want to do this: I already have a job and commitments. Also, I might not even be in the same state as the company I am talking to.
Otherwise, just ask for a coded solution to some specific problem or user story and I'll work on it after hours and send it in.
Just the fact they were willing to do that meant a lot for me (because no one else had ever bothered how inconvenient it was for me to do interviews).
Still here 3 years later and the attitude shown during the interview process extended right through the company.
Hiring for prior experience - the author emphasizes this, and based on my experience this should be de-weighted. The issue is that as soon as you start to define an area of experience, your hiring pool shrinks dramatically. Consequently you are more likely to pick a less powerful engineer. Another, bigger downside here is that companies are terrible at figuring out what they really need. Consequently you've hired someone who can do A and B but one month in you realize that the thing you were missing is really C. A less powerful engineer will have a harder time transitioning over to doing C work, and you'll be worse off. Moreover they might be upset - they came in with the expectation of doing A & B, and they're sitting here doing something unrelated instead. Can't count the number of times I've seen this happen, to both the employee's and company's detriment.
The thing that this article misses is how to avoid hiring jerks. Terrible employees can be a net negative to a company. Jerks, particularly those in senior positions, are the thermonuclear version of terrible employees. They can demotivate and destroy the productivity of entire teams and cause your best engineers to rage-quit. The best interviewing tactic I've seen to weed out these people is watching how someone behaves when you "play dumb" in an interview. See if they're happy to explain something to you, or if they get impatient with your lack of knowledge. A related trick is challenging them on something that is obviously correct - see if they carefully prove why their answer is correct, or become antagonistic. Really smart people are good, but much more effective are those that can help bring everyone else up to their level.
Absolutely true. The reason I point this out is that the cost of a jerk goes up as the size of the team he/she is on increases. If you are a 3-man start-up, hiring a superstar coder who's sometimes a PITA can be worthwhile - have them work on some part of the project where they drive the whole thing. At 20 people, that person can become a huge liability - they will misdirect the team, demoralize other engineers, and eventually cause your best people to leave.
My examples illustrate how you might identify people with an uncontrollable mean streak during the candidate process, and hence filter them out. If you're building a scalable organization you must start with founders that aren't jerks and weed these people out of the candidate pool. If you start the company off with such a person, well, either live with it or find a new company.
Because of that, I generally avoid people to self-evaluate versus others. I think it's ok to ask them to compare on purely internal things (e.g., "What languages are you strongest in"); people seem pretty good at that.
I think that approach makes it a lot easier to hire nice people; niceness and humility are correlated.
'Those who know don't talk. Those who talk don't know.'.
Sometimes it's appropriate to hire a brilliant person who's unfamiliar with your problem space, but not often. Sometimes you need an expert who already knows what they're doing and doesn't need to ramp up on the company's dime.
Are you seriously proposing that any serious applicant would be willing to take a week off from their current job just to interview for a possible job at Amazon?
Yeah, this is pretty painful. I've phone interviewed with Amazon twice. At the time my resume was pure front-end developer with just a smattering of server-side experience. From the Amazon recruiter everything was smiles and 'perfect'.
The interview was for a server-side & database developer position with warehouse software. WTF?! Did you look at my resume for even twenty seconds? I got database design and management questions mostly.
That interview was brutal. What a waste of time for both parties.
I mostly had fun with it, but I was also frustrated.
The phone screens were the hardest in a purely technical sense, lots of algorithm questions that really didn't have anything to do with the job at all, I had to reach into my way-back machine to dig up some of the runtimes. But I made it through three of phone calls like that and had a bit of fun. After the phone calls and a chat with a senior person in the department, I was pretty sure I had a feel for where I was going to end up and what kind of projects I'd be managing. So I boned up on management theory and algorithms and a few other odds and ends.
Flying out to Amazon for the face-to-face, I realized I was wrong. Each face-to-face interview was wildly unfocused, I was interviewing in subjects the interviewer often didn't have any experience in and completely didn't reflect the nature of the phone interviews or the statements I had been given by the senior guy I spoke with. In one interview I'm designing Java APIs for movie tickets, in another I'm specifying contingency plans for overseas warehouse fulfillment issues. In another I was asked by two very junior people who frowned in confusion the entire time as I discussed complex employee and co-worker conflict resolution cases I've experienced over the last 18 years. In fact I was never asked a question about the first 15 years of my work experience, most of which would have been more relevant. But I was asked tons of questions about my side-startup, which was just me and my wife hacking around on the side and getting passive ad revenue and had no relevancy at all to the job.
The "lunch chat" was with somebody who couldn't have made small talk if he had a gun to his head. He just sat in silence and ate his soup the entire time while I uncomfortably ate one of the sloppiest turkey pesto sandwiches I've ever encountered.
I definitely didn't nail the face-to-face, but that's because I was playing trivial pursuit with the interviewers and the answers were arbitrary, instead of knuckling down on the job, the role, the department, the expectations of performance and the responsibilities of the position.
But I did something right because I was called in the next day to go over compensation and was blown away to find out the absolute max pay cap for the entire company was almost a 20% pay cut from what I'd been making in my previous job (a job that I was leaving because I was 10-15% under my peers and had been in a pay freeze for 3 years while the company worked towards profitability -- and yes it was from a metro area with a similar cost of living to Seattle).
Despite that, Amazon would have been good to work at, but even my absolute pay floor was 10% over. I even asked to make up the compensation difference with a mix of stocks or other methods, but eventually we had to agree that there was no way I could live the life I had become accustomed to with what Amazon was willing to pay.
To be fair, I think this describes most people.
People who are excellent at small talk have to be very charismatic and seem genuinely interested in the conversation even when it's very boring. Very few people are capable of either of these things, and in my experience it's actually management & marketing types who are the worst at both.
If not they waste 2 weeks of their time and probably other SDE time. It leaves no room for people who apply while still working at another job. I had to miss 2 days of work for my interview, and that was about the biggest risk I could afford. Even if you say you will pay me for the 2 weeks, I still wouldn't have a job to come back to. It's probably still a poor job, I was just starting to get into my first task towards the end of 2 weeks. I spend most of the time just getting everything setup (granted we ran into issues that had to be resolved with another team, but still).
I appreciated the fact that a lot of the interview focused on core data structures / algorithms and not languages / frameworks of the moment kind of thing. It caused me to go back through my university books and study a lot of the core concepts that I enjoyed learning and have gotten away from by doing more menial programming tasks (not to say that I won't be don't those sorts of things at the job).
Also, if they said that they wanted to try me out for a couple weeks I would have refused the offer. I already had a job that I was happy (and well paid) at, so why should I leave or take vacation to move across the country and be tried out at another company?
The article pays lip service to the fact "There's no way to A/B test hiring decisions" but then goes on to describe a handful of rules to "build an effective organization, regardless of size or resources." The thing is, there's no evidence that these are effective and definitely none to suggest they're generalizable to organizations of every size. There's an appeal to the author's authority as an Amazon VP (admittedly a hell of a credential), but that's not evidence of effectiveness.
The fact is, hiring is tough to learn. It's made especially difficult because your feedback comes at least 6 months after you make the decision. I've had the good fortune to be in an organization which grew from ~10 employees this time last year to ~80 now, so I've gotten to see how a few dozen hires panned out in a short period of time. I learned a lot through that. Even so, as a professional data geek, I couldn't say with a straight face that I "know" anything about hiring. I do my best day in and day out, but saying you've got hiring all figured out is the professional equivalent of knowing "What (wo)men want." It's a sign of confidence, but not necessarily knowledge.
Don't get me wrong, in the absence of quality data, I'll take the wise words of someone who has been there and done that. But I'd much rather some empirical proof or at least some "YMMV" humility from those who purport to know.
I really don't care about what you did at your last company or if you raised this by 50% or that by 30%. Rather, I want three things in a candidate: smart, talented and hungry. If I get someone who is smart, talented and hungry and I drop them into a situation where there is unlimited upside, my job then becomes to guide them to greatness.
I also strongly disagree with the notion of not asking questions you do not know the answer to. Anyone can read an obscure section of the C++ language spec or a chapter on some data structure 5 minutes before an interview and make themselves feel superior to the candidate. What does this prove? Nothing. I often ask questions I don't know the answer to in order to determine whether or not the candidate can teach me something. I love having people on my team that have the ability to teach others.
The author's point is very well taken -- interviewing is an important process and it should be conducted in a way that results in the best possible outcome. But, he's looking at the situation far too rigidly for my taste.
At this point, they don't need money or validation, they just figure they can help you solve things and are talking to you because they enjoy solving new things.
Hungry is overrated when what you need is focus on solving the business' problems. This is one reason VC like to see founders take "shelter money" off the table early.
Ugh. I hate this. For a technical interview, why is leadership potential a priority? Speaking for myself, I've done the leadership thing and don't care for it. People tell me I'm good at it, but I still prefer dealing with algorithms over humans, and I actively decline offers for "promotion". (I already have my very own batch of little humans of to deal with, thank you very much.)
Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker. In terms of the corporate efficiency calculation of effectiveness/hour, I suppose that's still true. But for me, personally, that's not the equation that I care to maximize.
Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate.
> Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker.
The places where that is least true are also the places where evaluating leadership potential in technical workers is the most relevant to immediate job performance.
Does it work, by which does it produce the desired results or benefits? It seems somewhat likely that this would have a side-effect of pushing employees to asserting a dominating personality.
http://www.amazon.com/Values-Careers-Homepage/b?ie=UTF8&node...
It's being a leader in what you do, which isn't necessarily the same as leading a group of other people.
What's a Roseman object? Is it some psychological classification of the means by which we just things? Of course, I mis-read it. Shame, that could have been an interesting topic.
Wait...what? Is this a thing? Are people actually putting this on their resumes?
I had the highest SAT in my county (still just a 1360/1600), but I don't go around putting 2002 "STAR student"[1] on my resume.
[1]Local award for highest SAT score. I think each school in Georgia awards this.
And more recently, a former Zynga VP.
Surprising that such an expert at evaluating people chose to work for Zynga, of all places.
I've interviewed persons that did very very well during the interview process, but failed miserably on a day-to-day basis.
(1) Spend a lot of time discussing accomplishments at their current / previous employers. If you drill down on particular projects they should be able to explain, in great detail, exactly what their contribution was and how they worked with the rest of the team to get that done.
(2) Pick a technical question that has a lot of depth so that you avoid false positives because people have just looked up the answer on the internet. I would also add that in my view the best questions have no tricks and require no "aha!" moments. It should be something the candidate can be reasoned through step by step so you can see their thought process. When they come up with an initial solution, then you can dive deeper by expanding the problem or adding additional constraints.
Using "calibrated" questions that you have used before and know how far people get with.
Focusing on a specific accomplishment or area of strength of the candidate to probe. (As opposed to choosing something from the resume that, say, the interviewer was most comfortable with, or that the interviewer found interesting.)
Paying attention to preparing the interviewers, so they know why the candidate is being considered, they know why they are on the interview list, and they are experienced enough with interviewing to give a useful opinion.
That isn't even getting into the the conflict between whats good for the company and whats good for you.
To quote the NY Times interview with Laszlo Black of Google - "Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess, except for one guy who was highly predictive because he only interviewed people for a very specialized area, where he happened to be the world’s leading expert."
That being said, there are a lot of very disorganized interviewers out there, especially among seed stage companies. I've been amazed about how many people grumble about the "talent shortage", yet regularly miss interviews with "hot" candidates because of meetings, forget them altogether, or just lack a sense of urgency & speed when it comes to running an effective hiring process.
A large part of having a hiring culture is about decisiveness. Loosing a fantastic Engineer because you spent 4 weeks making up your mind, or you didn't want to pay an extra $10K/year is just silly when it means that you won't hit your product roadmap goals.
I think it is far more useful to ask somebody about what they actually did. Are they excited and engaged by it, did they come up with novel thoughts about it, do they understand the CS literature in relation to it, and so on. Program a hash table on the whiteboard when you have been just fine using the built in one in your favorite library or language? I'm not sure what that predicts about my on job performance. If I had to do that I'd reread the literature and then whip one up, easy peasy. I can state that with confidence because I read the literature on other topics and do that every day. Have me explain the new numerical filtering technique I just implemented and I'll give you good interview. Ask me to implement a smart pointer? Not so much.
I believe this is a backdoor method to requiring a college degree for positions that do not require one within labor law.
That said, the big G does hire a number of people without degrees, I hear.
I believe software engineering wages will continue to rise over time, and will eventually reflect the true value the industry produces.
In fact, the way you phrased it, being a quant sounds really fun :) you don't want to play multi-million dollar poker?
I mean, sure, meaning and existentialism and all that, but I bet it's at least somewhat exciting.
Valuations of companies are only important to investors because of the inherent gambling. If you purchase capital at a single valuation that never rises or falls, what's the point? Even an index fund pursues gradual increases over time.
...Or am I building a strawman here? Is this not where you were going?
"What does it do?"
"It revolutionizes the way you share pictures of your cats!"
...Yeah. Playing multi-million dollar poker on Wall Street sounds a lot sexier than the shiny startup veneer put on another photo sharing app.
Sorry, caught a nasty flu..
Soory if you have problems with reading as a literate dyslexic I had no problem with reading writing not so much :)
For the record, I do not see hiring the "best available" as settling unless they do not meet some minimum requirements of the business. The purpose of hiring people is to help execute a product, service, or whatever else your business is trying to make money off of. Anyone who can increase your chances of succeeding or increase your level of success is a good hire. Yes, that includes looking at intangibles like team and culture fit, work ethic, etc. But this person does not need to be the "best" to provide this boost to your business. In fact, it is effectively impossible to hire the best, because there will always be someone better.
The PHB might think they are over paid but like it or not this is Laissez-faire capitalism pay is determined by scarcity and bargaining power.
ps for those outside the UK Luton is the unfortunate University that props up the bottom of the university league table - though to be fair they do rank well for Nursing degrees.
If you couldn't find such a person, the person you hired is hardly overpaid.
Github.
Why is this so hard?
i've wondered lately if it'd be better, during an interview, to choose a problem that both you and the candidate don't know...and try to solve it together...
it's somewhat disturbing that he likes to use the STAR framework, which normally leads candidates into artificial responses. is there any reason to believe that this framework leads to better results?