Today's Coding Interview Game
shubharamani.com
shubharamani.com
From what I can tell, Google (and other companies) started using these kinds of questions as a way to cheaply filter sudden floods of applicants, not because this is the way to find the greatest geniuses of our time. Policies implemented under the gun have two unfortunate properties: they are wasteful and hard to change after the fact.
As the tactics of successful companies percolate through the rest of the industry, they take on a third unfortunate property: cargo-cult foolishness. Google asks binary search questions and they have great engineers. I want only the best engineers, therefore I will ask binary search questions. I don't know what else to do so I shall grade strictly on style points, because points are scientific, right?
Bleh. Put it another way: if there are a large number of productive creative people shut out by a culture of thoughtless interview checklisting, that's an opportunity for some smarter company to snap them up. Such a company would by necessity have an outlaw air about them. Who else would hire misfits? But gangs of talented misfits can be a powerful force.
You've just described how we ended up with many of our problems in the government, military, education & private industry all in one sentence.
A single upvote doesn't seem like enough.
Of the people actively applying for jobs at any given point in time, the "truly excellent" are under-represented as compared to the total population of programmers.
I'm not a Joel fan boy, but I think http://www.joelonsoftware.com/items/2005/01/27.html states this problem more clearly than I can in a few paragraphs on HN.
A poor salesman will have a hard time coming up with good or enough talking points. This alone is a reason resumes are mis-filtered.
I want to say that you should avoid filtering candidates based on their resumes, thanks to the holes, but that would be wrong: Sometimes, based on what they write down, you can make a reasonable judgment. For example, if you are looking for a software developer and the resume mentions NOTHING about that field, that is an easy filter.
Not quite so easy a filter: "I know C/C#/C++." but I always find that one amusing.
1) Select a set of candidates based on their resumes making the assumption at this point that their resumes are honest. Good candidates should have at least worked on some interesting projects so their resumes should look interesting. Most lame candidates wouldn't even be able to invent a credible interesting project.
2) Do phone interviews where I would ask knowledge based (ie not problem solving based) questions related to their resumes. The goal here is mainly to filter out fake/exaggerated resumes.
3) Invite the selected candidates to give a presentation on a topic of their choice related to work they've done that is at least marginally relevant to the job. I think you can learn a lot about someone by seeing how they present their work and respond to questions (though it doesn't tell you if they're good coders, that's for the next step). The presentations could probably be done remotely.
4) Hire the selected candidates for short-term contract work. Set things up so they can work remotely. Assign them real tasks that you would expect them to perform in the job.
Make final selection based on evaluation of the candidate's work on step 4 and offer positions to the best.
You are basically filtering for presentation ability. But 3/4 of programmers are introverts, and giving presentations is simply not part of their job. You're therefore selecting for the wrong skills until the contract step.
At the contract step you create another problem. Good programmers who want to be employees probably already have jobs and are somewhat risk adverse. That kind of person won't find the option of a short-term contract with no promises at all enticing.
Remember, as bad as the economy is, the economy for programmers right now is pretty good. And as annoying as the coding interviews may be, really good candidates don't have too much trouble with them.
...unless you are looking to hire someone in sales or someone to represent you. :) But I agree with you.
For something like software development, I think the presentation process can be improved by talking to the candidate as part of the presentation, not just letting them starve in front of an audience. Let's not rule out the presentation mechanism; you can learn about the candidate's ability to give presentations and handle pressure.
The assumption I'm making here is that of hiring for someone to work on technically challenging problems (ie. machine learning, computer vision, data analytics, large-scale systems design, etc . . . - the list isn't meant to be exhaustive). For those problems you need people with strong general and/or mathematical reasoning ability. If I was looking for a very junior coder I might skip the presentation step.
At the contract step there doesn't need to be any risk for the candidate - there's no reason he would need to leave his current employer during this phase. Of course there would have to be an incentive of some kind if he was hired - better salary, more interesting work, etc.
Presentations are not about powerpoint. They are about geeks geeking out with other geeks. Every talented programmer I know has a pet project he'd love to talk your ear off about.
The presentations we solicit are often little more than the candidate opening up a text editor and showing off some code, and fielding tame questions from the other developers.
As for the contract position, everyone that is motivated to switch jobs, or to come work for us, can find time to do the really simple tasks we split off for $100/hr.
It is also made clear that if you've made it to the contract, we are extremely interested in hiring you, or else we wouldn't give you access to the repo or the team. The gig is to give both sides a chance to see what working on something real feels like.
We have found only excellent developers after this process. That said, it probably doesn't scale very well, and I'm sure we've unfairly weeded out many good devs.
Also, we skip a lot of the BS if there someone comes highly recommended from current employees or other partners we trust.
And the problem with coding interviews is that they filter out other good candidates, and I've seen plenty of companies complaining that "They can't find any qualified developers."
I would not blame the developers for this, I'd blame their selection process for focusing on things most developers don't really touch or care about when they're working on a project.
This works because game development is relatively specialized and there actually are things that I expect anyone claiming to be a game developer to know.
Stuff like basic linear algebra (What can you use a dot product for? What can you use a cross product for?) and simple data structure do's and don'ts (What's one advantage of using an array over using a linked list?). Throwing in a few simple domain specific questions for the position their applying for works well too (I wouldn't ask a network programmer to tell me something about shaders, but I would expect a graphics programmer to be able to talk about them for a few minutes)
Some of the questions are designed so that there are lots of right answers, and some of them impress me more than others. For instance, if you mention that an array is generally more cache-friendly than a linked list and you can explain why when I ask you, you get bonus points.
If they do well on most of the questions then follow up interviews (phone or in person) are in order. Occasionally someone will nail all the questions, I tend to get excited when this happens.
I don't expect you to be able to code quicksort from memory. I do expect you to be able to code quicksort given the spec for it.
Just because I point out an error doesn't mean I'm counting it against you. I'm pointing it out to see how you handle it.
I do expect you to ask questions. If you are unfamiliar with an algorithm or forget the details, ask. I may not give you the answer but I'll at least point you in the right direction.
I generally try to make this clear to candidates so that they don't get uncomfortable if I point out some problem, but a lot of interviewers don't, which can be hard on the interviewee. I doubt, however, that there are many who expect perfect code on a whiteboard.
It's as much about the process than the result.
It can be astonishingly random who gets through an interview. Bizarre that just a few words can tip the balance.
As the blogpost pointed out, it's pretty useless to require the knowledge of the implementation of these items because 99% of programmers will never need to code it from scratch. If someone on our (web) team was building a new linked list library, they're doing something wrong.
In programming, almost anyone can start on the job - but we play out the hazing ritual throughout entire careers. Every time a programmer has to change jobs, she has to face a barrage of pointless trivia questions from megalomaniacs. Of course, many of them then grumble that they can't find competent programmers.
Also, I think that the interviewer anti-loop that Steve Yegge mentions (http://steve-yegge.blogspot.com/2008/03/get-that-job-at-goog...) is real.
Well, back when I was at Amazon, we did (and they undoubtedly still do) a LOT of soul-searching about this exact problem. We eventually concluded that every single employee E at Amazon has at least one "Interview Anti-Loop": a set of other employees S who would not hire E.
I certainly understand your frustration, but I really don't think either the tone or the content of this blog post is going to help land a job in this era where hiring managers ask Google what it knows about a candidate.
While I do give mental brownie points for people who can just write it in one go, I don't consider it a non-starter if you can't. But if you can't get to the point of having written it, with prompting and help, in half an hour, that's a non-starter.
Everyone assumes that the battle is between...
Person 1: -Great resume -Relevant work history -Strong sense of corporate citizenship -Can't code on white board
and Person 2: -Can code on whiteboard -Smokes crack in the bathroom
But the reality is, you're all interviewing for the same job! You're not getting the job, not because you can't whiteboard code, but because someone else has the same qualifications AND CAN whiteboard code.
Which would you choose?
I do interviews, I care less about the right answer than I do about the problem solving approach. I interview electrical engineers and I make them do simple transistor problems. If someone gave up and said that they couldn't do it without some sort of SPICE program, I'd cut the interview pretty short. But if the fundamentals are there, even if they don't remember something stupid, knowing how they approach a problem and if they know how to ask for help is more important.
Every time someone says "I didn't get the job b/c I couldn't do a linked list on the whiteboard." I'm fairly certain it was more than that.
Is it that you just don't like the kind of problems that are being asked (fundamental computer science) and would rather see more real-world problems? Or do you genuinely think the wide, stubby pen and vertical orientation impairs some people's coding ability? Just to note, I can see the performance aspect of the whiteboard is stressful, but coding on a machine with everyone looking at you is stressful too.
However, if someone couldn't code a linked list on a whiteboard, I'd be severely worried. Same goes for a binary tree. If you know what a linked list or binary tree is, you should have no issues figuring out how to represent them with code in 10 minutes.
The other trickier problems are there to test problem solving ability, which I believe is a valid thing to test.
I think the reason for this style of interview is that companies are afraid of false negatives. They'd rather reject 95% of the qualified people and 99.9% of the unqualified people rather than 94% qualified and 99.5% unqualified.
Additionally, swearing all over the place never helps your case...
And then they complain that they can't hire developers.
I don't think these types of interviews are well conceived. Human memory is extremely context dependent and a whiteboard/interview type situation doesn't establish the right context for someone who's accustomed to working alone with a mouse, keyboard and compiler.
Also the more knowledge you have the more difficult it becomes to retrieve that knowledge quickly. So these types of interviews might be good at selecting people who know all about linked lists and binary trees but not much else.
I also know I failed miserably at a trivial coding task in an interview.
So I don't know. If I'm the only one who has that kind of problem, maybe my brain works differently from other people's.
void move( char** colNames, int toColumn, int fromColumn );
Now I just spent 43 min implementing this with a compiler and some trial and error.
During the interview I was expected to do it with pen and paper.
Now my IQ is above the 99th percentile but apparently there are people who can do this sort of thing in their head without errors and that's what some interviewers appear to be looking for. That's why this post hit home for me, though personally I probably wouldn't have had much trouble with linked lists.
I've never heard this about the human brain before.
I'm not saying you're wrong, because I don't know, but statements of fact like this deserve a good reference.
Do you have anything to back up this assertion of fact?
Let's hook up your laptop to the projector so I can get a glimpse of how you interact with your computer. I learn more about you by seeing your relationship with your text editor, how you look up information you can't remember, etc. Not watching you sweat with a marker in your hand.
Are you sure you are not just learning whether the candidate is superficially similar to you?
PS: Oddly enough, I have little issue working on a multi language programs, it's just I think in terms of pointer not C pointer.
That said, I can imagine some cases where opportunity cost and being more concerned about getting good employees than immediately weeding out potentially bad employees would lend one to the probationary thing.
Huh? You're worrying about whether or not what you write on a white board will compile? How do you know? Press a magic button to OCR what you've written, download it into some computer somewhere, and compile it? Sounds like you're interviewing at firms with technology I didn't know existed.
Your interviewers have you code on a white board so that they can evaluate you, not your code. They want to see how you handle a problem, how you approach your work, how you think on your feet, how you deal with interaction, and how your intelligence and experience applies to their business. Anyone who worries about whether white board code actually compiles is missing the point. If they're more interested in perfect syntax than embracing you and your potential contribution, then you don't want to work for them anyway.
I think you're making this too hard on yourself. Memorize nothing. Just be yourself. Programming is like riding a bike; once it's part of your DNA, you don't have to worry about it. Just relax, trust your inner programmer, and let this become a self-solving problem; some jerks may reject you, but the right fit will come when someone sees who you really are.
Those sorts of questions seems appropriate for the position in question. Asking me to implement a linked list or binary tree? Not so applicable. Might I use these sorts of things? Yeah, but the senior engineers assume I know how to use Google.
It's also important to realize that most technical interviews are really about weeding out bad candidates than finding good ones.
It may also be necessary to have questions general enough that any given candidate stands a chance at solving it. If a candidate doesn't have SQL experience your example would be bad (assuming the company didn't require SQL experience).
This is why you see questions like "implement a binary tree." Any competent programmer should be able to implement a binary tree. If they have never seen a binary tree before, they are simple enough to describe in 5 minutes.
I once asked a candidate to implement a trie. He had never heard of a trie before so I drew one on the whiteboard and briefly explained it. He asked me a few questions and then proceeded to implement one. We then discussed the pros and cons of different methods to store the nodes (vector, list, map). It showed me that, given a spec, he could code a data structure and the following discussion showed me he understood memory and time constraints.
Of course, I can only speak for myself. I'm sure many programmers would immediately write them off.
That's a pretty glaring assumption, especially since it's subject to empirical verification. It also contains an assumption of its own -- that all programming needs a heavyweight familiarity with algorithms. There's a lot of programming contexts where the productive programmer just uses a hash so the mental bandwidth can be devoted to something else.
(FWIW, I do sometimes wish those programmers knew a little more about algorithms.)
And this is exactly why I would give the programmer the benefit of the doubt. If they are able to implement a binary tree then I wouldn't hold it against them (and I hope most other programmers would extend the same courtesy).
However I have never met a programmer who did not know what a binary tree was.
I wouldn't be surprised if there are programmers out there who ding candidates for silly reasons like that, but in my experience they are the minority.
From the other side, I certainly don't expect perfect code on a whiteboard, especially if they're typical mistakes that even experienced coders sometimes make.
She. And yes, probably she knows. One usually has the feeling which part of the interview they fail.
Good luck to you with job search.
At least Google's interview questions (from what I saw) made sense in their challenge. It's not stuff you would know from being an everyday programmer but it's stuff a truly great programmer should know.
1. It shows unfamiliarity with my work. It seems rude to invite me all the way into your office just to be filtered. In a perfect world (or at least the one I knew some years ago) I'd be invited into an interview because you were already familiar with my work and wanted to talk to me -- not just to see if I was worth talking to in the first place.
2. Anxiety. I don't operate normally under conditions where everything I say is being evaluated.
3. They can be easily regurgitated.
4. They don't really tell you anything unless you know what you're asking for. I've been in interviews for web development positions where we started getting into the finer details of C++, sorting algorithms, and file system implementations. These are positions where I was being asked to write web applications in Python. I've been writing web applications in Python for years -- I don't think I've ever actually needed to implement a file system or even write a sort routine (tim sort is pretty good and most sorting in web apps is done by search engines or rdbms's anyway).
The problem is most people seem to think that since Google does it, they should do it too. However, if you don't know how to ask the right questions, you're just going to waste everyone's time. My time has been wasted in interviews as the interviewee. Boring technical questions that lead no-where and have little to do with the relevant skills for the job; often just for the sake of asking technical questions and appearing smart. Anyone can do that even if you yourself don't know the answers.
In a perfect world you wouldn't need to look at my resume to remember my name when I walk in the door. You wouldn't need to bother "screening" me. You want me to be there to meet me and see if I'm the right fit for the job.
But I also understand that businesses get a mountain of resumes and that some large portion of them just aren't worth looking at. It'd be nice if there was some way that you could at least know where to look in the pile for relevant resumes. That way you could spend less time screening (how draconian sounding) and more time interviewing. Sounds like a problem some decent software could solve...
Even a lot of industry people don't have much to show. Plenty of decent programmers don't have serious side projects (or aren't comfortable releasing them to the public).
As for regurgitated answers, I keep my eye out for those. You can usually tell if someone has seen a similar problem before. If I think they have I ask them. I sometimes ask a different question to see if it was raw ability or familiarity.
They do tell you a lot. They tell you whether the candidate can actually write basic code or not. I'm not trying to see if you are a great programmer, I'm trying to see if you are a bad programmer. It can be really difficult to spot a great programmer, but it's usually very easy to spot a bad one.
Anxiety is a real problem however. I try to take that into account but it affects so many people in so many different ways.
Personally I think resume's are worthless. I think the best form of application would involve some form of code submission - either a simple "screener" problem or perhaps a personal project. That might still be followed by an on-site screening interview (people lie and cheat all the time).
All these tests are really assignments from Data Structures and Algorithms class, and being fresh out of college can help here a lot.
Interviewing is a two way street, and programming jobs require more than just programming wizardry. You will need to work with a team, you will need to communicate with management, and you will need to understand people who aren't programmers.
Start by understanding interviewers.
"He" could be gender-neutral, and perhaps they're encouraging equality by ignoring the sex of the referent.
Interviewers are coming to realize credentials, be they Microsoft certs or computer science degrees, are not great indicators of software engineering aptitude.
The more you believe in the accuracy of credentials, the less you would want to ask coding questions in an interview. Better to simply examine resumes for the appropriate certifications and then ask fit questions.
If on the other hand, you believe nothing but on the spot coding demonstrations can prove a candidates abilities, you would do exactly what these interviewers are doing. Perhaps these tests are not the optimal tests, but they sure beat credentialism.
I need to see someone code.
No interview process can be perfect, so unfortunately the trick is in correcting hiring mistakes early.
> people who are great but can't deal with the pressure
> and awkwardness of coding on a whiteboard in an interview.
Additionally, in our line of work sometimes you have to program in a team under extreme time pressures. We've found that most people who can't talk about programming while sketching ideas and solutions in interview, can't do it for real.
The upshot is that you should always be testing for what you actually want. We want programmers who can design solutions, work in a team, get stuff done, ask questions when unsure, make decisions when necessary, change their mind when appropriate, and write programs that work. That's what we try to test for in interview, and before.
%The Chicago Manual of style lists "Addison, Austen, Chesterfield, Fielding, Ruskin, Scott, and Shakespeare."
One word: Emergent design.
Coding shouldn't be done in large patches of perfect code, it should be done in small segments of perfect code that are individually tested for their correctness and build up the whole solution.
The fact I couldn't run my code every few lines would completely destroy my ability to produce anything beyond what I would deem pseudo-code even if it did, on the surface of it, look like real code.
I've boiled it down to my salesmanship, or lack thereof. I don't like talking myself up, and therefore I have difficulty conveying myself confidently. And, as it goes with vicious cycles, each time I fail to do so lessens my ability at future attempts.
> Mistrust in applicants runs deep among hiring managers, and I have found that I must still perform Houdini tricks during the interview process, even if an insider vouches for me.
Even though I know I'm accountable for my salesmanship, I get this sense as well. It's incredibly frustrating to sense that you're being looked over as being a phony trying to pull one over on a company (how does this work exactly, anyways?) More often than not though, I've noticed this happens more with hiring managers who are not qualified technically to hire developers.
Also, combine that with the fact that I hate my current job so much that I am afraid that my next just might suck as much, so I am being /very/ picky. I've turned down a couple of job offers that in hindsight I probably should have taken.
If you are shy or anti-marketing, you may need to work just a little harder in mediums you are comfortable with to network. (This isn't directed at you specifically, MC.)
HN is networking. Make yourself discoverable.
If you're extremely uncomfortable listing an email address (or lisp form that evaluates to one or something similar), you can reach out to me based on my profile, but I'd really urge you to list some contact information in your profile, even if it's a throwaway account that you can easily walk away later from if the volume becomes overwhelming. If you're reaching out to me, add "HackerNews" in the subject line, please.
I would add that you should hold an intelligent dialogue with your interviewee. This means doing more than just asking a question. Follow up. Not only find out if they have done something, but figure out if they understood what they were working on. Find some specific details in it. If you need a nice canned starter question for this: "What was the hardest problem you ran into when implementing X?"
In other words, interviewers should have filter dialogues, not filter questions. Genuinely learn about your interviewees. You must direct the conversation intelligently. That is how you can find the diamonds amongst the poor salesmen.
In my humble estimation, experts tend to be bad teachers because they have lost touch with the frames of novices. The best teachers are ones who have most recently mastered a skill.
In my current job writing server-side Java and Ruby on Rails code, I wouldn't dream of asking that question. It would be pointless.
I've never, ever, needed to write a linked list, a sorting algorithm, access memory directly and pass around pointers or any of the sample problems that are being cited here. I know the theory, I've done them in academia many years ago but I've got more important and useful things to remember on a day-to-day basis. These are firmly under the category of 'look them up if I ever need them again' and I'd be perfectly happy with a hypothetical candidate (I've not interviewed anyone in years) who said they knew the theory and would look up the exact details. I do that all the time with the more esoteric features.
At my last place, we did have a coding test as part of the interview. We checked all sorts of little things but could usually mark the entire test with some accuracy on the first question.
1. Please write a valid SQL inner join statement.
Seriously. For a position specifying SQL, to maintain an SQL backed custom business application, a majority of applicants couldn't write an inner join.
I suppose I'm saying - do whiteboard / paper coding tests, but do the _right_ ones for your projects. Don't give them a test on something that's clever but unnecessary for what you do, check they know the basics and are up to speed with independent learning and they should be fine.
As a linked list is nothing else but pointers and blocks of memory that are the nodes, and setting the pointers of nodes to point to other nodes. This is something any programmer worth his salt would soon deduce on his own even if he, for some mysterious reason, didn't know linked lists.
Many probably have in their younger days. And when they finally read about linked lists they go "ha, what I have written actually has a name". If you just merely keep doing allocations of similar structs or objects that you want to stash some place, you're implementing a singly linked list the minute you think of a "next" member in your struct.
blond = old.
Maybe you haven't been in their shoes, so I can't blame you for your approach, but the phrase "diamond in the rough" exists for a reason.
But I wouldn't pass judgment either way based on that blog post.
I speak my mind...I live authentically. Yes, it's a risk to live that way, but I am willing to accept the consequences of my actions.