Whiteboard Interviews Suck, Get Good at Them Anyway
alexkras.com
alexkras.com
Sure, as the author notes, WBI have a high false negative rate. But if you're actually a good programmer, learning basic CS concepts should be extremely easy, and these types of problems, perhaps with a bit of practice, should be relatively trivial. I personally wouldn't want to work with someone who can't extract a path from a DFS, not out of any sense of elitism but just because such an employee likely would be unable to do many other problems properly.
> there's no good way to independently evaluate people's
> abilities
In addition, in my experience most WBI are not particularly challenging problems that you have probably never seen before. I mentioned the ability to extract a path from a DFS; I would argue that this is such a basic CS skill that being unable to solve that problem indicates an insufficient amount of knowledge of CS.
There are definitely WBI that are poorly matched with real job requirements and knowledge. I think these are often seen at smaller companies trying to copy FB/Google without knowing what they are doing. WBI that involve using important CS constructs such as a binary tree or a directed graph IMO are testing important concepts
On the other hand, people's performance in whiteboard interviews may positively correlate with later performance as employees. In which case they let you make a reasonable assumption about later performance without being able to evaluate it directly, which is really what interviews are all about.
That's why this practice should be abandoned. Give the candidates a small, two dayish homework that reveals their ability to do their would-be daily jobs. If it looks good, call them in and talk about their submission. Heck, you might even ask them do draw some diagrams or detailed explanation of their own solution.
Not if the cost of a false positive is much higher than the cost of a false negative, which for many employers seems like the likely case.
I find the opposite. Most real world projects require a week or two of infrastructure setup (logging, error handling, IOC setup, database setup, etc) that you won't have in the two day version, yet in my experience most companies will grade you on the lack of such things.
Also, anyone already employed isn't going to invest that much time, which largely limits your candidate pool to employees no one else will hire.
http://www.straitstimes.com/world/extreme-measures-to-crack-...
https://blogs.voanews.com/student-union/2017/02/27/sat-score...
https://www.quora.com/Do-Chinese-people-have-creativity/answ...
I think this is absolutely the case at large companies which need to screen hundreds of thousands of candidates per year, perhaps less so at smaller companies.
It makes perfect sense to look for a way to evaluate people before we hire them. It's a real problem, and we must do something, but do we have any basis to say that the whiteboard interviews correlate with good performance. Otherwise, we go back to the classic "syllogism" to justify policy proposals
1. This is an important problem, and we must do something. 2. X is something. 3. Therefore, we must do this.
I've been developing professionally for over 15 years, and I am quite good at whiteboard interviews: 100% offer rate. That said, Many of the best developers I've worked with are not any good at whiteboard interviews, often because they lack the composure under pressure, or like to take some time to think everything through, which are correlated with scoring badly in those interviews. Therefore, no matter how good their code is, they'll have a lot of trouble getting into a company that does whiteboarding interviewing, or even practical interviews under time pressure.
I also have found that, in my list of things on what makes an effective member of a programming team, being able to come up with a DFS algorithm under pressure is nowhere near the top of the list. An attitude towards trying to learn what others bring to the table, instead of discounting them immediately because they aren't good at some specific CS shibboleth, is far closer to the top though. We just test for random algorithm questions for this because it's easy to measure, not because it's really all that helpful, and that's pretty unfortunate IMO.
Depending on the exact role and project, that can be a severe negative as well (e.g. devops). In that case, putting them through the stress of a WBI is a side benefit.
And Im not saying that meeting deadlines is the only stress, Im just saying that being stressed out about how you will provide for yourself is different than the stress of having to perform well at a job
How about straight up pay them to solve a real problem with you? If they're clearly incompetent, cut it short fast (under an hour). If they're competent, you know that they're competent at the job, not at a whiteboard interview. Yes, I've seen this work at small-medium outfits.
> I personally wouldn't want to work with someone who can't extract a path from a DFS
What if this person hacks blockchain merkle trees for breakfast, but can't be bothered to regurgitate tangential runtimes of whatever algo you decided to ask about, which they memorized and forgot 10 years ago? I've seen this to be a nontrivial intersection of skills.
So then why is a whiteboard interview different than a real problem? They don't have access to a developer environment but that's the only major difference.
One should be able to reason about running times, not memorize them. If a candidate cannot explain why their implementation of an algorithm they've never worked with is actually quadratic in disguise, for example, then I'd need to babysit their code if they were hired to make sure the system stays efficient.
But most runtime derivations are pretty basic math and logic. You don't need to memorize a runtime of some algorithm, because if you understand the algorithm and what runtime analysis is about, you can derive it (in an informal hand-wavy way, at least) on the spot on the whiteboard. I wouldn't expect a rigorous derivation of the master theorem during an interview, but being able to reason a bit about worst-case scenarios and the iteration bounds of nested loops or a tree traversal seems like a pretty fundamental skill if you're discussing algorithmic problems.
Not only does it take a lot of time and bandwidth to fire someone incompetent, but now you have to deal with the damage they've done to your codebase and projects in the meantime.
Even if your WBI process rejects 10 good coders for every bad coder it weeds out, you're coming out ahead.
As for all these posts from famous or well-known hackers who assert that they don't need to show that they know their stuff -- sure, if you have a portfolio of open source projects that everyone uses, then you have the luxury of not needing to demonstrate your technical competence.
How does that help employers evaluate anyone else?
1) All new employees go on 90 day contract-to-hire. So reduce the firing overhead to 0.
2) Similarly, just build some sort of extra ring of QA around new hire code, so all their codebase changes get reviewed by a peer -specifically to assess their long-term employment at the company.-
That 10:1 ratio's not set in stone.
The pressure of dumping pseudo code is a poor metric.
Brainteasers and fizzbuzz are very different things though: I'm not totally against brain teasers as a prompt for discussion, but they're not my favorite types of questions, and can be hard to judge or of little use if the person either already knows the trick, or just doesn't figure it out. Fizzbuzz on the other hand makes a lot of sense when you're interviewing someone who you don't have any other knowledge of their coding. That type of question absolutely makes a good first pass filter: the problem is easy to explain, the implementation is trivial to do in a few minutes if you are at all qualified. Sure, it's annoying for an experienced person to have to do a fizzbuzz variant for the 200th time, but given that it takes almost no time, and helps the interviewer quickly jump to more advanced stuff or cut off an interview that isn't going to go anywhere, it's a pretty minor inconvenience.
I think the author is hitting on some deep-seated psychological aversion people have to white board interviews. If you can identify that you were turned down based on your failure to do it well, you have to accept that it's your fault - not that you happened not to be good at that tech stack, or didn't have a "personality fit".
(I know there are different kinds of white board interviews - I'm assuming the generic pseudo-code algos and data structures type in keeping with the article's generality.)
Exposure to that kind of evidence of your inadequacy can be hard to accept. But it's also remediable. Get cracking and practice.
Whiteboard interviews have a very high false negative rate (rejecting people who are good), but they also have a low false positive rate (hiring people who are not good).
On what basis is this statement made? No study or source appears to be cited. The specific false negative and false positive rates are not given. Also, what exactly is the definition of "good" used?
I am quite interested in the false positive rate which can be computed by correlating performance reviews with interview performance.
Isn't it more likely that they were correctly rejected multiple times and just got lucky enough to squeak through one time?
You would still be excluding all the false negatives making it kind of relative performance of interview survivors. Meaning that you won't have anything to compare your false positive rate to proclaim it as low or as high.
You can pick up unfamiliar code/data structures easily
You can understand a given problem space without too much explaining
You can code the "easy stuff" without thinking about it too much
You can distill and discuss the difficult parts of an algorithm
the next best approach --- a take home assignment --- is also good at this (or better, IMO, since you're free to implement that solution however you damn well please and it more closely translates into the work they'll actually be doing), but people complain that this is too invasive of their time and that they are doing free work. which is it?
I have to imagine that what most people want is to be asked about their experience, talk through it, and expect that to be enough. which is a very solid first pass filter. it's just not strong enough of a filter to extend an offer with confidence in many situations; it's too easy to bullshit and most of us engineers are damn expensive. (it's not just salary; we want dedicated offices and sit stand desks and super comfy chairs and free food and drinks and games and cadillac health insurance and 401ks and so on and so on)
i also never understood why whiteboarding was hard. if you know how to code and know your language passably well, then why can't you write code on something that isn't a computer?
I am not saying WBIs are good or bad - just stating that this kind of rally won't have the desired effect.
It really might be the correct way to interview for Google. Certainly they've put a lot of thought into how to manage their hiring process and they still lean heavily on the whiteboard. I'm almost positive, however, that most companies are really nothing like Google and really should not be using the same interviewing techniques.
Smaller companies with more quotidian problems would be better served with a more conversational and less time-pressured interview style.
That's a pretty bold assertion. Is there data to show this?
I consider myself a competent programmer since I have developed decent work@work and my peers recognise it too. But how would any interviewer can know about this ? My resume is like any other resume in tech world. They cannot. So I've to be put through WBI. For many years I couldn't have known what was wrong with me. Every time I failed I felt miserable to be honest. Eventually, I have narrowed down the reason. It is not just at interview I fail to problem solve - even when I try doing at home with a white board and 'explanation' I fail. English is not my region language but one would not doubt on my conversation abilities as I manage it ok. It is that I simply fail to explain solving programming puzzles in english since while programming I 'do' think in my regional language and most of the time in 'silent' mode in my head. Now 'silent' mode is again dreaded in interviews - as interviewers they want to know 'how you approach solving a problem'.
I think a better solution to WBI would be to make interviewee do programming questions on a laptop (alone) where interview is held - say make it an hour worth of exercise. Have that program run through some time bound tests (like google code jam etc) and may be do a code walk through later with interviewer. This way interviewer is sure interviewee can code.
Also, one of things that will definitely help would be to have some sort of evidence that you can code (git, code competitions etc). This gives some sort of confidence to interviewer.
I agree the process sucks and it's a hoop to jump through, but why take it out on the interviewer?
I've written integer programs, ray tracing simulation software for fields of heliostats, custom orchestration tools and deployment pipelines, etc. and not once has anything said or done during a whiteboard interview been relevant to any of it. The actual job is pretty much orthogonal to whatever the interviewer thinks they are figuring out during a whiteboard interview and as above the stuff I've worked on goes over the heads of 90% of the programmer population so 9/10 they can't and are not qualified to judge things properly but somehow they still think through the magic of the whiteboard they'll divine the candidate's true capabilities.
When working on hard problems I go to the library and don't stand in front of a whiteboard to see if I'm smart enough to reinvent some wheel. Chances are someone else has already figured it out and I just need to read some paper and then implement it. If they haven't figured it out then that means the problem is harder than I thought and will either need to rethink my approach entirely or spend my time on something else.
Unless of course the job is to stand in front of a whiteboard and wave furiously, in which case you should be using a whiteboard to interview.
I get that its a horrible thing for some to consider that there is creative freedom involved in work and not everything in life follows exact and well defined rules. Anything else, is just holding humanity back. I ve seen standards written by assembly programmers, still alive and walking the earth, torturing hundreds and undead for ever, because what is once set in motion.
Really good standards, standardize intentions. "Every Programm shall consists of layers and components. Every component shall be sperated by a interface."
A bad standard tells you how the interface has to look like- every time.
(Obviously, this goes both ways, shitty companies tend to have trouble getting good people to interview.)
... then perhaps it's best to let them know and save everybody some time (including your own).
Unfortunately, the whiteboard thing is not as standardized, so it's somewhat harder to teach to the test, but it is not impossible. People waste a lot of time I think when they can memorize a small number of basic algorithms and then learn to map the question given to one of those techniques.
But writing the code with a marker on a board definitely takes practice as well.
The available training materials are good, but they are not great. A focus on shortcuts would be nice. I remember my verbal SAT improving significantly just from the single tip to read the reading comprehension questions before reading the passage. I suspect there are similar tips that can improve your speed at figuring out the trick that solves these interview problems quickly.
So, I generally feel like it makes more sense to try to determine the actual aptitude when you have an opportunity, instead of trying to measure it by possible decades old proxy with huge and unknown variance. If you had good grade and/or spent some time studying for the interview, and really meet the hiring bar, you'll most likely do well in the interviews anyway. There's some noise in the process, and good qualified candidates are sometimes rejected, but from company's perspective, it's better to have some false negatives, than to give another metric that can be easily gamed, and get false positives. Google has enough qualified candidates applying for a job to make it not that big of a deal if you lose some due to the noise.
You have no idea how much more difficult that made it to follow sorts and tree algorithms. Thankfully, I seemed to have some sort of coding gift that allowed me to write the damn things at the terminal and make them work.
I got A's in all my CS classes, but now I guess I am so used to the magics of PHP object handling I've literally stopped caring about that shit.
It's hard to see why I would want to again.
So why should I be expected to regurgitate trivia at a whiteboard to get your job?
As an aside, I've done many, many interviews, and for a long time my manager insisted on jumping in and asking candidates to recite the components of SOLID and explain them. You can probably guess that very few (none that I can recall) got it on the first attempt. Good candidates could talk about the concepts if you reminded them what the acronym stands for. To me this means the question is useless. Ask about the concepts and skip the trivia.
That's essential stuff, not something to be dismissed as "textbook details".
If you don't use these as your whiteboard questions, great. But I've not run into you or others like you in my own searching.
+1 though for "Senior Whiteboard Engineer"--that term is genius!
I've been through that system a couple of times, it was both more enjoyable and gave people a far better picture of my skills than just how well I remember a tree traversal algorithm that I haven't had to write in forever, as data structures often handle their own traversal.
In general though, when I interview it has three main sections, tailored for the role.
First, is conversational. Talk about the resume, talk about work experience, talk about technology, etc.
Second, is a practical example. Sometimes (but not always) this involves solving an algorithmic problem on a whiteboard. Sometimes it involves a guided tour of the codebase and functionality of a relevant project the candidate has worked on. Rarely it might be a take-home coding task.
Third, is a more general example usually of some kind of system design problem. A lot of candidates like to write/draw on the whiteboard for this section but it's not strictly necessary nor do I ask them to do so. There's a board in the room if they want to but I don't ask them to use it.
Have a conversation about "how would you solve this (real) problem" - the same kind of conversation you have with a peer when they're stuck on a problem. If both you and they are professional developers, you will be able to get into a nice in-depth discussion of the pros and cons, and go into nice tangental discussions along the way.
If they aren't a professional developer, it will show in no time at all; the longest I've let such a conversation go was 10 minutes just to see how long the interviewee would keep trying and snowball me.
I've found this is simply much faster and more informative about their experience with subjects that matter in the daily work of a software dev.
So yes, whiteboard interviews or not optimal and they are unfair, but if you are rejecting them out of hand and refusing to study for them, you are only taking away some opportunities.
I say this as someone who hasn't had to do a whiteboard interview for 10+ years. The jury is out as to whether I'll take the pain to study for them again should I do a full job search among tech companies again.
Also: note to author, I believe, "They have little co-relation" in your opening paragraph should be "They have little correlation".
If they do WBI's then they probably do lots of other things which run counter to best practices. Isn't it so much better to expose to the outside world a culture of cargo culting inability to evaluate techniques for the efficacy.
There are so many orgs who do run themselves well. The real tricky party is how to determine that quickly. With WBI's we can get a short cut.
Long Live The White Board Interview!
I don't think the data supports this. Say what you will about Google, but they do WBIs and the average quality of their engineers is very very high. Same with FB.
How would you hire?
Perhaps use it as a warning sign, but you might want to talk to your potential co-workers as well and see what they're actually doing and how.
And what are the best practices for interviewing? Whiteboard interviews are employed by plenty of successful software companies (Google, Facebook etc), "best practices" implies there is something clearly accepted as more effective that they should be using instead. What is it?
Now if you say whiteboard programming, and that the candidate has to pay attention to curly braces and semicolons and typos, then I agree its not a great process.
So for me, whiteboard interviewing is perfectly fine, whiteboard programming is not.