Take-home vs. whiteboard coding: The problem is bad interviews
andrewrondeau.com
andrewrondeau.com
- take home interviews incentivize the candidate to spend more time than they’re told. You can choose a three hour problem, but candidates that really want the job will probably spend more time. Less experienced candidates will spend more time on it too.
- I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
- whiteboarding can go badly too. The caveat is that if it does, only so much of the candidate’s time is taken up; there’s a hard ceiling.
- from the perspective of conducting whiteboard interviews, the company loses out if the interviewer doesn’t do a good job. This is really on the company though - it’s worth investing in a good whiteboarding process. If there’s no investment here, it’s only the company that suffers (competent candidates will get hired by a company with a better process).
In terms of conducting a good whiteboard interview, my $0.02 is no trick questions, just solid problem solving with plenty of scope to show creativity and proficiency.
The candidate should be encouraged to submit early and often for feedback, and to not be afraid to ask questions. Sometimes a subtle or complex element of the problem statement can help spur this interaction.
Finally the grading should be a blend of both objective criteria such as a series of heuristics and unit tests, as well as subjective measures such as code cleanliness and structure.
Multiple people should be involved in the grading, and a repository of previous submissions should be compared against for consistency.
I and my team used this technique for years to great success, hiring some of the world's top talent.
How do you judge someone's half-finished work? Do you expect me to write half-finished code on the job?
It's just an unnecessary source of stress during the test.
Take homes can & ought to be timed.
I've had the opportunity to do this fornat twice and I think it works well.
Here's how it works: They ask you to pick a 3 hour time window to work on the project. You pick the time. They send you the project prompt in an email right at the beginning of your time interval. You have 3 hours to read the prompt, design, implement, test a solution. If you miss the deadline, project counts for nothing.
Both the times I did this the prompts were absolutely doable in 3 hours. They were the kind thing you could do in about 2 hours if you had the ability to read the project prompt ahead of time and think about it for a few days.
Personally, I work well with time constraints. This format feels closer to a workday where mid-afternoon you might challenge yourself to finish a task before 5pm. Its a lot less pressure than having to juggle a conversation and having your code scrutinized in real time.
It forces a real time constraint that keeps you from piling time into something you just don't know.
Its hard to cheat, and it also forces the company to make the task achievable in the time frame. This format forces them to realize when the task is too big for anyone to finish in the time.
I think I’d be fine if someone gave me 3 hours to complete a typical take home exercise, but I worry that some exceptional engineers I know might not be. Applying excessive time pressure could lead to cultural homogeneity.
Whoops, your home internet has gone down. Or Gmail has decided it's not having any mail from their domain. Or your mail provider think they're spam. Or your mail client doesn't pick up the email for half an hour. There's a whole raft of problems coming from "email delivery is reliable enough to be a timer."
My brain just not work that way. If I have to modify something in a project I already worked on. That's fine. But interviews always ask tho build something from the ground up.
Once I got the requirements I need a few hours to think about it, before I write any actual code. I usually do something else while I do this. On a job, it's when I home and do some chores and my mind can wander around freely. But I can't make this process much faster. If I'm on the clock I have to sit down and start spitting out code. Also, there is no time for any major refactoring if you went in a wrong direction. This adds a lot of unnecessary stress.
I like it when you have a day or an afternoon to finish a 3 hours long task. I work faster if there's no time pressure.
> I need a few hours to think...before I write any actual code...I work faster if there's no time pressure.
It sounds like you work better but not faster. If something absolutely must be done in 3 hours, you usually will not have 2 hours to do the dishes while you let it marinate.
To be clear, I prefer to let a problem sit in my head for a bit, too. I think the end result is better in most cases, for most people. But the real world has problems with hard and fast time constraints, or huge financial incentives to get something working in ASAP and clean it up later if necessary.
If something /brand new/ needs to be built in HOURS with huge financial incentives, that is a failure on so many levels of business that blaming the engineer for their inability to actually get it out the door in 3 hours is ridiculous
When I presented with a completely new problem I have to think about what is the best tool for the job, how I want to structure my code, how can I test it, what are the edge cases, etc.
A take home interview therefore gives me a much better picture of the real-life skill level of a candidate compared to a very stressful situation like a real-time whiteboard interview.
Also: Give your candidate a real problem to solve and pay them for their time!
The whole interview was inspiring. It's about learning and exploration. Worth watching if you understand Hungarian: https://youtu.be/rtqhA4TtBrI
Strange reasoning. If you want to cut corners, you use stack overflow.
Unless you know by heart the optimal way on how to reverse a string in C#, you do a 3 second google search and do a 5 sec copy paste of the one liner, from the top answer on stack overflow.
When I was programming as a 14 year old and came across a problem, it sometimes took me a day to figure something out. There was no internet back then. Now the most obscure issue is probably nicely documented with a top 5 of solutions.
Also, when I interviewed a year ago I was asked to find if something is a palindrome. This is programming 101 and it caught me off-guard. I got so used to all the methods available in C#, that going back to pen/paper and writing a simple for loop felt odd.
1. Obviously, its faster. I, personally, get paid for results, not for how "clever" I am. (Which is good, 'cause I'm not very clever.)
2. Frequently, the answer on the internet is better/more complete/tested. Take implementing a Singleton class; its so simple that anyone could make one. Most home grown implementations would have race condition though.
One of the ideas that is always brought out about the awful interviewing process is that there are just a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down. I'm not sure how many this is and not just bad interviewees based on experiences, but I do also accept that they exist (My guess it's 10-20% of programmers).
At one project I was on we had a guy who was pretty bad (he was then hired away by a competitor to their super programmers team so whatever) but I know he spent days on something that should maybe have taken 1 day, and it looked pretty bad (I mean just looking at it)
I can say that no amount of time would have allowed him to write something that did not look messed up (based on his habit of using map to update multiple arrays at once and not using the returned array from map anywhere)
Maybe they've used extra hours to get a good result, but if they are able to recognize a good result they are not in that small group of programmers that must be kept out at all costs.
either that or 25 dollars, same as in town.
I suppose it depends how you define "quality" - in my recent experience, it's always been the "rockstars" that are the worst developers. Because they generally write code which ends up being a nightmare to support and maintain when they're gone. Sure, it's great you wrote your own ORM in Perl but 10 years later it'll be crippling the company because no-one can make any changes without it breaking 27 other things.
(Ok, it's often Perl people...)
For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.
Also he thought I had done something using ElasticSearch because I was an ElasticSearch guy, instead of having learned ElasticSearch because I thought it was what was needed (and no amount of explaining this simple concept could penetrate)
Agree. But it should be noted that the needs of a startup may not always make that easy. I'm personally responsible for transmogrifying an existing Perl ORM for a company, because it only supported a single database handle / class. Which is a pain when you're in a master / slave environment. This hack allowed the company to meet the performance challenges that came along each year by summer. It also allowed growing the company from ~35 people in 2004 to about 5000 people in 2012 (when I left).
It also froze the version of the ORM in question, until it got rewritten from scratch at great expense. But that expense could now be easily carried by the company.
Hiring heavily from colleges results in 60%+ of applicants that somehow have a B.Sc. from a university that produces amazing applicants with great (3.5+) GPA's but stall at either fizzbuzz or (just in case they memorized fizzbuzz) i-can't-believe-it's-not-fizzbuzz.
Furthermore, some candidates (20ish percent?) can't even answer basic, non-trivia questions about their language they self-report to know best. e.g. Java programmers who either don't know or cannot articulate what the static keyword does.
I'm convinced there's a path through every university, no matter how good, where you can avoid all the good professors and only pick babysitter professors and squeak through your four years with a piece of paper and a well-tuned sensor for which profs actually grade homework instead of just giving 100's so people don't complain.
But often when I am brought into a big organization there is one out of every 5 programmers that just should be something else. They're not necessarily stupid, just shouldn't program.
on edit: in my experience small organizations don't have quite as bad because for the organization to be able to exist their programmers must be able to program. (for any organization heavily focused on tech of course)
If you just evaluate them based on their code you doing it wrong. You should organize an in-person code review. Where you ask: Why they do what? How would they improve their code? If something is not right, point out. Look at them how they react. Good candidates usually try to fix it. Bad team players usually defend their implementation until the end. It really easy to find out if they just copy-pasted some code without understanding it.
Instead give them multiple different hard (but realistic for the position) problems and be on their side every time to guide them. Try to get them to do the best they can and observe them all the time.
This way you can get good feeling for how they are as a person, how they react to guidance and teamwork, and false negatives where a simple miscommunication leads to wrong solutions and a bad result, despite the candidate actually beeing good, are reduced.
If they constantly and heavily rely on your guidance that is a bad sign. If they actively fight against your guidance, this is bad as well. But in the end you are ”casting” them for a certain position within an existing "cast", and sometimes a person who initially needs more guidance will do great after a while if they are in an environment where they can receive it, while people who actively avoid guidance could work just fine in a position where they don't get too much anyways etc.
The opposite of this approach would be to ask an actor which pieces of [insert obscure european postramatic theatre director here] they know and if they know none assuming they can't act. Bonus points if the project doesn't have any connection to the topic asked at all. If anybody ever does this, it says more about the person asking, than the person answering. No — try really hard to get the best out of the candidates in the most accurate emulation of the job you want to give to them, and have a grasp of what they will get up to speed with in no time and what actually will get worse (e.g. personality traits). And if you areninsecure about someone give it another shot in a second round.
As for the opposite scenario, you’ve got that right as well, except that the interviewer is a 25 year old, recently graduated theater major.
Insert there what you want, hehe. It always reminds me of one teacher I had, who always wanted to get oddly specific answers on his (always a little to unspecific) questions. He was so happy when nobody could answer it, he probably forgot what his job actually was. Behaviour like this tells more about the psyche of the interviewer than it tells about the candidate. Sometimes a behaviour like this is also the interviewers way of making sure the candidate doesn't expose them as the fraud they deep down feel they are: "See they didn't know that oddly specific bit (that I know!), so they must be worse than me – thank god!" In case of my teacher he felt insecure enough to feel the need to one-up teenagers and instead of teaching them something.
Interviewers are only human after all as well. But it shouldn't be the candidates job to rub interviewers in the right way in order to make them comfortable with their own shortcomings as it shouldn't be the pupils job to massage teachers into actually teaching them something.
The person gave me a piece of paper with a program on it with a number of problems, and asked me to find them. But she was too involved with my problem solving, sort of looking at me as I worked through it, and I was sort of off-balance.
I was almost about to say wait, let me do this on my own, but it was hard for me to be sort of rudely assertive because it was an interview.
Meanwhile other interviews where the person would give me a problem and ignore me would have me doing fine.
What you should not do is constantly interrupt while they are doing their thing. Of course you watch them. Beeing watched is part of their job — it isn't for a programmer or someone in tech.
However it can also be hard for actors to feel watched. Contrary to popular expectation they are often not extroverts — this is why it is key to develope some sense of trust beforehand, e.g. I tell them that I will guide them and do all I can to get the best result from them. So I usually make it about solving the problem and not about them.
You want a wholistic interview process that treats the interviewers with respect and mimics what the actual job will be like, not some artificial interview scenario.
It was just sort of an anomalous interview for me. I remember being asked, are you sure there's nothing more wrong with the program? and I looked back and couldn't see anything. And she pointed with her pencil and said "how about this?" and I was "Oh, yeah I see that now".
After the interview, of course I was kicking myself.
A friend of mine interviewed for the police, and he told me one of their questions. It was something along the lines of "You pull over a person for drunk driving, and when you get to the car, it's your mother. What do you do?"
My friend said "You call another car to take your mother home"
and I was thinking "what??? that seems so..."
and he said, "Look, they don't want someone on the force who will throw their mother in jail."
sometimes the answer to a novel situation - during a test - only dawns on you later.
When presented with white-boarding interviews, I'll usually feel physically sick. This includes intense sweating and weirdly enough, gassy-ness. I found this old blogpost pretty relatable: https://aneccodeal.blogspot.com/2014/02/interviewing-for-anx...
The take-home portion serves a few purposes: it filters out uninterested applicants, weeds out non-coders, and lets people prepare in their own time and way (rather than on the spot in an interview). The in-person portion then checks that the candidate can do the work themselves, can communicate about their work, isn't obviously a pain to work with.
Most take-homes test the wrong things: whether someone can start a greenfield project, think up interesting new features, or put in more extra hours than their competition. And most in-person interviews test the wrong things too: have you studied interview problems, are you good at coding and problem-solving out loud under time pressure, and how well do you scrawl syntactically correct code on a whiteboard.
First, I actually believe white boarding questions (at least the automated kind) are assigned more indiscriminately than take home assignments. Many companies just send you online timed hackerrank or leetcode questions as a preliminary filter.
Second, the entire interview process rewards people who study for the interviews rather than those who like to build projects. Unfortunately I believe companies miss out on a lot of qualified candidates, albeit reducing their false-positives from obscure algorithm-type problems. With so many applicants, reducing false-positives ensures quality hires so it works on the company’s end.
Overall, my opinion is the interview process is relatively broken. Take-home assignments offer more insight into actual development skill, while whiteboard questions filter out a large portion of people (many who would be just as good, if not better, at actual implementations.) But it works for the interviewers so change seems unlikely. This is just my experience for new graduate interviews; perhaps it is different at higher levels.
However, you will stand out if you ask “why did you choose those questions?” Or “how did you come up with those problems?”
Even the best interview that fits in all the check boxes trumps a similar candidate who shows they give a shit.
(Disclosure: We did this at a company I worked at).
There's still quite a lot of signal for an interviewer in someone being ready to do the extra work to achieve the goal (e.g. study for interview) vs. someone how doesn't want to invest time into studying and just wants to throw stuff together. This is especially important for junior developers (since as a company you'll want to train them up long-term to be promoted into higher level), but still remains important for senior developers (since being stuck with someone who refuses to learn new skills is a long-term problem for the company).
There's still quite a lot of signal for an interviewer in someone being ready to do the extra work to achieve the goal (e.g. write a real application) vs. someone how doesn't want to invest time into writing a real application and just wants to play with little algorithms and exercises.
Now if you smell a straw man or false dichotomy (hint: writing software instead of cramming leetcode doesn't mean you're not learning new skills; and vice versa) in this, you're on the right track.
The other thing is, what exactly is this signal? What does it tell the interviewer? They have their prejudice, so the signal tells them what they want to hear.
Maybe the actual signal is "I'm bored and desperate and don't have real skills and can't think of anything creative to do so I'll just cram leetcode until some unfortunate company thinks I'm leet and hires me. Because someone on the internet told me that's how you get a job."
This is why I'm not very keen to hand wave about "signal" if there's no way to give it an unambiguous interpretation. In these hiring discussions, I've seen so many examples of "signal" that can be interpreted any number of ways but someone thought they came up with a clever interviewing method so their interpretation of the signal must be right... right?
If it's a hazing ritual and lots of jumping through hoops, the only signal you have is that a person is willing to go through a hazing ritual and jump through hoops. Don't assume anything else. In particular, don't assume it says anything about the skills or capabilities of those who are not willing to go through it.
Sadly I've even seen this from a company founder who felt that as the company grew they were no longer the "big kahuna" and felt a need to make up for it by showing off in an interview.
(experiences as a co-interviewer not candidate)
The most fun organizations are where everyone is glad to be working with all those other people and things everyone else is smarter than them (so with a lot of opportunities to learn). Organizations like that really do exist -- I've worked in a bunch, but the larger the organization gets the harder that is to maintain.
But when a founder is doing this it poisons the well right away.
The goal of the interview is not to subject someone to a test, it's to become informed about them and their abilities to do the job.
The industry has this weird defensive position where everyone believes there's an army of imposters trying to invade them, and it's become a complete parody of itself.
The value of a sunk cost is zero. Who cares what you invested into the process so far? It's time to make the best judgment you can for the benefit of your organization. If you learn that the previous investment was misspent, great. Refine the process for next time.
You should also be providing a positive candidate experience if possible. They may be a customer, have friends you want to hire, or be in an anti-loop and actually good (but you can't tell, and that's okay, because, well, it happens).
You're there to (in order): 1) make the best judgment for the organization 2) represent the company well for the candidate (hopefully this is a faithful representation, since you can't work long term for the company if you're tricking the candidate)
It sounds like you do this already, since your interviews are challenging and friendly. Hopefully they're discriminating at and below job level to allow you to inform a hiring decision. Friendly almost always makes sense (excepting for threatening, dangerous, illegal, or otherwise red flag interviews).
Some houses cut bad interviews short at a halfway point. We don't. The candidate might show you something later in the day that points to another position or area of strength, and, either way, they should feel like they got a fair shake. It also helps with cross-calibration. I've had two candidates that I would have walked out at other companies (less than 1%) at the halfway point. 2-3 incremental person-hours is almost always worth it, though. I'm glad we put in the effort.
There's pros and cons. For on-site, there's a point where you just don't want to waste time.
When I run an interview, if it's clear early on that a candidate is a no, I'll just move the interview along quickly so I don't hurt feelings.
Generally the focus of someone hiring should be on the position they are hiring for, and they should actively ask themselves how the candidate with their own specific character traits and skills would cope in that position.
For this the person conducting the interview needs a profound understanding of the open position, which requires work that is often not done — and this results in lazy trick questions and ultimately gets the wrong people hired. Just like in a scientific measurement the person "measuring" the candidate needs to be aware of their own influence. And while taking of half of the stack of applications, throwing them into the bin and pronouncing that you don't like unlucky people certainly reduces your workload, it also reduces your chances of getting the best candidate.
So far so good. I sometimes attempt to requalify by adding a second step where I try an alternative mechanism of testing, just to see if someone who did poorly on the whiteboard (which is a coderpad-style thing) would do better in a different mechanism. So far very high correlation. Pity, I'm very social so I'd prefer a conversation and walking through code at a higher level.
I have always wondered why this isn't more common, it makes so much sense to me. When you submit an application you usually submit links to your blog, GitHub, projects etc. When I attend an interview I expect us to talk about those things. This is what the person is supposed to be good at. Instead you are met with a predefined set of puzzle questions.
I think the purpose of the interview is to allow the candidate to show what they are good at and what they are excited about and to see if this fits well with what your company is doing.
Now I have a wife and three kids and a house and I like woodworking and flying and fixing up old buildings. I sell all of my programming output to my employer, who doesn't happen to make open source. In my free time, I play with my kids or fix something on a house or build something in my shop or something that doesn't leave a mark on GitHub.
If you are evaluating me for a job and ask me to show you some code I've written recently, I can't do it. I do want you to test me, because I want to work in a shop where everybody can pull their weight - I've worked in shops that don't test candidates, and they end up with some really bad programmers who are sometimes decent sweet talkers. But give me something on a whiteboard, or a take home thing that I can finish in a couple hours after the kids go to bed, or just let me bang on a keyboard in your offices or whatever.
If you ask your candidates to show you some code they wrote, you're selecting for candidates who either do open source in their day job, or don't have hobbies that don't produce code. There's lots of great programmers that meet neither of those criteria.
I've had a couple of situations where I go in an interview and the interviews ask me questions that show to me that they have not even looked at my CV and the cover letter they asked me to write. They were just reading off a list of generic tech interview questions.
To me that feels like if I went to interview at a company and told them I have no idea who they are or what they do, I just know their name and that they are looking for a coder. That's just counterproductive.
But I totally agree with you that there is no single solution. I just think there are different "good" questions for different people, interviewers and interviewees need to adapt to that.
We also leverage take home assignments, which helps people get brushed up before the live challenge... It also tells us where people are in their skills -- code commented? It self-commenting? Any error handing? Etc. I've also pushed for this take home work to be relatively short, it's unfair to have candidates spend 4+ hours in a project with no promise if an interview. It always driver me nuts when businesses asked for that, so I try to avoid it myself.
All this said, it had dramatically United the quality of our candidates we actually see in an interview, which saves our developers time interviewing someone who can't figure out how to do JavaScript sorting using built in functions, or printing out only of numbers in an array of alphanumeric values.
> The exercise was supposed to take "a few hours" according to the hiring manager, but I'm extremely rusty with ASP.
Well, if they look for an ASP expert, this self-selection saves time on both sides!
I use take-home assignments with boilerplate code, to reduce the initial time investment and make it easier to compare parts of the projects. (Otherwise, it may be a too open-ended task.)
I try to make sure that the skills I require the skills I need. For example, when I needed a deep learning engineer (for a PyTorch project), I expect fluency in at least one deep learning framework. I made it explicit that instead PyTorch boilerplate then can use any other framework (a few people send in Keras).
Yes, people can Google things (good!). And it some sense it balances more junior people (but willing to put more effort) with people with more experience. The later could be told easily by interview. When I ask questions about their solutions, it is easy to see which parts are done with understanding, which - not.
The reason why the stack requirement bothered me was that, during the phone screening, I basically told the hiring manager that take-homes are fine, but I walk away from anything that has a laundry list of required frameworks.
They were looking for a very senior developer, not someone who would come in and know their stack on day 1.
In my case, the problem wasn't ASP, it was all the other frameworks and patterns they wanted.
- The first hour is spent having the candidates work through a problem in a sand boxed environment using the actual code base.
- The second hour is spent recapping the work session, discussing their professional background, and general questions about other things they should know to be successful on our team.
The coding test is designed to be solvable within the allotted time and without explicit knowledge of the business rules in the app and only requires an understanding of the actual technology they purport to have the required experience in. Running this test is a great way for us to understand what the candidate's skill level is and what problem solving approaches they use in a real-world scenario. They are free to talk about what they are doing (but is not required) and of course we offer guidance if they get stuck on something. Solving the test is NOT a pre-requisite for getting hired.
In order to reduce anxiety for the candidate, I like to frame the test by telling them to imagine themselves as a short-order contractor who is coming in to help us with a problem that we are having a hard time solving ourselves. I find that this really helps.
The feedback from the candidates for this testing style has always been positive, and I can tell you from post-hire experience, that it has helped us to bring some great developers on board. So far, there have been no false positives. In fact, it's so popular that other teams have either asked me to conduct their interviews or plan on adopting the same methodology for conducting theirs.
If your struggling to get more consistency from your hiring approach, it might be worth a try.
I think you could potentially learn a lot more from that than you could with your own problem. First, there would be a lot less stress on them since it would be a problem they're presumably very familiar with. If they do a terrible job coding it up, well, that's a super clear signal. Second, stuff like how well the do setting up the problem, what kind of problem they choose (is it some generic Leetcode problem, or something they came up with themselves), what they look for in a good solution (just that it works/that it's over-optimized/obeys some arbitrary style standard/etc.) would all provide a less clear signal, but still useful information, especially regarding seniority.
I strongly believe the candidate should be given every possible direction on what they are being judged on.
I'll ask, "What are some of the things you look for in a code review?" If they hesitate or seem unsure about how to answer, I'll help them out with "What are some of the things that will get a PR rejected?" From there I try to open it up to a conversation rather than getting them to list the "correct" five things to cover in a code review.
The point of an interview is to bring the best out in a Candidate, not to conform that the interviewer knows better, and definitely not to feed u to their ego.
The interview process is an assessment on your programming knowledge and its also an assessment on whether you'd be able to hang mentally (intellectually) around the people within the organization (read culture).
If the culture of the organization is made of people who are very proficient in computer science theory. It is imperative they'd would expect any new individual to at least be like them. That's human nature!
I'm also not criticising the process... The process is good! "The problem is bad interviews" is meant to direct criticism to bad interviews.
A religious war of whiteboard vs take-home vs whatever interview format is in fashion is silly.
I only have vague insights into these industries but science jobs afaik rely a lot on recommendations - on calling prior collegues and asking. And for writers - well reading their previous stuff works well (easier with open source stuff)
So i would suggest right approach is more a simple FizzBuzz screen, a cultural fit (careful of biases!) and then reading their past OSS / other work (so maybe a prepared portfolio is a good "take home") and calling their references.
A lot of people don't have any polished open source code. I have a lot of stuff on Github, but I wrote it for myself. I don't care if it brakes If I only using it once or twice a year.
I find that take-home exercises have two major flaws:
1. The push the effort onto interviewee. Interviewee should not be punished for the interviewer's inability to prepare good questions and exercises and inability to design the process so that bad candidates are filtered out early in the process.
2. They are not really meaningful. I've been there as a candidate. It is difficult to voluntarily stick to the timelines given when you know you could do this or do that to improve the solution and that other candidates will not hesitate to do it. As an interviewer, I have no idea if the candidate did the exercise by themselves or with outside help. Some (not all) candidates will do anything to game the system. You don't want to get these people onboard.
Based on that I am strongly against take home tasks.
On the other hand I like whiteboard exercises because they let me discuss things in an environment and closely to circumstances we could be having at work.
It is normal for me to draw algorithms or even implement pieces of (pseudo) code on the whiteboard and to use it for discussion.
What I try to keep in mind that different people deal differently with the whiteboard. Some people like to visualize, some think a little bit more abstract. I had to work hard to be able to draw anything in a way that will be digestible by other people and I still think am doing only mediocre job of it.
As an industry we should take our dignity back.
Beyond that, I think articles like this are important to read and circulate widely. Young people need to know that if they become doctors, lawyers, nurses, actuaries, licensed processional engineers, dentists, or financial planners, they will indeed need to take a test. However, the test will be managed by a central authority, with predictable content, with a clear study path, and once you take the entrance exam, you won't have to take it over and over during job interviews for the rest of your life. There may be refresher exams, but again, they will be managed by the licensing body and will provide a clear preparation path, and will be graded consistently. It won't all be left to the whims of whoever thought up a question.
If you become a dev, you will deal with interview exams, often conducted by people who are not really qualified to conduct exams, every time you interview, even if you have decades of experience. If you are a senior actuary, you will not need to integrate by parts every time you interview, but if you are a developer, you may very well have to find all matching subtrees in a binary tree every time you interview. You are signing up for a lifetime of retaking your intro to algorithms midterm.
You may be able to avoid this by starting a consulting company, or going into management, but at this point, if you plan to be a "knowledge worker", I would highly recommend taking a look at other fields.
Universities are not trade schools, they are often focused on education for the sake of pursuing knowledge, not how to train you for everything you need in your career.
Computer science is not software engineering; think of the difference between physics and mechanical engineering.
A lot of computer science classes are highly theoretical and have only tangential relationship to what it takes to write a commercially successful program.
BTW: This is the similar for doctors, in the US, your doctor has to get a license in addition to a degree, and the license requires self-study in topics that aren't covered in medical school.
Likewise, once you make someone put 4+ hours into tech work and it’s good work, you’d better hire them or apologize profusely for wasting their time.
For companies, you are probably looking at multiple developers and if you do only have one opening then even if more than one is good there is going to be a best.
There were other phases of the interview, coding wasn’t the whole thing, but I really enjoyed it. I bombed the object modeling on a whiteboard part of the interview, but the company made a great offer anyway based on being able to code. I’m modeling my hiring now on this type of interview because it was both fun and effective.
Flubbing the Twitter take home worked out for the best, Google was a much better fit for me anyways. Their interview process was great also, and doing whiteboarding over VC affords certain advantages (getting to type in code, if only into Google docs, is a huge win for me, Jamboard + an iPad Pro worked out great as well).
Besides, having been an interviewer more than an interviewee at this point I find being able to competently discuss previous projects with a degree of excitement more predictive of success anyway. I still ask to see interview coding, but it's more a check box for basic skills than a major focus
As for other matters in article, my current company just paying candidate for finished assignment, plain and simple.
When doing coding someone can have to solve an issue which doesn't have an answer on stackoverflow, so having problem solving skills is valuable.
Also, being good at algorithmics and data structures enable someone to come up with better solutions which might matter in a particular case.
I think algorithmic questions are better than asking questions about random methods from a particular framework.
Which is fine. If you're naturally talented at whiteboarding, great. If not, but you've prepared months ahead of time, great also.
If you're not naturally talented, and haven't prepared because of naivety or due to some principle, well, that's not exactly a great signal to an employer.
If software engineers actually took the time to look at how intelligence is actually measured in fields outside their own, and how it relates to job performance, we'd have a lot less broken interviews.
Take-home exams obviously also need competent people to analyze the results, and also have their pitfalls - especially if they're just being used as free labor.
Over the past 15 years, I have interviewed with many different companies. I once read that when you're a good fit for the company, the interview will feel suspiciously easy. This has been the case for all the times I've gotten offers.
Reading this article, a lot of the good points remind me of a particular interview I had though. It was at relatively large tier-2 tech firm a few years ago (not the FAANG tier); one where I didn't get an offer, but the interview process and experience was memorably good. I remember specifically telling the recruiter even after the rejection that I really enjoyed the interview there.
One of the rounds was with the principal engineer at the company, who was the most senior engineer at the company at the time if I recall correctly. It was the system architecture interview. I was asked to prepare by thinking of a system I had designed and architected in my past experience, and present it as if the principal engineer was my coworker and we would hold tech discussions. It's the closest thing I've ever gotten to the "Present code they wrote" idea that Andrew mentions here.
For the coding round of this interview, I was asked to bring my own laptop (another point that Andrew mentioned here; this was also the only company of all the interviews I've had in 15 years that asked me to do this.). I was then given a problem at the beginning of that interview, and given 45 minutes or so to write code on my own (the interviewer actually walked out of the room and left me alone while I did). It was a reasonable problem to code for the allotted time, and had reasonable complexity but not overly difficult. When time's up, the interview came back and we discussed my code.
Both of those rounds were something that was so well done that I think it was the most fair way a company could have assessed me correctly. I did well on the coding round, but my experience in system architecture was not a good match for the type of work they did, and in retrospect I could tell. Since I was interviewing at the Staff Engineer level, I think they correctly evaluated me and turned me down for the role. (Although I would point out that after that day of interview, they ended up being undecided, and asked me to come back one more time to have one more interview round, with one of the first engineers of the company there. I did poorly in that round for a variety of reasons, and that was ultimately what got me the rejection.)
Take homes are basically free labor.
The result is entry level inflationism.
The bottom line is that software engineering jobs are very different but we keep expecting to put on same cloths on them in terms of interview process.
I'm not inclined to agree. If I look at my job, well, it's rather "full-stack", spanning from hardware to bootloader to kernel to init system to application level. Debugging techniques range from probing with an oscilloscope, disassembly, dissecting core dumps, printf-tracing of asynchronous multithreaded systems, packet captures and traffic analysis at raw ethernet level..
I can't think of a single "CS whiteboarding" question that would do much to help me select a candidate for my position. As far as algo wizardry goes.. dude, naive O(n) and O(n*n) algorithms work for damn well near everything, and if you can figure out how to use one of the existing binary tree libraries or call bsearch(), you're pretty much set. I'm sure we could give you a few days to figure out some more advanced algorithm if profiling proves it to be necessary for some particular job.
Honestly I think the most important skill is the ability to learn new stuff and dig deep. I don't know how to test for that in an interview, I really have no clue.
Probably understanding concurrency and races is one of the more "critical skills" and I'd like to see that in a candidate, but I don't know if lacking a good grasp of that is grounds for rejecting a candidate. All too often I see lists prerequisite "critical skills" that any good programmer would be able to pick up on the job as needed. Do you really need to test people for something they can figure out in a day or three and get good at over the coming few weeks?
I've been in the interviewer role as well and I tend to not rely on coding exercises, whiteboard BS, etc. I usually just wing it knowing full well that interviews are in any case a bit of a crap shoot in terms of objective results. My goals when interviewing are very simple: 1) verify that what is in the person's CV holds up and try to get them to talk about what they did, how they did it, etc. 2) inject a few technical questions to see if they can come up with some good answers or whether they start bullshitting. 3) Figure out what they don't know and how they would deal with having to learn something new. 4) Figure out how well their experience aligns with my team's needs. I'm OK with a less than perfect match if a person has the potential to learn new stuff. Especially with junior roles, that's the most important thing because they probably don't know much yet.
But really, I tend to go with my first impressions here and have learned the hard way that ignoring those leads to bad hires. Yes, there's a risk I'm wrong about not hiring somebody. But if it feels wrong, it's not going to happen. Usually, I know within a few minutes what it's going to be; the rest of the interview is then either about making sure the candidate is sold on the job or giving me enough material to defend the choice not to hire to my managers, HR, etc. Coding exercises don't really factor into this. If I have any doubts whatsoever about a person's ability to actually produce code, I'm not going to hire them.
Unfortunately hiring is hard. There are a lot of mediocre candidates out there with a lot of mediocre recruiters representing them. I vastly prefer warm introductions through trusted friends and colleagues. So, as an interviewer, I deal with screening a lot of obviously not great candidates. The flip side is that when somebody good comes along, you need to be able to act fast because they'll have other options. Coding exercises are an obstacle for both sides.