In Defence of the Technical Interview

This is something I need to study each time I want to change jobs. I suspect that I will have to create a new project somewhere in the interview process but it is just something I very rarely do in my day to day job, if ever. I suspect that it is the same for many engineers.

FWIW, at my current company, the interview projects come preconfigured, so you only have to import them and start coding. We have also already imported some popular libs.

Watching somebody rightfully struggle to do that is :

- a pain for the interviewee

- awkward and uninteresting for the interviewer .. I don't expect many people to have to create new projects often so it is totally ok to have forgotten it

- a waste of time for the both of us.

We also actually don't do algorithms.

We have a debugging interview, with a shitty codebase where you have to find the issues and correct them. When you are done with the bugs, you can collect your thoughts on what's wrong with that codebase so we can discuss how to do it better.

Another interview is to have to design a product similar to ours (a simplified version of course) and you have to come up with API and app technical designs.

"Companies do code testing because they have encountered so many candidates who look good on paper, and may even be able to talk about computers convincingly, but can’t actually write a program when asked. Any program. At all."

I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test.

We get a lot of English-as-second-language and shy developers (who can code) and I can still figure out by asking pretty general questions about development. The idea that you need a code test to figure out who can code and who can't is false. I'm not arguing against it, I'm simply arguing that it's not strictly necessary.

They panicked.

I would like to see a large company trying a replication study on their interview practice with their current employees. You'd need to make sure that the test was significantly different from the one used to hire them, and that they were put under sufficient economic and social psychological pressure - perhaps tell them that if they fail they will be fired and immediately escorted from the building (+), then make them do it with the CEO sitting behind them. I would expect over a 10% failure rate, including some people who are normally regarded as business-critical important team players.

You then have to hope that people lean the lesson "interviews are a poor predictor of performance" and not "some of our people are really imposters"..

(+) obviously don't actually do this, but like fire drills you need to convince people that it might be a real fire sometimes. I am also aware that there are ethics problems with performing stress experiments on employees without their consent, although nobody seems to care about this for interviewees

We know that cognitive ability has one of the strongest correlations with job performance.

So "technical interviews" that are heavy on algorithm puzzles are just a loophole to get around the IQ test issue. They aren't actually testing domain knowledge.

[1] https://en.m.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.

Maybe 1 in 10 engineers who pass a code test end up being actually good software engineers. The yield definitely varies by the role.

Wait, if code tests aren’t perfectly predictive, then how do interviewers get feedback about failures (or even successes) of the screening process? They don’t, because recruiters and hiring managers tend to keep that to themselves.

The story is way way more complicated than this post alludes.

Once I applied for a job, and received an online technical test where every single question used the wrong terminology for a bunch of stuff and half the "answers" in the multiple choice section were wrong, while some of the "wrong answers" happened to be right.

That technical interview served it's purpose perfectly, assuming it's purpose was to save me from accidentally taking a job that would have been a fucking train wreck.

Minor nit about what is really a pretty good post. This paragraph sounds quite reasonable to this non-pilot, but do you notice what it doesn't say? It doesn't tell you whether pilots are actually interviewed this way.

Here's an actual example I know of, though it's not making the point the author wants to make. At an old job, every so often, a guy used to wander over from the other side of the yard and ask us to cut a few random pieces of pipe. We then carried them across the yard for a candidate pipe welder to weld them. If the welds looked good, they'd probably hire the candidate. It was pretty close to actual work you'd do, and the results seemed ok.

I'm sure there are examples that would suit his point and be real, not just speculation.

This article is a defense of an asymmetric process that stresses out software developers and forces them to jump through hoops. Slightly better than random selection is a garbage pay-off for the nonsense I've experienced in technical interviews.

Later, I sold $20,000 worth of source code that took me 60 hours to produce. The client thought it was easy to read.

I still don't think I'm a real programmer. The test sure seemed to prove that. But, people still buy my software today (I have other apps).

From my perspective, it seems the current technical interviews are looking for something in particular, from which you don't later actually do that for the real job. Kinda like seeing if you can survive a skydive, but you'll actually only be packing chutes after hire.

I wish there was a way to say, up front, that there are different types of programmers, vs the catch all "programmer" job title. Because it sure seems like I'm typing out code. The computer has always accepted it.

The alternative is widely varying interviews from one company to the next with no idea what to expect, which leads to a mismatch in expectations and mostly wasted time. Take home coding assignments is one alternative I see as well but as someone interviewing that is a much bigger time commitment and something I would not want to do especially if I’m interviewing at a few places.

Often when writing code I think, "hmm, this problem would make a good interview question" so I keep a note of it. Here's a recent example from an accounting program where I was reporting debit and credit transactions: Given a list of integers, remove all pairs where the magnitude is the same but opposite sign. Eg. given [-1, 4, 6, 1, -2, -4, 4], the result should be [6, -2, 4].

For more in-depth assessment, the author dismisses the idea of asking the candidate to do an all-day task or homework because it isn't respectful of the candidate's time. So why not just pay the candidate for that time?

I've started doing a small whiteboard coding exercise during interviews, and over the past couple of iterations it's devolved down to something simple...and I mean really simple. And I'm watching senior developers fail the test.

For example: I'm trying to fill a bootloader developer position so I get a couple of senior embedded developers to the table. Passed the screen, HR likes them, yadda yadda. Some even have U-boot experience. Should be easy, I think.

So I'll ask them to perform a simple bit-flipping exercise on the whiteboard. Nothing insanely tricky and it's usually something that any register-level developer should be able to blurt out without thinking twice about it. And they are choking on the question.

It's kind of freaking me out. Is the test that hard? Is the developer that inexperienced or distant from doing this kind of work even though they're applying for it?

What am I missing here?

Irrelevant if you've already fixed or implemented it and do not intend to use the candidate's code.

>there’s no good way to make this process repeatable,

Select fixed bug/feature, isolate code, create question, use in interview, iterate on task.

>no way to ensure the question always fits the available time,

Other than selecting the question such that it does and/or tweaking it such that it takes more/less time.

>and no way to ensure you see a good range of programming skills (e.g. the feature may simply involve copy/pasting some existing code with minor tweaks).

Other than selecting the question such that it isn't that.

>It also assumes you’re hiring Java developers for a Java codebase or Ruby developers for a Ruby codebase

I'm certainly not going to hire ruby developers for a java code base or java developers for a ruby code base, and I find it weird to assume that I would.

As an example, consider that we could be asking questions on API design: get candidates to work through their semantics/axioms (introductory forms to create objects, methods to transform and eliminate them). Get them to write out high-level testable properties such as the way objects compose, associate, commute etc. Using types to enforce invariants etc

> Firstly, how do we hire at R3? We’re recruiting for technical roles around the world as part of building an open source Bitcoin-inspired decentralised database. The interview process for developers consists of, firstly, a piece of code that you’re asked to send back a quick code review of (this is meant to take about 5 minutes), then an 30–60 minute long video chat+screenshare in which you may join from home and some coding is done in an editor and language of your choice*, then finally an invitation on site to meet the team and talk to senior management. The code test sometimes also includes design and ‘talky’ questions, depending on the nature of the candidate and precise job role: it’s not just coding. We’ve found this process to be pretty accurate whilst still being quite lightweight (compared to some hiring processes at least!).

That's commendable, but I think the argument is still faulty.

As I understand it, the premise of the argument for interviews and against work-sample testing is that candidates won't perform work-sample challenges. I've interviewed almost as many candidates as this person has (and screened out far more based on challenges) and that simply hasn't been my experience. More importantly, the argument is incomplete. Stipulate for a moment that the "best" developers won't take work-sample tests (I'll come back to that)... and?

The point of work-sample tests is twofold: to stop being beholden to subjective impressions of how well candidates can perform the work you need them to do, and to provide a framework for tech qualification that's straightforward to iterate on. If your work-sample tests are sound and you're getting enough candidates to saturate your headcount requirements after setting a bar you're comfortable with, why obsess over who exactly you're hiring? The industry is full of developers with glittering resumes and lists of past achievements who are dead weight on teams†. Why assume the developers with the best marketing are the ones who will perform best on your team? Why not instead just spend the time to figure out what exactly a good performer is, and then directly test (and iterate on testing) for that?

Regardless, the belief that strong candidates won't do work-sample tests is pervasive. And for good reason: work-sample testing at most companies seems cargo-culted: "take-home tests" (that often follow tech-out interviews) that can only rule candidates out and never rule them in. I agree: that's bad. But the solution is straightforward:

* (1) Allot a budget to the entire technical qualification process, including all interviews and challenges, (2) try to make as much of the process objectively scored based on a rubric your team has committed to writing, and (3) ideally move as much of the process "offline" (so the candidate can perform it at home and without you watching) as possible.

* Make the work-sample challenges as determinative as possible. Candidates who do well on our work-sample challenges are presumed technically qualified; 1:1 interviews that follow it are largely pro-forma. At Matasano / NCC Group (while I was there) the bar for a technical interviewer to overrule the output of the work sample challenges was quite high.

Regardless of whether you use work-sample challenges or scripted, structured interviews, a bit of low-hanging fruit that nobody seems to pluck is: relentlessly keep candidates informed about what to expect in your process. Since about 2010 we've been buying books (expensive books!) for candidates and providing study guides to go with them; we also provide fairly detailed descriptions of what our challenges will be. We're even moving to a system now where we offer candidates a practice version of some of our challenges (if I could fake up the entire AWS API, I'd do practice challenges for all our challenges).

But most employers seem determined to make their tech-out process as much of a gotcha game as possible, and tell candidates almost nothing about the actual hard problems they'll be expected to deal with.

This post uses a framing that says whatever tech companies do to qualify candidates must be rational; after all, if they're wrong, they're raising their own costs. I respectfully disagree: engineering teams do not, as a general rule, optimize their processes for efficiency or shareholder value. Many teams literally use interviews as a hazing process (one they often exempt "in-network" hires from). Still more delegate interviewing out to the entire engineering team and exhibit little or no interest in establishing rigor or rolling back bias. But engineers are as biased as the rest of humans, and most of hiring today is not in fact rational.

Also: sorting algorithms make dumb interview questions.

† This is in part due to a dynamics in our industry that make it easy to "gradually fail upwards": most companies can't fire fast enough after making a bad hiring decision to prevent the bad hire from accruing another positive line-item on their resume, so smart people can bounce from team to team for years without ever making a serious contribution and end out ahead.

As an aside, this lopsided penalty reminds me of how Typhoon forecasters in different cities systematically bias the predicted stormtrack toward their own city: https://twitter.com/m_clem/status/1081953447358984192?s=19

FAANG-level companies can put out a job role and have hundreds, if not thousands of people apply. Out of all of these people, there might be dozens of people with phenomenal CV's - relevant experience to the role, mastery of a programming language or two, public publications of their work, and provable experiences through living projects.

You reach a point where it's not practical to compare these people through programming ability alone, and that's where the algorithm interview comes in. It allows companies to choose a suitable person while alienating good developers, because through sheer interest alone they can afford to do so.

I just wish that more companies were honest about why this approach is taken, because it would stop startups asking you to find a join in two linked lists when the role itself is to maintain a shitty CRUD app. These companies cannot afford to alienate good developers because they're not getting the sheer amount of interest a Google or Facebook will get.

Again, just my opinion, but people need to embrace the fact that interviewing isn't perfect. A code test should purely exist to ensure that you can weed out people that cannot do ANY programming. Outside of that, use the interview itself to determine their fit for the role.

I have been in this industry for over 25 year from being a grunt, all the way to three letter jobs. The most uncomfortable question in decision to open a job is "What are we trying to accomplish by opening this job?"

In my experience if one is cut through the posturing the answer is "We need two developers with the following credentials because we believe that by having those developers we can increase the output -- more lines of code that we need to generate to make feature we promised on a road map ship. They need to know Angular/Javascript/Css/Node/Go/React/Kafka/K8s/RMQ and be ninjas". Should one drill down what that feature is one would find "It is a new dialog box and if a user is to click this button some things will happen and X email would be sent. Our users want this. We currently use backbone but we will be using Angular in future. API is written in node but we have a few projects to explore Go and we are using RMQ for queues though there's a team that is tinkering with Kafka" That's what the manager is actually hiring for.

So the real job requirement is:

"Need to get two butts in seats to build a dialog box to trigger an API action that would publish a message to a workflow queue. Must know a bit about backbone/css/javascript, nodejs and rabbitmq" [0]

Recognizing that one wants to put butts in seats rather than someone who can spearhead the API move from nodejs to Go would change the entire dynamics. Hardcore interviews for the 1st make no sense -- if a person can think, you can teach them how to write some javascript, mangle some backbone code, create a new route in an existing Express application and publish a message to a queue.

[0] Look at the hiring ads here. It looks like everyone is hiring Chief Architects that code.

Maybe I'm reading too much into this and it's just semantics quibbling but maybe talking about computers is not talking about coding ? I've never met someone able to pass for a programmer when they are not. I've always felt just asking candidates to talk about their last or favorite project help catching a glimpse of how they think about software problems.

Little anectode : my brother is the first software engineer of the family

Father : "Son, what kind of computer should I get ?"

Son : "I don't know much about choosing computers"

Father : "Don't you write software for a living ?"

Son : "Yes, exactly"

My first reaction to this was: "Damn... where's my ASCII table?"

Because this question can be taken two ways: "What comes after f in hexadecimal notation?" and "What is the UTF-8 code for 'G'?"

A very important thing to keep in mind for both sides in an interview is that misunderstandings will happen, for the most mundane and obvious things. How you respond to these will determine your success in interviewing. Give the benefit of the doubt until the other party has conclusively proven that they don't know what they're talking about.

In fact, in many FAANGs, your first performance review year is "freebie" because they know that you're not going to come in and know the role and the tools used intnerally.

So, then, what is the technical aptitude test (is it really anything other than that?) for? Well, it could be used to measure a baseline for basic programming skills; however, this isn't how most technical interviews are designed.

Interviews aren't designed around the premise that knowledge on the internet exists or that your colleagues could, oft, be your first line of support (or code reviews, if the company has a good culture). Hell, there are interviews where the candidate has to code in notepad (or other text editor equivalent) because they don't want the candidate availing of things like Intellisense.

If we look at it from a different perspective, though, we live in a time when we can now put our source code online for the world to see. We can publish packages for the world to consume. We can make changes to production software (FOSS) and documentation and that's easily traceable (if you're not obfuscating your identity through sixty different handles).

It seems we don't consider these viable avenues of purview to view the candidate's ability before even asking them questions.

With that being said, are you really hiring the candidate because they would be good for the team or are you just hiring the candidate who can check off some interview-type question boxes?

I have interviewed a number of people in my day and the most excelled candidate that we interviewed and hired, because he checked all of the technical boxes, ended-up being a raging asshole and bringing the team morale down, considerably.

So, at the end of the day, you're not just hiring someone to fill a role. A single role is all but mostly dead, anymore. You're hiring someone to join a team and the human aspect should outweigh the technical aspect.

After all, just because someone checks all of the boxes, it doesn't imply that they would "be a good fit".

Or, you know, just use a whiteboard in the interview for the same purpose you use it in meetings:

- not at all

- for high level algorithmic descriptions, data flow or architecture sketching

(Of course, the UML being demonized is a bit of a misfortune because now boxes and arrows can mean anything, but hey, better than having a visual language that has been developed by a bunch of old CIS White males, according to tech diversity Twitter.)

This bothers me immensely. If you are job hunting, you want to apply to several jobs to get a number of offers and get an overall better deal. How do people manage this when they got a full time job? Do they literally take an entire week vacation to have 5 all day onsites?

There are points in this blog post I strongly agree with like how disrespectful "homework" and the importance of letting a candidate code on their own machine, where they have all their shortcuts etc. setup and are able to be productive. But I've found the algorithmic coding challenge to be in equal parts harmful as useful.

The biggest problem, as I see it, is as software engineers we are strongly biased towards a rational understanding of the world. For example I've seen hiring teams argue _for_ a candidate they clearly don't like because that person aced the coding challenge. In such situations, as hiring manager, I then challenge the hiring team with "OK it's Monday morning. You had a bad weekend. And now you're in the office sat next to the candidate. How do you feel about that?" and usually the truth _then_ comes out and we avoid hiring someone that would destroy the team. The point here is we have a natural bias to see things in absolutes; zeros or ones. This _tends_ to lead to an over-focus on the coding challenge because it gives us the illusion of a black or white answer on whether we should hire.

When I've interview people, I always ask myself "what's my gut feeling about this person?". What I've found there is "my gut" is sometimes wrong in the positive direction - I liked someone but they turned into a bad hire but always right in the negative direction; if I have a bad gut feeling about someone that I can't fully explain, these days I always go with it. The "evidence" I have there was two hires that I had a bad feeling about, one I later had to fire myself and the other that stuck around for a couple of years until someone else fired them.

Otherwise, when it comes to assessing coding ability, these days most programmers have code on github, even if it's just "hobby" code. I'd rather review that, look at their commit messages etc. and get a feel for how they program when relaxed and not trying to impress.

And if we must make candidates do homework, how about contributing to the greater good and asking them to make a contribution to some Open Source project e.g. fix a bug in numpy or otherwise?

Which is probably correlated with solving problems in algorithm puzzle (aka "programming") contests.

If solving algorithm puzzles or competing in "programming" contests is your company's core business, then probably it's a good interview test.

Here are the problems that I have with coding interviews:

1. Often I feel that interviewers are looking for exact, optimal solutions rather than caring about how the interviewer actually approaches problem solving. Forget about being asked FizzBuzz-style questions or about being asked to delete a node properly in a binary search tree; in the interviews I've had in the past three years, I've encountered difficult Leetcode- or ACM International Collegiate Programming Contest-style problems where it's expected that I come up with an optimal solution within 30-45 minutes. It's even worse with companies that give you a hard-level Leetcode-style problem that is automatically graded instead of being examined by a human.

2. The lack of receiving feedback about interviews that didn't go well makes the process difficult. At least on a standardized test you receive a score, and at least at the end of a final exam in class, you get your final class grade. Case in point: I've had two successful Google software engineering internships with great reviews from my mentors, and so it's not like I'm incapable of programming FizzBuzz or writing a for loop, but after three tries I haven't been able to get an offer for a full-time position there, despite making it to the on-site round each and every time. It's similar with other companies: I'm usually able to make it past the phone screen and about 50% of the time past the initial programming question, but I have a hard time making it past the on-site round for software engineering positions.

3. The sheer breadth of possible questions to be asked. Not only are Leetcode-style puzzles "fair game," but also domain-specific questions. For example, suppose I'm interviewing for a position where I'm working on a DBMS. Despite having taken graduate-level courses on the implementation of databases and distributed systems, there is still a very large amount of questions that I could be answered, including specific details of specific databases that I might not have been exposed to.

The frustration I have with the software engineering interview process is enough for me to want to change fields at times. Thankfully I've found the interview process for more research-style groups to be less about coding acrobatics and more about conveying previous research experiences and successes as well as proving competency and curiosity. I have a pleasant role as a research engineer where I'm doing very interesting research work while also maintaining my coding skills. Unfortunately such jobs are hard to find in industry, which means that one day I'm going to have to take up the gauntlet of software engineering coding interviews all over again.

Then talk about how you "dodged a bad one" because they couldn't say a single sentence without an "uhm".

Said nobody with 10+ years of experience. Starting a new project in an editor is as obscure as changing the color scheme in your editor.

My question is, do companies give interviewees the option to work on a problem the company is currently facing?

If not.... why not?

Telling them they should change is like telling Tom Brady he should change the way he holds the football — even if you’re right, he’s not going to listen to you while he’s still winning the Super Bowl every other year.