The problems with live coding interviews
garrettdimon.com
garrettdimon.com
There's a fundamental skill that a good programmer has to have, and that is to be able to take a novel problem that they haven't seen before and break it down to solve it in a sensible way.
There are plenty of programmers who fake their way through a career without having that skill. They just copy stuff and never really understand it. They're fine for some types of programming career, but if your business involves solving new and novel problems then you have to know which type of programmer you're hiring.
A contrived live coding exercise gives you a strong signal on this. It does have a decent chance of producing a false negative, but only a very small chance of a false positive, and that's the trade that has to be made with this kind of approach.
Is a better option to not do this kind of assessment, hire the person, find out that they can't do the hard bits and then fire them within 6 months? I'll take some convincing of that...
Its more that businesses don’t know what type of programming they do and are just hazing people for no reason
More like one person is just hazing people due to a strongly held unobjective opinion like yours
Don’t know the better option, just the limitations of this one
But who does that professionally as a stand-up performance, clock ticking, a judge breathing down your neck who has been equipped with a script that tells him things like “if candidate doesn’t do X within the first ten minutes it’s very bad”?
Doing well in that situation depends more on social performance skills than problem-solving skills. You’re essentially trying to infer the interview script that the interviewer has in mind and act it out convincingly.
If someone says “I’d like to start by taking ten minutes alone in another room and think it out first”, will they get hired? Probably not. It’s not the expected performance even if the solution is fine.
As for nervousness for being forced to partaking in a stand-up performance, I'd argue that "social performance skills" can work against you, since the more "antisocial" you are, the more you can ignore (or are oblivious to) what other people think about you and you can focus on your task. It's only people who actually have the minimum requisite "social skills" that would be conscious of other people intensely watching them "perform".
But even if you're right -- interviewing is inherently a "social performance" activity. If you're not coding you're doing some other stand up performance to present yourself anyway.
I mean, this is very elementary.
So the issue isn’t that it’s a stand up performance it’s that you need to split your attention between the performance and solving some trivial problem. Becoming really good at programming requires being able to focus on an actually difficult problem not work through a simplified example while entertaining an audience.
There might be a correlation between solving trivial problems quickly and being able to figure out a heisenbug from some multithreaded monstrosity, but it’s not strong enough for minor differences in performance to be meaningful.
Ah, so you prefer being asked "where do you see yourself in 5 years?", and regurgitating the text generated by ChatGPT instead?
Alternatively, as I replied in the sibling thread, do you prefer to be handed a difficult take home task that is expected to take 2 weeks to complete?
I think I can live with the first, but I'd be insane to work 80 hours on a task just for a small chance to get a job.
There are multiple ways to "become really good at programming".
A company of any size needs people who can churn through well-defined problems quickly, and most live coding tests are relatively well-suited to selecting for that. They also need people who do other things well:
* tackle large, ill-defined problems, alone or in a small team * identify, triage, and (if necessary) solve problems as they emerge * refactor existing implementations that have outgrown their initial architecture * identify trends and estimate when they will lead to scaling issues in the future * communicate complex problems to people without the necessary context * break down complex solutions into discrete, well-defined steps that can be tracked to completion
They also need people who can accurately estimate how long all of the above will take... unfortunately, as we all know, such people do not exist.
If you're hiring someone to churn through well-defined problems, live coding interviews are likely a good fit out of the box. If you're hiring someone to reinforce a weakness in your organization's ability to do any of the other things above, live coding exercises are at best a loose framework that we're all familiar with to try to get at whether or not they have those skills.
If one doesn't even know how to complete well-defined tasks reasonably quickly, I can't imagine how they would be able to break down complex tasks to these smaller, well-defined tasks, and accurately estimate the expected effort needed.
This sounds almost Dilbert-esque. It seems like a lot of people here want to be a pointy-haired boss (or pointy-haired tech lead)!
Ie if someone asks you to find the median value of an array the number of elements and how to handle nulls isn’t obvious, but when you want the median value in an array the context is clear.
You can know how to do something without being able to consistently do it.
Maintaining velocity is something that I've never been very good at. I always struggled as a junior and mid-level developer with having periods of very high productivity followed by periods of very low. I've been fired for it.
On the other hand, if you measure my output on a monthly or quarter time horizon, I'm very consistently at the top of the class.
My strong suits are around helping others get through tough problems, quickly switching between levels of abstraction (including within a single conversation), understanding how a part of a process fits into the whole, and in guiding/limiting architectural discussions by managing the scope of the problem space under consideration.
This fits very well with my current "staff level" role, but was a real struggle getting to this point.
> This sounds almost Dilbert-esque. It seems like a lot of people here want to be a pointy-haired boss (or pointy-haired tech lead)!
I can see why you'd say that, but I think there's a big difference: PHB was incompetent in addition to having a different role from his team. I've found a sweet spot for myself where I can fill the parts of the "PHB role" well that actually do help the team, but I'm also more than capable of switching contexts and knocking out some tickets when there's a crunch. When I'm especially "in the flow" I work on that sort of stuff and do well. When I'm not, I run interference and keep my team from having to deal with the organizational stuff that doesn't directly help them get their jobs done.
The best functioning teams I've experienced all have one thing in common: they set people up for success by allowing them to do play to their strengths. As a team grows, you can find gaps in those strengths. I see hiring as an opportunity to find someone who is strong in the areas the team is weak. Let the new person come in and do the things they like to do and are good at!
The interviewer and the hiring company does not have a God's eye view of the candidate and must work within the limitations of the hiring process. The best one can do (without resorting to metaphysical processes) is to make a decision that has the highest expected value (or highest chance of hiring a good candidate).
The fact that there could be a candidate that lucked out because they were prepared for the specific set of questions you asked is not specific to live coding problems. You could have asked them anything and they could have lucked out and just happened to be prepared for the answer you were expecting.
That said, I can tell you for certain that if given a task that the average candidate takes 10 minutes, and a candidate completes it in half a day, you're looking at a 0.1x developer right there. That's by definition.
...if the job is to complete abstract problems in live coding interviews, sure.
Is that your point? Because I'm getting tired repeating the point about take home tasks in the sibling comments by now.
Or are you going to, as I mentioned in sibling comments, give them a take home task that takes two weeks to complete, just to be sure?
They are stronger in the specific problem space you’re testing for. If your business has a lot of novel coding problems that need to be solved in under 9 minutes, then sure, this is a great measurement.
On the other hand, if your business has a lot of hard problems that take days, weeks, or quarters to solve, measuring someone’s ability to solve a 9 minute problem is a terrible metric. If anything it’s a counter signal, since it tends to select for candidates that optimize short term thinking over long term planning and problem solving.
I suppose your ideal interview as an interviewer would be to give the candidate a take home task and ask them to spend 2 weeks to work on it?
> If anything it’s a counter signal
Or perhaps give the candidate a task that normally takes 30 minutes, and hire the candidate if they take 60 mins to finish it?
I mean, you do you, but I hope you (and everyone else) can see why I'm not convinced otherwise.
Probably most useful if it's a contribution to open source.
Metrics like a 10 minute task, or a 30 minute task is all relative. Do they know they language, the IDE, the operating system, the documentation, any experience in whatever abstract problem topic you choose, and personal comfort levels will all come into play.
If you want to filter someone to do a very specific thing then post to hire a contractor with the specifics of what you need. If you want a developer that can grow within your organization then pop quizzes area good way to dismiss good candidates.
This just doesn't work. Lots of people can bullshit very convincingly, and even if they can't produce novel solutions, can very well explain them in a matter that makes you think they for-sure know how to do it.
Hell, I can probably still prove lots of interesting things about set theory in math, as can many other people who studied math - doesn't make me the equivalent of the world-class mathematicians who actually came up with the proof.
Empirically, "talk to someone about what they did" doesn't give me hires that actually know what they're doing.
Law firms don’t ask lawyers to litigate in a mock courtroom setting before hiring them. Hospitals don’t ask surgeons to perform a little test surgery on a person before hiring them. Engineers aren’t asked to build a bridge to prove they know how.
I think the reason we are so unique is because we CAN demonstrate our ability to quickly and easily, and without risking human life. CAN of course does not equal SHOULD.
Same for medicine, engineering...
Any chump can call themselves a software engineer. That's the real problem here.
Any idiot can write things on a resume and say they did things they didn't do (aka lying). You would never do such a thing, of course, but as crazy as it sounds, there are people out there who would do just that! So because there's no professional license to write code, the only way to prove to these strangers that you actually can write code is some sort of exercise where you prove it to them.
I really don't get why this is so hard to understand, either. I get that live coding in front of someone else is a crazy stressful situation - I've failed multiple interviews because I couldn't perform on demand and answer the interview question in the interview setting, when I could easily have done so after taking a proverbial shower to have a think, so I'd love to get rid of them too. But unless we all band together and start a software developers guild or something, the live coding interview is here to stay. (Though, Triplebyte, now Karat, and others did take a run at improving the process, so there's that.)
I know what I know, but you don't know what I know. It's only by communicating, in a sufficiently unfakeable way, like a 45-minute in-person interview, that one can pass or fail the unwritten "can program" shibboleth.
Requiring salesmanship to enter a company may well correlate with requiring salesmanship to advance and be recognized at the company. (I don't regret that job, because the rank-and-file were great, but I don't want the same structure of overlords.)
This time around, I am fortunate to have effectively lifelong runway, so I'm now trying to handle an interview as if I were discussing a problem with a co-worker. If that filters me out, that's probably best for me, too.
When interviewing I always start with a discussion about the problem. On the whiteboard. I'm explicit with the candidate that this is totally expected.
I'm actually measuring something perhaps more important than the code that will get written later: can this person formulate a plan and have a technical discussion.
But with platforms like leetcode and more, the candidates are split into those who have had the time to prepare (and see the problem before) and candidates who can’t put in that time. And most modern interview processes do not give you credit for thinking through it. You only get through if you solve it the right way, and fast. There is no time to “think and solve a novel problem you’ve never seen before”.
Interview processes that do it that way are fundamentally broken. I wouldn't want to work at such a company. Interviews are two-way streets, and that would be a case of the company failing the interview.
The real benefit of this sort of interview practice is not learning whether or not the candidate can arrive at the correct answer. If we're at the point where this level of interview is even happening at all, you have (or should have) reasonable confidence that the candidate is capable of solving the problem. Whether or not that they do so in the interview is irrelevant. The real benefit is in being able to see how the candidate thinks through the problem.
While I think these whiteboard exercises are not the best approach, when I've been the interviewer at companies that required it, I chose a very difficult problem, gave a time limit, and told the candidate that I do not expect them to actually arrive at the solution within the given time. I just want to see how they approach the problem.
I’m talking about the “detect a cycle in a linked list” kind of question. If you ever actually need to do that in practice (though I would question the choices that lead up to that), it’s easy enough to google. The hard part in practice is figuring out that your messy practical problem decomposes into an algorithmic question, but that skill is rarely tested in interviews.
[Edit: I in mind ‘Detecting the cycles’ rather than ‘Detecting a cycle’ but wrote the wrong thing. Mea culpa.]
Your actual job will be more like implementing a maximally performant directed graph in safe Rust. (Not really. It won't be that interesting.)
Doesn't the standard cute trick answer for a linked list (tortoise and hare) work fine? Do two parallel breadth-first traversals of the graph starting at the root node, one at twice the speed of the other. (If there's multiple root nodes, introduce a unique super-root that points to the roots.) If there's at least one cyclic path, both iterations will enter the same cyclic path because iteration order is the same, at which point it reduces to the linked list problem.
With "parallel bfs" comparing "layers" of traversal has runtime cost that effects the big oh so I think the above solution is better overall even though it feels cheap.
I'm not sure what you mean by comparing "layers" of traversal. Tortoise and hare is about provide a termination condition in the case that a cycle exists. In the case that there's no cycle, every algorithm must inspect each node, so is O(n). In both the linked list and the DAG case, if there is at least one cycle, (a) both pointers will enter it in at most O(n) time, and they will match in at most O(n) additional time, so this stays big-oh optimal.
Yeah but the "parallel BFS" solution is actually slower. Overall people sacrifice memory for speed in these algorithm problems.
Additionally the input is already O(N) so if you count the input as memory the addition of O(2N) doesn't shift it. If you don't count it well you're deliberately removing real world actuality and making the problem harder.
>I'm not sure what you mean by comparing "layers" of traversal. Tortoise and hare is about provide a termination condition in the case that a cycle exists. In the case that there's no cycle, every algorithm must inspect each node, so is O(n). In both the linked list and the DAG case, if there is at least one cycle, (a) both pointers will enter it in at most O(n) time, and they will match in at most O(n) additional time, so this stays big-oh optimal.
It's layer based traversal via BFS. You have to compare the layers produced by the tortoise and the hare. Let's see for a binary tree:
For traversal:
hare: O(N)
tortoise: O(N)
There are N total nodes so traversal via bfs or dfs will always be N.
Because at each stage of traversal from the turtle and the hare we have to compare the "layer" of traversal between the hare and the tortoise to see if they overlap there is a roughly O(((N+1)/2 )^ 2) comparison as we cycle through each node in the layer to see if it exists in the other layer being compared. This reduces to O(N^2) processing time.
So total is O(N^3) because you have O(N) traversal time, and O(N^2) comparison time on each traversal step. You can reduce the layer comparison to O(N) if you save one layer to a hashset and then compare that set to the other layer, but that has a memory cost and only reduces the total big O to O(N^2).
The comparison happens at every level and NOT every node so, it's O(N) for traversal then when a layer is completed a comparison occurs at a cost of O(N^2). The amount of times this happens is equal to the height of the tree which can be simplified to logN. So total big O is O(N) + O(N^2 * log2(N)).
This simplifies to O(N^2 * log2(N)) which, while better than O(N^3), is still really bad.
Whether they arrive at an O(1) solution doesn't matter a bit to me. Most of the time, I don't even care if they phrase it in big-O notation - or have even heard the term!
Given the choice between someone who completed the exercise with an O(1) solution in half the allotted time and someone who never wrote a single line of valid code, I'll choose the person who is able to articulate the problem and potential solutions best every time.
It's far easier to search the web and find a "cute solution" than it is to learn how to think about and communicate complexity.
But, testing if the candidate has seen a problem before may be a proxy for something meaningful.
Honestly speaking, 70% of the candidates can do the job, but I fail 90% because they can't pass the coding question. It's a filter with a serious problem. Yeah it's all we got, but that filter rate justifies a big discussion because it's crazy.
The author suggests take-home coding exercises with "maybe ... a live call to discuss their solution" as giving a stronger signal than a live coding exercise.
The author also suggested to "present them with a complex technical problem that requires architecture-level decisions across various parts of a technology stack, and have them talk through how they would approach it" as a better signal than a live coding interviews.
So even if what you say is true, you aren't presenting a devil's advocate position. Instead, it makes it seem like you think the alternative to live coding is no assessment, which the author very clearly argues is not true.
The problem with this is that many, many interviewers do this badly. They think they are looking for someone who gets the right answer. Or they are not good at assessing how other people break down and analyze problems.
It is artificial. All interviews are artificial. This is why everyone would prefer to hire candidates they already know a lot about. But a good interviewer isn't looking for a perfect performance.
Programming puzzles are the equivalent of the GRE for programming jobs. It might tell you something about the candidate, but I'm skeptical that it measures the most important traits and skills that really good programmers have. It's an empirical question, but your skill at solving programming puzzles might not predict your ability to systematically break down large problems, write readable, robust, well-documented code, or design and debug complex systems.
I also have my own theory about this which might just be my own personal bias. I feel like standardized tests encourage conformity in thinking. By relying on standardized tests, you are selecting for individuals who are primarily motivated by external measures of success and approval. On the flip side, you are selecting against individuals who are intrinsically motivated to learn and build things, but who may not care about those external factors. My theory is that organizations with too much of the former and too little of the latter are going to have issues with groupthink and will be less likely to be innovative.
Totally agree
Since there is no CS equivalent of a professional engineering exam, employers have no guarantees that applicants will meet minimal qualifications, and so we continue to see an arms race of increasingly complicated whiteboard / leetcode interview processes.
Immediately after the time was up I magically started to realize how much I'd over complicated my one almost finished answer and quickly came up with much more efficient answer. Suddenly it started clicking how simple the first two problems were and how easy it would have been to crank them out if I wasn't panicking about the tech bubble collapse.
The recruiter later asked me how it went and I grumbled something about how it was a leetcode test. He said "oh well they need to make sure you have the skills for the job." At that point I was over the whole thing and honestly pretty fired up.
Over the next week I proved to myself that I do have the technical skills for that job, and that's honestly what counts.
¯ \ _ ( ツ ) _ / ¯
Fortunately, I interview well and can discuss many, many technical concepts deeply. But put me in front of a whiteboard and I couldn't tell you my mom's name.
- C/C++ on Win16/Win32
- Assembly language development with Z80/8051/ARM on embedded microcontrollers
- Java (core java, Servlets, J2EE)
- Ruby on Rails
- NodeJS / Javascript
- Worked with AWS tech (the full stack)
- Relational DB (MsSQL, PostGres, MySql), NoSQL db (MonoDB)
- Coded for Linux/Unix, MacOS, Windows 16/32, PalmOS, iOS
I can provide reference for each of those skillsets form my past colleagues.
You know what - I probably couldn't pass half of the insane coding puzzles these interviewers throw out. Not because I can't solve them, I just don't remember enough of the syntax or library semantics of the top of my head.
At my experience can we just assume that I am a competent coder (maybe not the top 1%, but at least in the top %20) and talk about the job and how I can contribute ? I mean its almost insulting if you ask me to make a linked list/reverse a binary tree or other such nonsense looking over my shoulder me with a time limit.
> At my experience can we just assume that I am a competent coder and talk about the job and how I can contribute ? I mean its almost insulting if you ask me to make a linked list/reverse a binary tree or other such nonsense looking over my shoulder me with a time limit.
I might inclined to agree had you not gotten all the names of the databases in your list wrong.
If there was a way to verify what you say reliably, then of course that would be better. But there isn't, and writing down that list is extremely easy - in fact, half of the CVs I've ever seen look very similar in terms of the length of the list, amount of technologies mentioned, etc. Even for people with far less experience.
There has to be some way to check whether someone actually knows what they're doing. For sure some of the time a strong reference is enough proof. That's why people in the industry a long time with many contacts will often go from job to job without even interviewing anywhere - they just move to places with former colleagues that already know them.
But for a new place that doesn't know you, thinking it's insulting to show what you know is... weird. Is it going to be insulting on day one when you actually have to do the work?
jeez, i would reject if they took more than 5 minutes
When I'm the applicant, I make it a point to take control of the narrative by saying something like:
> If it's alright with you, I'd like to approach this as an opportunity to expose how I approach problems in general rather than how I'd solve this specific problem. I'll speak stream-of-consciousness as I go through it so you can get an idea of how I'm thinking about it. Feel free to ask questions if you'd like; I'll rely on you to decide whether it's more important to you that I complete the task or explain my reasoning. I'm happy to switch to pseudo-code or just discuss potential approaches if we run short on time.
When I'm the interviewer, I open with pretty much the same thing. My goal is to put the applicant at ease (to the extent that I can) and make it clear what I'm trying to get out of the session:
> First, let me say that it doesn't matter to me if you complete the exercise or not. At this stage of the interview process I'm confident that you're more than capable of solving the problem, so lets use this as an opportunity to get to know each other and see if the way we think about logic is compatible. I'd love it if you could point out things you'd change, but don't worry about trying to 'finish' or end up with production-ready code. It's just a means to an end.
i had one who could not deal with that at all. he preferred to show me some of the code that he had worked on. fine, i let him do that instead. i passed on him not because he refused the coding session but because we didn't communicate well enough for me to be confident that i could work with him.
as a candidate i would skip the introduction though and just start talking like i would when pair programming, mainly because i'd be uncomfortable to ask for permission first.
It’d be more akin to pair-programming, which some of the interviews I’ve conducted have evolved into, depending on the strength of the candidates.
Would that capture enough to get a sense whether this person gets the gist of the code being written such that they could replicate the same output in a less contrived scenario?
That job had a lot of people working in pairs very frequently so it made tons of sense to do it that way there.
there is also the problem that instructions given by the candidate may not be clear and then i have to decide whether i just assume what they meant (because i already know how to solve the problem) or if i try to take them literally leading to frustration.
What else are we supposed to do? Take the fact that you can talk a good game as enough of a signal to invest 10s of 1000s? Assume that everyone with 20 years experience is as good as everyone else?
The problem is that there are no reliable signals. Most Developers I have interviewed have a massively inaccurate ability to judge their own ability (in both directions). I've lost count of the number of times candidates have rpomised that they can just learn whatever they don't already know and haven't been able to do it to any degree.
Qualifications are meaningful in some contexts more than others but most people in the UK don't have comp-sci qualifications.
So yes, I will use various coding exercises because depending on the level, it shouldn't phase someone to be given something quite simple and to see how they approach it (do they write tests first? Ask some good scope questions? Explain why they've done something the way they did?)
I have failed one of these tests in the past thinking I was a good Developer (I am!) but I don't blame the test or the process, I realised that my approach was haphazard and not an objective good look to an Interviewer so it was actually helpful.
Interviewer: "Gosh darn it, you're hired!!"
Whether this is actually the case comes down to the hiring manager.
If the hiring manager decides to optimize for performance in the coding interview then yes everything said here is true. They will typically hire people who can perform well and fast at simple coding test type problems above other any other desirable attributes.
If however, they simply evaluate the ability to work through (at any reasonable pace) a fairly trivial coding task, but make the hiring decision on bulk of the rest of the interview, then it shouldn't be a problem.
The problem is most hiring managers have not been selected for their ability, or even trained, in interviewing. A coding test is easy to set up (or copy from somewhere), administer and evaluate. It is often literally the least they could do.
What this post describes is simply hiring managers who lack interviewing skills. Personally, I would probably want to avoid reporting to such a person.
I agree, although this is why I feel a small project and participation in code review is a more realistic and useful gauge for real world ability.
Can the person understand the code base well enough to propose a sensible change, and how collaborative are they in the code review process? Do they document/comment their code, add tests, do they answer questions and take criticisms patiently, is the solution simple, does it work?
You'll learn about zero of these things watching them write a fizzbuzz in a shared editor window.
Yeaaaaah, that's not how any of this works. I've done 100+ interviews for MegaCorp. Who ultimately gets hired is ultimately a dice roll no matter how "data driven" we call it. Did you get a good loop? Was one of your interviewers in a bad mood? Did you end up with a hiring manager who "used to code" and now measures everyone against whether or not they use out dated OOP techniques everywhere? Ah crap, did you get Gary? That guy sucks so much. He asks "hard" questions to make himself feel good.
The sausage is what you'd expect if you remember one key thing: it's humans on the other side of the desk. They're finicky and arbitrary ceatures.
Well, I'm not sure I believe that's universally true (or even particularly common) but even if you're right, what difference does it make? You still have to be good enough not to bomb the interview but then just "hit the tables" often enough to get a lucky roll. That would be the case for any sort of interview process, whether it was live coding, an informal discussion, or tests of strength.
People who cannot perform socially or technically on the spot in a completely unnatural setup are sort of left in the dust. I'm not sure i have a solution other than throwing out technical interviews and actually trusting peoples prior work. I have work in the public (published academic papers, patents) but none of that seems to mean anything in an interview lol. Its only about can you perform right here right now for 1-4 hours.
I've said it on other threads. (1) trust peoples past employment, use background checks or something to make sure they actually worked where they said they have worked (2) look at their body of public work if they have any (3) just hire the people that look good off those two metrics and youll probably end up with a good employe 90% of the time and save countless hours and dollars. I'm almost convinced random choice + team vetting of resumes + a little background check would be just as effective as endless technical interviews.
I don't think that's a fair statement. One of the alternatives, take-home coding exercises (with possible live call afterward), is a kind of "coding during the interview process", yes?
The essay more specifically concerns a certain specific type of coding evaluation, with live evaluator who is unknown to you, in a setup which inhibits many normal problem-solving techniques (pacing, drawing notes, thinking silently for 10 minutes, etc.)
> It can literally be a simple task that one would expect any working software developer to be able to complete without too much fuss.
I think the two of you are in agreement even for that point. That is, the author seems to agree, writing "They likely do a great job filtering out people who are incapable of programming".
Unfortunately this varies dramatically from person to person during a high pressure interview.
I interviewed a very senior developer once that actually used to be my current manager's boss. The coding question was pretty simple and most people didn't struggle too much with it. He BOMBED it. Like, you could have taken someone with 30 minutes of coding experience and they would have gotten as far as he did.
It was also very obvious he was just incredibly nervous so I put him through to the next round anyway. We hired him and he was great.
Having done take home before from the hiring side, it was incredibly time-consuming for us. We had someone anonymize the three finalist submissions, and then we had three people each individually review and comment on each one, and then we got together to discuss and choose the final candidate. Once we agreed on one, only then was it de-anonymized.
All-in, it took way more time than a single developer doing three live coding interviews. But my guess would be that most companies wouldn’t be willing to be that deliberate with take home.
Grinding leetcode at least scales horizontally to a huge number of companies.
Homework is typically useless outside of the single company you're doing it for.
I generally refuse all takehome assignments unless:
1.) It sounds uniquely interesting and fun to do. 2.) The company is prestigious enough, or pays well enough, that making any effort to try to get the job worth it.
It really depends on how you create the test and how you review it. You can prepare one where the actual solution is 10 lines of code and anything else is exactly for showing what the candidate knows / could do.
I used something like "read data from CSV, write it to sqlite, treat it like a mature production app, feel free to use placeholders (usage doc goes here), go nuts". Then the test was really about knowing about error recovery, encoding issues, documentation, error reporting, monitoring, ci pipelines, etc. Many badly organised tests don't make the whole concept of a take home test bad.
No take home test is even needed. A simple conversation would let you know if they understand the importance of those things.
You can't really ask about those things directly. If I ask "(how) would you add monitoring to this?" you can make something up on the spot. If I ask "what else would a mature production app contain?" that will give me a better idea of what you're familiar with.
> you'd tell them they need to add it and they would
Different levels. We were not after juniors in that case, but people who can do independent work. The question would be way more specific otherwise.
> No need to obfuscate your intentions and have them read your mind in what you're looking for.
It's not about mind reading. For that role, you either know how to create/manage serious applications or not. Your specific ideas may be different than mine and that's fine, but the general areas of interest will match. E.g. whether you register some event notifications, do text logging, persist decision journal, or do something else — you'll likely mention that. And you may forget/miss one or two things - that happens. But you don't miss them all if you're a good candidate for that position.
Or they have enough surface knowledge to risk an answer that could potentially work. It depends if that's the threshold acceptable to you.
> You haven't explained why a take home test is needed for that.
We're commenting on the blog post about this very question. Making the initial stage of this test take-home is at least partially solving points 2,5,6,8 from the list.
(FTR, the answers were pretty consistent in what elements they included, so it wasn't confusing for the candidates)
That solves the skin in the game problem.
At least, that way you might get something out of it instead of nothing, although I definitely do agree that being paid for it is the only way to solve the skin in the game problem.
Maybe programming in another few years will just be glorified autocomplete and little more.
And perhaps testing people on how to write code was a mistake to begin with. It's one thing to write code, but reading code is another.
The real challenge is to your testing mechanism and not the candidate
It won't. For the same reason code generation from UML diagrams hasn't replaced programmers [1]. Or why business owner/analysts don't write acceptance tests in Gherkin (Given-When-Then).
Though some tasks that programmers had to do manually in the past may be automated (not the first time).
Of course, in order to be able to assess the answers to these questions in an effective way, you need to be able to very knowledgeable yourself, and that's the real root of the problem here.
What's important is how people approach computational problem solving, not if they can write a solution in 1 to 45 minutes. Really, who cares?
One of my go-to examples is trying to work from home. Which is great, yet has its challenges. We have three German Shepherds. They are lovely. However, when a delivery arrives or the gardeners are out nearby, well, it's mayhem for a few minutes. I've come to understand that I should just take a coffee break when that happens. I can't even do mid-level math, much less focus on a difficult CS problem during those moments. And it takes a good 30 minutes before my head can be back on task.
Stop treating software development like performing art or athletes having to perform in the heat of a game. That is not what we do. At all.
Write a function that takes in a string and returns 1 if the amount of letters in the string is odd. It returns 0 if the amount of letters in the string is even.
If you can do this, we are good. I dont need NP hard or whiteboarding or algorithms. I can tell how good you are just by talking to you about your background.
"I can't remember of n % 2 returns 0 or 1 on even, so in place of that I wrote (if ((n/2)* 2 == n) for even, assuming integral division"
Another problem is that we're prone to thinking that being able to do well on tests equates to doing well in life and work--despite a stunning lack of evidence in support it.
After many years of sitting on both sides of the table, I've have come down to this: "hire lightly and fire lightly". In practice, this means beyond the (very) basics, hiring is not algorithmic; it's a crapshoot.
Companies need to realize this and come to grips with it. Additionally, this holds as well for the following: beyond a certain skill threshold, a company will not really be any less competitive by failing to hire the best or even second best candidate. If you are a regional telecommunications firm, and hire the 3rd best candidate, the 1st and 2nd candidates will likely get hired in unrelated firms or unrelated regions (especially in the remote work era). And even if this isn’t so, because most everyone is mediocre, the competitiveness of your firm won’t drop appreciably.
It's a real skill you'll actually employ. You're coming at the code cold, which is actually a realistic scenario you'll encounter on the job. Your ability to catch bad ideas and prevent them from getting literally codified is a valuable skill. And all of that is worthless if you're in a state where you can see a mistake, but are too afraid to speak up; this gets tested too.
It might not be so great for newbies and people fresh out of college, but even they should be able to read the code and discuss it.
The candidate would come to the office (when we were still doing in-person interviews) and meet a couple of the developers for a little bit of chatting. They would then be given a couple of printed pages of Java code with a few basic classes representing banking accounts with some typical methods for depositing, withdrawing, persisting to a database etc. (with all of the details stubbed out) - and they would be given these instructions:
Read the provided Java source code files and identify any problems that you can find.
The problems can include things such as actual bugs, design problems or code quality issues.
You do not need to search for syntax errors - the code does compile.
You have 15 minutes to find as many problems as you can.
The rest of the interview would then be spent discussing their findings. Most candidates would find the obvious problems in the logic, missing null-checks etc., while trickier things like synchronization issues were missed by quite a few. Even though we had a list of all the bugs/issues that had been put into the code, the important part wasn't for a candidate to check off as many of these as possible - the important part was the discussion about the issues that followed.
After the candidate had told us what they found, we would start hinting about the remaining issues and eventually tell about all of them. How quickly someone would pick up on an issue when it was pointed out told us quite a lot. It was a way to get a feel for how the candidate thought and reasoned about code, without the pressure of them having to actually write code with someone looking over the shoulder.
Throwing it through find bugs is always a good option, and should find something
The juniors spot obvious errors. The seniors also spot logical flaws and the conversation quickly moves into "how could this be done better, overall".
as a devops engineer as my main job, I also try to explain to interviewers that on any given day I am reading and writing half a dozen languages of vastly different paradigms, and although I'm very proficient in many of them, I definitely need to reference things that maybe shouldn't "need" to be referenced (like confusing bash/python loop syntax is very common for me as an example). This rarely ever slows me down in reality, but will definitely cause me to fail interviews I shouldn't.
If I was an interviewer, I wouldn't care if a dev knew whatever esoteric language syntax or API calls by heart. I'd just expect them to know how to use them intelligently. The former does not always imply the latter.
An impressive number of people still fail the interview despite the questions being pretty simple (in leetcode terms, probably on the easier side of medium).
YES! Yes, exactly. A consistent and unbiased process that reliably weeds out people who is very bad at programming is incredibly useful. I'm not convinced live coding interviews are either of those things, but assuming they are, they are absolutely worth the listed downsides. Do they filter out lots of great programmers along with the bad applicants? Yes, totally, and that's a major waste. But all of the listed alternatives are less reliable, less consistent, slower, and friendlier to cheating.
I would love to eliminate live coding interviews where I work. I hate the things. But I have never encountered a mostly consistent and kinda objective solution that compares. I was hopeful that the essay was leading to a proposal for one and disappointed once again at its lack. Please, someone tell me what the giant tech companies should use instead, and I will gladly throw these "please reverse this linked list" interviews into the trash.
and very unnatural
and insulting to an experienced person (whos been promoted, and whos survived many layoffs in the past, etc) with many accomplishments. someone who has clearly solved and shipped, repeatedly. and one who has artifacts visible out in public. and has praise testimonials from former bosses, coworkers, clients, etc
This type of interview related blog post seems to hit the front page every week. Almost everyone agrees that interviewing doesn't feel good and seems overly complicated/difficult/whatever.
I hate that interviewing is a skill that has to be developed, in large part, separately from other engineering skills. I also don't really enjoy SQL/Database work. But I've gained enough competence in SQL that I'll be fine in most jobs. The same is true for my interviewing skills.
This format is popular because it's the best time/effort trade-off for both the company and the candidate. It's massively flawed, but everything else attempted so far turned out to be even worse.
Is programming weird because you can just ask someone to prove they know how to use a hammer? And so other industries just have to hire based on work history and/or bias "culture fit" during the interview? And they suffer terribly from people who can talk the talk but not walk the walk?
Or is programming weird because there's so much propensity for people to be able to talk but not walk? We rely on nerdspeak and jargon so much that just being able to prattle on in a dilbert-esque way would otherwise convince someone to hire you?
This is very simple problem to solve, and I think anyone who can't do that is not suitable for SDE3 (or 2 or 1, in my opinion)
One could argue about very difficult problems - can a candidate really be expected to invent the solution on the spot, or is it just because they saw it on LeetCode? Is it luck or ability?
But for simple problems, they are a strong negative signal and a very weak positive signal: just because you can merge sorted arrays, doesn't mean you're qualified, but if you can't even do that, you're most likely not qualified.
There's lots of other ways, but some of them suck.
Some fields attach a lot of weight to your alma mater, maybe they only hire people who studied law at Yale or Harvard.
Some fields require not just a degree, but also years of study under an industry veteran. Sometimes that also involves hazing like working 70-hour weeks, for some reason.
Some fields require work-sample tests where you show up at a given location and demonstrate your abilities on demand.
Some fields require not only a degree, but also years of working for free in order to break into paid work. And the paid work is far from guaranteed, that free work only pays off for 10% of people.
Some fields don't offer permanent employment, instead hiring people for much shorter periods - so bad hires can just not be rehired for the next project.
Given surgeons don't have to do live pancreatectomies as part of the interview process, I'm not convinced that's such a bad thing.
The candidate gets some code and a ticket description of what the code should do.
If he spots an issue, he can fix it right in the code or simply explain it.
I'd that testing the ability to read and understand code is both less stressful for the interviewee (no "judging every key stroke" stress) and more helpful for the interviewer.
Works great.
In my experience conducting these live coding interviews, it's almost a universal rule that if the candidate starts coding immediately, they will waste a lot of time on irrelevant details, or take a long time to see that their approach can't work.
I always try to encourage the candidate to talk through their solution or draw a diagram if it helps them. Candidates who follow this advice always perform better.
To me this is the key to the whole problem. For many companies live coding interviews are a cargo cult approach to interviewing. Since the people that do them don't understand how to evaluate others they become a checkbox exercise that only evaluates if the candidate would tackle the problem the same way the interviewer thinks they would tackle the problem.
I refuse to take those interviews. I had one company that promised they wouldn’t give me a live test, and when I told that to the interviewer who was trying to give me a test he said “hm, well we’re going to do it anyway”
I passed the test and was given an offer which I shot down for the company which respected my terms
In the end the other company was left high and dry when they needed the new team member
Just do it. Enough of us see this for what it is. If we just act how we think we change the industry
The other problem is after lay offs, if all companies ask you to dance then you have no choice. Dance or stay unemployed.
Obviously nobody wants to "dance" and nobody would if they had the choice.
Sometimes you’re desperate and will do things you otherwise wouldn’t. That isn’t and should be thought of as the norm
Here's the thing, when you look at the behavior of large groups of people the overall incentive is the overall outcome. As an individual you can resist but groups act predictably and follow straightforward patterns.
Think of it like drugs. You going to tell people to stop doing heroin and end the problem forever? No. You gonna tell people to stop dancing and end the problem forever? No.
The only thing that will bend the industry in favor of your idea is software unions. Centralized control can influence individuals to act in cohesion. Without leadership the short term incentive wins.
I fundamentally don’t agree. It’s simple: prioritize dignity in yourself and the world will give it
I don’t accept interviews where i’m required to dance. My career has only gotten better since I made that choice.
Even on the individual level you can make this right. It just takes effort.
Yes, I am saying people in aggregate are fundamentally weak. Certain individuals can overcome weaknesses but in aggregate we can't.
>I fundamentally don’t agree. It’s simple: prioritize dignity in yourself and the world will give it
Then you fundamentally don't agree with reality. Tell that dignity thing to a drug addict. Tell him to Prioritize his dignity and stop doing heroin. End the drug problem because people are strong. A few people will listen. In aggregate, nothing changes.
>I don’t accept interviews where i’m required to dance. My career has only gotten better since I made that choice.
On one hand you have control. On the other hand maybe you don't dance because you can't dance. There are people where "dancing" is equivalent to donating a dime to charity. A Trivial effort. To them the interview process is the equivalent of giving a dime in return for half a million dollars. Why the hell not?
>Even on the individual level you can make this right. It just takes effort.
The individual level is the only place where someone can refuse to "dance." For the industry to change it has to happen in aggregate. The aggregate change is impossible. There will always be people willing to dance, and there will always be people where "dancing" is a zero effort hand wave.
The path of least resistance long run is demand dignity. The path of more resistance long term but ease short term is accept undignifying circumstances
You can make your situation and the worlds better by doing the right thing. Or you can sit and argue to a stranger why it’s ‘impossible’ to tell a potential employer to pound sand when they ask you to do something demeaning
Choice is yours
Did you not read what I wrote? You or I both doing the right thing changes nothing.
The author concludes with a decent summary of the issue (ie. Is this the best method among the worst?). But doesn't actually find a better way of doing things.
And in the million forum threads on this issue, no one else has either.
But I don't know that many people. Properly conducted live coding is the least worst alternative out there for complete strangers.
Works great.
There is a nice side effect about doing this as a coding interview; there are often obvious indicators that a candidate is a poor fit -- for instance, big knowledge gaps in the standard library of the primary coding language.
More important is how the candidate structures how they would solve the problem, and how they communicate it to the person on the other side of the discussion. Are they taking testing into account? Do they iterate rapidly, or have a monolithic solution they have a hard time conveying?
For lack of an effective whiteboarding solution with remote interviews, coding interviews are here to stay. They should be reframed, though, as focusing on collaboration -- not on being a leetcode champion.
The way we do these at my current job is extremely productive for us. We look for two things, apart from basic competency: problem solving, and asking for help.
It's structured as a two 90 minute pair programming sessions. Interviewee shares their screen, and we work through the problems together. Obviously it's pretty hands-off, but we guide and nudge where appropriate. Here and there, when they use something that relates to a deeper topic, I'll ask questions to gauge how deep their knowledge goes. Like asking if they know how C#'s foreach works under the hood. Not as a selection criteria, but simply to get a sense of how much they know.
Use of a search engine is openly encouraged. A lot of the time, we don't even care if the program actually runs. If they struggle with syntax or the correct function overload, we'll help them out after giving them a little time to find the solution.
I also throw in a problem designed to get them stuck, and ask questions I expect they can't answer. A good programmer asks for help and admits when they don't know. A bad one bullshits their way through.
We want to hire programmers who can do a real job in the real world. Implementing red/black trees on a whiteboard blindfolded isn't a job skill, it's a party trick. That's not something a programmer will ever need to do.
In the real world, real people use google and stack overflow. They don't have encyclopedic knowledge of the entire language's syntax. They ask their coworkers for help or opinions.
Our interview process is designed to show us how a person will function in a scenario as close to the job as possible. Because that's what we're hiring them for. We look for their ability to work through a problem with the resources that everyone always has. We look for how they work with others and how much they lean on coworkers.
This has worked out extremely well for us. We've hired some very talented individuals, and have totally avoided the archetypal shitty dev. The people we hire immediately mesh with the team, and learn and grow the way all programmers do.
Granted, we are a small company and we have the time to have our own programmers giving interviews. We also have a much higher need to be so selective. But every single person who has made it to the technical interview has remarked unprompted that it's the best interview they've ever had. And I mean 100%.
It's because we treat candidates the way we'd treat our own employees. They get to know what the job is like, and we get to know how they'll do the job.
First, how we got here. I didn't experience this first hand, so I'm just making a recount based on what I've heard and read from "old timers". My understanding is that before whiteboard, leetcode-style interviews became the norm, tech interviews were mostly unstructured and quite informal. In the really old times (pre 90s) you could get a job just by knowing how to use a computer. I believe this wasn't that unusual in the 90s and early 2000s. I still remember hearing a founder-CEO bragging that his interview process was a 30 min chat with each candidate, and if he liked them, he would hire them.
From what I could gather Microsoft is the first big company that started using these "coding challenges". Then they became wildly popular thanks to Joel Spolsky [1], the publishing of Cracking the Coding Interview [2], and Google made brain teasers world famous.
Second, why we are still here. First, I believe there is a huge cargo-cult factor. Companies want to copy big tech, and alumni from these companies go on to found their own. This kind of interview has been honed and polished over the years, landing in a local optimum. An entire industry of websites and products has been created, and there are many entrenched interests. People might hate it, but the process works well enough for tech companies that they don't need to worry too much about it. Another under-appreciated factor is scale. This kind of interview sort-of scales well, which is important when you hire at a massive scale. That's why things like "have the candidate come to work one day and pay them" won't work for companies that need to screen thousands of people a week. Lastly, a standardized and "well known" process introduces some guardrails that can avoid some obvious pitfalls. The book "Working Backwards" explain how the Bar Raiser program was created at Amazon when a bad senior leader hire used the unstructured approach to hiring to build an empire misaligned with the company. At a big enough company this is bound to happen sooner or later.
Third, where do we go from here? I find it extremely unlike existing big companies will change their methodology any time soon. It might not seem like it, but for a company like Google it would be a massive undertaking to overhaul their hiring process. It would take years to achieve the level of efficiency and effectiveness of the current system, and surely there would be tons of opposition.
I believe the only way forward is for new companies to experiment with other methods of hiring, particularly at the beginning when they are nimble and can experiment freely. As they grow they will face challenges scaling, polishing and standardizing their process. At some point they will become the next generation of Big Tech, and the cargo cult wheel might spin again.
In any event it seems we need some sort of structured approach to hiring where we assess the match between company and candidate.
[1] https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guid...
About 1, big companies need regular standard hiring sessions also for anti discrimination, and simplifying comparisons of larger numbers of candidates. Bureaucracy can come about from involving big numbers of people.
1. Tech phone-screen: work through a simple but relevant to your job exercise (no leetcode stuff and code didn't need to run... i.e. could be pseudo code).
2. Take-home exercise: where I got the chance to write quality code at peace and a technical design doc describing my process and findings. I'm certain the take-home made up for 80% of the decision.
3. On-site: chatted with both engineers and non-engineers about the exercise, and how I would've approached new hypothetical but real-life requirements.
Everything past the take-home exercise mostly revolved around that exercise. If let's say I had cheated the whole exercise, I guarantee you it would've been evident immediately for many reasons – including the fact that when I was working on the exercise I discovered and reported a bug in their own production software.
Sure, you can often tell the folks who know nothing when asking them to explain the code, but it's increasingly difficult to tell a great engineer apart from a mediocre one that's cheating, and once you start hiring cheaters the toxic effect to culture sets in fast.
"So at that point, do they want to see you muddle through it, or would they rather see that you know to have ChatGPT run through the initial pass and then refactor?"
"If a company is evaluating engineers with questions that can be easily answered by AI in seconds, what are they really evaluating for? Perhaps they’d be better off hiring a chat bot."
I would hope that all SWE candidates would understand software licensing to some extent and how to behave in a way that would not put the hiring organization in a legally risky position.
ChatGPT isn't the only legally risky software licensing issue.
A lot of people, including me, will use a search engine to find a solution to a given problem. That solution may be covered by copyright, and subject to a license which is not compatible with the company's business model. Some companies prohibit the use of GPL software. I consulted for one company which required authorization before using any open source software.
So it seems to me you should already be asking people about these sorts of issues.
And 40 years ago in the debate over letting kids use a calculator in school instead of calculating by hand.
A footballer follows a very constrained set of rules. If the footballer were allowed to use a car then 1) it would be easy to score a point, 2) the field would be ruined, and 3) people wouldn't pay to watch or support the team.
If a programmer uses ChatGPT to get a handle on a task with a new API, and saves a day of futzing around, how is that NOT relevant to job performance? (For the sake of argument, let's say that experiment showed that API doesn't scale well enough, resulting in a decision to scrap that approach entirely and use a different API.)
If the evaluation doesn't make sense, then the conditions placed on the evaluation don't really matter.
Similarly, "assimilated the material" only makes sense if the live coding interview really does cover "the material". To use your analogy, measuring a football prospect’s cycling times aren't that good of a test of football playing skills. I mean, yes, there's some overlap, but there are more useful ways.
And one of the example tests was "handle scores for bowling", which is far from most work-related issues.
"consistent process that’s fair for everyone."
The author addressed this idea at several points, including "Any belief that a live coding interview is a consistently reliable way to make an objective assessment represents willful ignorance at best."
Picking a name out of a hat containing potential employee names is also consistent and fair.
Just because it's easy to measure doesn't mean it's an effective predictor.