The business of takehome assessments

In my last job search (~3 years ago) I was presented with many requests to complete a design challenge. I rejected them outright and the responses I got (typically from a 3rd party "design recruiter") were quite astounding. Some acknowledged and moved on, but there were several who clearly expressed frustration, disdain, sometimes almost anger. One dropped me from another role I was working with them on for refusing.

Now it's a total red flag for me. But judging from Blind posts it's still a common practice.

Counterpoint: those fresh out of school are likely to do the best in some "invert a binary tree" leetcode style interview, with that fresh in their memory. Those who are older (and thus more likely to have families) who actually have experience at the craft are most likely to be able to bang out an actual coding task quickly.

It's been a long time since I've been on the job market - but even a decade ago, I had third-party recruiters who would forward me a take home test... and another successful candidate's solution, "for reference".

And even if candidate doesn't get external help, I'm extremely sceptical about time limits. Nothing stops someone spending 6 hours on a "2 hour" take home test, producing a better solution than someone who followed the time limit strictly.

It's pretty hard to extract a signal under such circumstances. Unless you're hiring for a job where a willingness to bend the rules to get results is desirable - used car sales, for example.

Realistically, people are going to spend more time on their take-home assignments, because they perceive there's an advantage to doing so. It's not hard to imagine how this spirals out of control over time, a feedback loop of candidates trying to do more and more free, meaningless work as the bar is raised. A three-hour assignment that takes an entire weekend, a three-hour assignment that takes a week, etc. But they shouldn't be openly encouraging this death spiral, they should be trying to keep it under control.

Better to say "if it looks like you spent 20 hours on a 3 hour assignment, we'll think of you as someone who isn't good at estimating, and wastes a lot of time fiddling with things that are out of scope".

I noticed the same thing. It seems take homes are mainly geared towards pre-selecting young single people with a lot of free time to code on the side besides their main job and other responsibilities, or who have no other responsibilities.

Now I'm also single and I have the time to invest in them, but I question the future of my career in tech, how will I be able to compete for new jobs later when I won't be, and take homes will be normalized? I don't remember my friends in other careers ever having to do unpaid work before getting a job. Perhaps I chose the wrong career.

What sucks even more is when you sink many hours in them and then just get ghosted, not even a rejection message, nada. I've already been burned twice by this. Or when the recruiter just spams you the take home link without first having a talk with you about the job details, the compensation range, if you're a good fit, etc. It's "first you solve this take home for us, then we can have a talk". Feels incredibly cold and inhumane.

Sadly how people recruit is completely broken for most companies that are not FAANG etc. (It is broken for FAANG as well but in a different way).

As I often say "you need to win, not win an argument". Similarly you want candidates that can be good productive members of your team at the salary you are paying and not the candidates who can nail your interview process.

The problem with take home tests is not the cheating. But rather failure of the recruiter to actually evaluate the assignment. Do you care about the end result or do you care about the art of craft ? Do you care about readability of the code or performance of the app ? Is the assignment similar to the work candidate will end up doing at workplace ?

My suggestion generally has been:

1. Look at candidate's past work. Has candidate worked with reputed companies ? 2. Talk with candidate with general technology topics around their work. Something like "what do you like about react?" 3. Give candidate a very simple codesignal test. Nothing to fancy. Say if you want react engineers just test their ability to implement simple components.

This vastly kills a lot of inferior talent in my opinion.

4. Give a very simple problem without any "trick" or "iq test" and ask the candidate to code it in front of you. Also let the candidate use internet, documentation and AI tools.

In this day and age asking candidates NOt to use AI tools like asking a candidate to write code without using keyboard. You want people who can use things like Github copilot.

I can spend time answering your take-home, or trying to get other interviews. From what I've seen on take homes: Other interviews are strongly EV+.

Before you go: But you never got a job off a take home, that's why. I did. It was one of the worst managed misfits I've had. I am sure others thrived there, for me it was a living hell, that didn't even pay that great.

So... Much like requiring a suit to interview and a few other things. It goes in my bin of "Your company has performed self selection."

Note: I will leetcode, at least there is a human there, we're talking, interacting, and I am learning about the company.. But most of all, that feel of: I put up and hour, you put up an hour. Let's talk. Is very real.

Which... doesn't necessarily correlate to someone being good at programming or web development or what not. Or whether they'll be good at the job in general, given that most engineering roles don't have someone screaming "get this done in under an hour or else!" with no time to think things through properly.

At the same time though, I can also see why they do that, since otherwise it comes down to "whoever's got the most free time and willingness to grind away at the problem", turning an hour long exercise into a 3 day one in the process.

We're biased, but we think the old form of take-home assessments (+ classic Leetcode tests, etc) are completely broken. Beyond reasons you all mention - they're completely unreliable today in the age of ChatGPT, etc. Way too easy to cheat.

We're seeing candidates copy takehome instructions into an LLM, paste the solution and submit in <5 minutes. It's hard to write a problem that's (1) low enough scope to solve in a short time, but (2) hard enough that LLMs can't step towards sovling it.

At Ropes - we're using LLMs to evaluate how candidates code, not just the final solution. So HM's can step in and look at a timeline of actions taken to reach the solution, instead of just the final answer. How do candidates debug? What edge cases do they consider? Etc. We think these answers hold real signal and can be answered for the first time async.

We're trying to make this better for candidates too. E.g. (1) shorter assessments, (2) you can often use your own IDE, (3) you're not purely evaluated on test cases, etc. But we're not yet perfect. If this sounds interesting / you have strong thoughts I'd love to talk to you - email is in my bio.

> Rather, they were the design studios, the lean web development agencies, the mobile dev teams based out of Eastern Europe.

Given a team of... let's make up some numbers.... {8} devs working on client projects that have {250} candidates to evaluate, how much time do the {senior} members of the team have to evaluate them? Note that taking {two} senior devs in such a shop out for {30 minutes} for each candidate for a live coding interview is {=about a week and a half} of nothing but doing interviews for each of them - removing them from other revenue generating tasks and a reduction of {at least 1/4th} of the work for that duration.

Extending this beyond 2 weeks of time means that the best candidates are likely to have found other opportunities.

Take home tasks shifts that time to a standardized "I can look at this sample code in 2-3 minutes and get a feel of if the candidate will be producing something maintainable or hot garbage."

At that point, one can narrow down the pool of candidates down to a small number that can be interviewed and decided with some expedience.

- Whiteboard coding tasks put a LOT of pressure on you to perform while being watched in a stressful high-stakes situation, which is a situation that the extreme majority of SWEs won't deal with, and many simply can't. They also tend to test for algorithms (inverting a binary tree, etc.), something that is unrelated to most actual jobs.

- Takehome assessments are unfair to those with a family that might not have the free time to do them, and if you're applying for multiple jobs, can easily become a full-time job on their own.

- Performing a 5+ round interview gauntlet is frustrating and absurd, and can make the hiring process take well over a month.

> If the very first thing in your application process is a 6 hour long unpaid assignment, then it’s hard to imagine you’re not wasting someone’s time because at this stage you’ve barely done any filtering.

I pretty much walk away at this point. It's like asking a person for sex after buying them a drink. (And if I have to explain why, you clearly don't get it.)

> But if the takehome is offered, say, after 3 rounds of interviews and will be discussed in detail during your final round with a panel of two senior engineers, I think that’s fair and provides a strong signal.

Exactly. That's the point that I'm willing to invest in a takehome.

And how are these big tech firms spending $billions on recruiting and interview time, as if the education credential is worthless, but still preferring to use it as a first-line filter?

But I don't think this type of evaluation is a valuable signal anymore in the age of AI. Anyone could nowadays finish your take home assignment in a fraction of the effort it would take them without AI. This might not be relevant after they're hired, since they would likely also use AI assistants, but for evaluation purposes you want to assess how someone thinks about problem solving, not how they generate the solution, and then confidently walk you through it.

The best interview style for software developers IMO is the code review challenge. Show them a piece of code from a hypothetical PR of a junior developer, and ask them to a) describe what the code does, b) point out any issues they see, and c) fix the issues. You can have several difficulty levels depending on the role and expected candidate experience, or you can cherry pick these from your own codebase. The benefits of this approach are that it's a simulation of a real-world task they would be doing on a daily basis, it's a collaborative effort and showcases their communication skills, while also allowing them to evaluate yours, it's much less stressful than white boarding and quizzing sessions though still involves some coding, and unlike take home assignments, it can be completed in an hour or 90 minutes at most.

the project was to make a django app for storing your pokemon application, and a script to download the list of pokemon from an api. So atleast it was fun, but i hated turning people down who took the time to do it.

Also, enough with the fucking Roman numerals, Jesus Christ.

I like respecting candidates' time, because I want the strongest candidates to not filter themselves out at the mention of a take-home. That's why I ask them to schedule 4 hours with me where at the start they will get the problem description and at the end they will submit what they have - this way they aren't pressured to put in more time because they know for sure that their competitors didn't - and I get to compare apples to apples.

I also designed the rules to minimize the perverse incentives set by the clock by giving multiple ways to get recognition for partial solutions.

This is the content of the document the candidate gets initially (they only get the actual problem later):

Meta

This is a real task we encountered and solved at Finubit. Out of respect for your time, we limit this task to 4 hours (of wallclock time, to remove the incentive to spend more than that to gain a competitive advantage). It took us much more than four hours to get the perfect solution, so don't worry, we don’t expect a full production-ready solution to 100% of the problem.

We ask that you approach this task as you would the first 4 hours out of as much time as you need to finish the task to a reasonable quality standard. We do want to see a working Proof Of Concept at the end of those 4 hours, like we would ask of any team member picking up a long, open-ended research task. The POC can be very limited in functionality, but functionality that it promises will be held to a standard of quality.

If you encounter ambiguities or mistakes, assume the most reasonable way to resolve them and make a note to check with Product that your understanding was correct. If you disagree with Product about something, make a note to push back against the design decision they made (but in the meantime assume the current spec, unless it's grossly wrong). If there's a subcomponent in your design that you don't know how to implement but believe to be solvable, it's OK to assume the existence of a black box that solves it.

Please take a moment to set an alarm clock. When the clock strikes 4 hours, please send us your code and research notes, and then go over the code and document everything you’d change before you consider the code production-ready and send us the annotated code.

We will be available to answer clarification questions if needed.

Then there is a checklist to reduce loss on technicalities (forgetting to set an alarm or send annotations).

(If you're in Israel, and you want to try it for fun, shoot me an email)