In today's market you will likely see a large percentage of candidates opt out of the interview process when you send them an automated technical interview.
Moreover, if this is necessary, I kinda assume the pay is mediocre at best. If you knew how to get enough value out of me to be able to pay me really, really well, and if I look like a good fit for the job, then again, my resume ought to justify our talking. If any of that's not the case, then it seems like the pay's gonna suck or I'm just not a good fit, so I'm out either way.
I want someone who's fishing for me with a lure, not by dragging a huge net through the water. Seems like a better sign all around.
But, from an employer's perspective, resumes tell you very little about a candidate and are essentially worthless. The sole usable metric we get from them is years of relevant experience.
We respond with a questionnaire that gets us more uniform answers about experience and technologies used.
Assuming the above doesn't filter the candidate out, we move on to a 90 min max "take home" exercise. In our case, it's not a coding exercise, but that's not that important for this discussion.
Only if all of the above hasn't filtered out the candidate, do we move to a Zoom meeting. In the Zoom meeting, after an initial introduction, we do a few easy coding exercises. Like, reverse a string or sum the integers in a file easy.
A few notable things from this experience:
1. The number of candidates who struggle with the Zoom interview coding exercise is surprising. It's a lot higher than expected and almost all of our candidates at that stage have 2+ years of professional development experience. 2. The number of candidates filtered out by the "take home" exercise is very high. 85+%
We literally don't have enough time to talk to every candidate that applies with the right amount of experience. And, even if we did, many/most candidates on the market aren't good developers. Our signal to noise ratio is already bad. In that case, it would be terrible.
As a developer, I sympathize with your perspective. But you may be passing on some good opportunities by not being willing to "play the game" a bit. Some employers are indeed bad actors and/or terrible at hiring. But given the current market, even those making a valiant effort to do it well, due to sheer volume, are usually going to have an exercise of some kind.
We post our salary ranges up-front, don't want to waste anyone's time that way. We also put a ton of effort into a very detailed job description and application process. The hope is that the candidate can see the effort we put in and will be willing to reciprocate. If that doesn't end up being the case, we assume, like you, it probably wouldn't have been a good for anyway.
FWIW.
Your funnel is steep because it has bullshit steps. And the only people coming through it are probably desperate, which is why your signal/noise ratio is "already terrible".
What kind of experience do you have hiring? If you have a better process, that has been proven over time to work, please share.
I'm part of a small group of leaders/owners from other development shops. We meet monthly to collaborate and learn from each other. Hiring is a frequent topic and I assure you that our steep funnel and perspective on candidate ability is not a unique experience, despite divergent hiring processes.
It is true that our process could be filtering out candidates who might otherwise apply. In that case, it's most likely working as intended.
You are very casual in claiming that there is such an intentional correlation between developers who are not willing to spend 90+ minutes on a non-technical prescreen with a 15% success rate and competent developers who can translate business needs into actionable steps. It's reasonable to say you are willing to accept that there are developers who are unwilling to go through this process, but to say you are filtering them out intentionally is needlessly contemptuous.
But, seriously, it's intentional in the sense that if someone is a fantastic developer, but can't appreciate why a process like this is valuable to them and us, then it's a good thing they aren't applying. I need our developers to be competent at development but also reasonable in their perspective on the give and take between a business and their employees.
It's deliberate that I don't want people with your perspective/attitude to apply. I don't want people who can't appreciate the others parties constraints. It's not going to be a good match in the long run. And that's true even if said person can code circles around our best developers.
To me it sounds like you needed some way to thin the stack of resumes, and this is the method you chose. I'm just skeptical that you're going to get a lot of quality candidates. How are you going to pull developers who are comfortable in their existing jobs with such onerous requirements? These are people worth hiring: they're competent, skilled, and don't need to find another job.
Though looking at your profile a little, I think the answer is clear: you don't try to. Rather, I think maybe what you mean when you say fit is "we filter for people who need this job so badly that they'll go through our hoops to get it."
perhaps its that a web dev shop can't afford to hire the best talent, so the that sort of a filter is useful.
Indeed, this simple filter is the original purpose of FizzBuzz and other simple coding tests. Circa 2007: https://blog.codinghorror.com/why-cant-programmers-program/
For a 90 minute take home exercise that I have to complete before I am allowed to speak to a human being, that is on average about 10 hours of work your company is expecting me to do, for free. And then I have to do a coding assignment afterwords anyways.
I will say that the fact that you post the salary range is good. This is the sort of time investment that would only be worth it for a top end salary.
Also, I wouldn't say it's "not allowed", just not the standard process. We have, for specific reasons at the candidates request, done a Zoom meeting earlier in the process.
Candidates are also encouraged to email us with questions and we put a lot of effort into being responsive and good communicators with the applicants.
And yes, I agree, it seems crazy. But it's our reality and seems validated by others I know in the industry and anecdotes here on HN.
There is just so much demand, it pulls people into the industry that can't really perform as professional developers.
I think the frustrating reality is that it's usually not possible to tell the good from the bad at the portfolio/resume level, some kind of skills/competency test must be given.
I personally don't look at GitHub repos because I don't know how long it took to write that code. If it's great code, but took 4x to write than it should have, I wouldn't know. There are also a lot of applicants that just don't have much public in GH.
Finally, to be clear, I'm not saying they are all bad actors. I'm sure some are and the same for employers. Mostly a result of the current world economics.
This is definitely the most important aspect of this discussion.
So, those are the first two things we test. We give the candidate a description of the work to be done and ask them to interact with it. What questions would you ask the client and how would you break this down into stories/issues? Then, we give the schema of the existing database and ask them to modify it so the new schema supports the work requested.
Still very software development oriented, just not coding. We have more in-depth coding exercises later in the process.
I realize in some orgs, there would be a project manager or product owner of some type who would break that work down. And we have team leads who do the majority of that work. But for our org, devs who couldn't do this struggled to perform well in the development work too. So it became a skill that correlated to being able to succeed after hire.
Why? Because people tend to hire people with qualities like themselves, and overlook many things. It also then caused people to overrate/underrate the analysis of coding tests based on the previous interview (as they (dis)liked the person).
This also massively increased the diversity of our team.
That's not what Litebulb is for though, we're mid/late funnel. As a candidate, if I have to choose between a 4 hour onsite or a 5 hour take home, I'm probably going to do the take home.
In terms of pay, I know some really good companies with a healthy eng culture and heavy pay that do take homes (not as a pre-screener, for sure).
Btw from my experience, resumes are a pretty weak indication of skill. I have interviewed people with 15+ years of experience, some at MAANG, that couldn't comprehend basic systems infra scenarios, and I've interviewed interns that had production-grade code. You need to actually try it out, one way or another, to be sure.
Note that it's an option, not mandatory, to do it async. If the candidate wanted to do it in a sit down session, then we prioritize making time for it to be there the entire session. The reason we offer this option is because devs that have busy lives often can't easily find a single consecutive multi-hour session to do an interview. They have full time jobs, and might have family obligations at home and over the weekends. Basically, the only way to actually do a full onsite interview is to take a day off work to go do the interview. We're giving the flexibility for the candidate to find a chunk of time here, a chunk of time there to complete the interview.
We tested this above approach, because just as you said, we did see a huge dropoff when we just emailed candidates the interview link. After we offered to hop on a call and to be available for support, we saw dropoff reduced to less than 15%.
Also, a good chunk of candidates that drop out of interview processes are senior engineers that can't be bothered with lengthy processes, which is fair. I wouldn't give a Litebulb interview to a senior engineer to begin with. Debatably I wouldn't give any kind of technical interview, actually. If they have decades of experience leading teams at big or high-growth tech companies, those achievements speak for itself. The interview process for seniors should be more geared towards finding a product-team-eng fit, rather than an evaluation.
That is the fundamental issue with technical interviews, because there is very little about that manufactured situation which is applicable to the candidate's ability to do the work. There are extremely few real-world coding situations that come anywhere close to the pressures felt while having someone hyper-analyze everything you do in real-time, particularly when that someone is responsible for deciding whether you're able to pay bills next month or feed your family next week. It's a needless, pointless and ultimately self-harming methodology that filters out exceptional candidates in favor of the most extroverted or arrogantly confident, which is a bizarre twist considering the most influential and game-changing coders are also famously some of the most introverted and "unusual" people.
On top of that, because we're all in dependency hell at this point, far more time is spent googling for framework errors that don't make any sense until you find that one undocumented command line argument that a single person posted deep inside a GitHub issue thread. If you want to test someone's ability to do the real work, then give them an obscure error code from a little-used utility and see how long it takes to find the fix.
This is actually a type of interview we're trying to support! A blocker right now is that once it's leaked, you basically have to throw the interview away. The good part about the "feature building" type interview questions is that it almost doesn't matter if the prompt + codebase is leaked, because your solution is probably still not optimal. "Good code" is super subjective and takes years of experience/learning to master, but an obscure bug fix shifts the end result to a boolean state, which is easy to leak (thus hard to detect plagiarism).
Please use your platform to spread this message to employers. I have increasingly been getting leads asking for some sort of automated testing, even when I bring 25 years of hands on and management industry experience. The tests typically require multiple hours of time investment, with no real guarantees. That's asking a lot.
1. Hop on a call, do the interview in one sitting together and share thoughts/questions
2. Hop on a call, make sure set up is smooth, prompt is clear, questions are answered, then candidate goes off on their own. Hop on another call later in the day to demo the solution and talk over design decisions.
3. Do it completely async, submit the solution whenever you have time
From what we've seen, this flexibility has simultaneously reduced dropoff rate and increased candidate experience, since we're catering to the candidate's needs. Some would prefer to do it face-to-face, others prefer to work on their own.
The point would not even be to try to necessarily solve the problem itself - more just to see how I got along with them. Of course some people might freeze up when asked to do this, but I think that happens in any higher pressure interview environment. And if I do not know the solution off the top of my head either and am obviously even struggling in some parts myself I think that takes a ton of the pressure off of them to be a wizard that can solve it in the moment.
As an add-on, an interesting question one of our clients asks as a conversation starter is "what's the most technically complex project you've ever worked on?". This question sets up the space such that the candidate is the expert (since they've already worked on it), and it almost becomes a session where the candidate teaches the interviewer.
I do agree though that some of our frontend interviews are a bit heavier on code + css and less on systems design thinking.
Btw just to clarify, we have a policy where we won't host any interviews that are just open dev tickets to build a feature that will be used in production. Like, we DO NOT tolerate companies using interviews to get free labor.
What does that change from a candidate perspective? In both cases the candidate wastes their time. I'd offer a better solution: the company pays market rate for the time needed for the test, and can then ask for anything they want - in which case an actual ticket is the best solution as it will evaluate the candidate's performance on the kinds of problems they'd actually be working day-to-day.
I do not take personal offense to being tested in an automated fashion.
In one case I did a prerecorded interview and they later contacted everyone because something went wrong and they wanted everyone to do it again.
Done feeling like a number.
There seem to be markets (or large enough organizations?) where coding job ads get a tonne of applicants, many of them inexperienced, and so automation is a good fit. But not when things are tipped the other way.
Our bet was for real-world tests too but it wasn't enough. A few things we missed that might help…
- Candidates don't want to be treated like cattle
- For many companies a good interview platform will be more beneficial than automation
- Companies say they care about the experience candidates receive, they'll say they're rigorous and try to be as objective as possible with the way they collate information and make decisions, they'll talk about how high their standards are, etc etc. Be weary
- There may be more gains to be made outside of the tech space
- The problem most companies complained about and is probably still the hardest: going out and finding people
I like the way you've solved the "real-world challenge" problem.
Good luck and all the best!
Those are mostly the "sweat-shops" that hire developers that can work for less for several reasons. I see that in this particular case this solution might be very welcome to assess new hires.
But for more high-profile jobs and companies, the company that eventually follow this path will probably end with low quality hires, unless of course it just create crud's anyway.
Because this is a self-fulfilling prophecy in the end. Once the candidates know what they will face, they will train until they get good in that game, which most of the time doesn't mirror the qualities required for the job.
Currently: candidate does a Litebulb interview, gets a report, that report can be shared with whoever they like, hopefully speeding up interviews at other companies.
Next up: candidate does a Litebulb interview, gets a list of actively hiring companies that use that stack and would like the candidate to be inserted mid-way into the interviewing funnel.
And if they mage to get a way to find underrated workers, they will make both sides extremely happy.
So kind of LinkedIn badges but with actual meaning :-D
If I could select which repos to include, that could be very interesting. Also interesting additions: open source contribution analyzer, Stack Overflow Q&A quality analyzer.
Stars should be counted but can't be the only measure.