Ask HN: Should we change the way we do software interviews? What's stopping us?
hxngsun.wordpress.com
hxngsun.wordpress.com
Software engineering is the science of avoiding coding! And the code that these elite teams produce is often pure shit anyways.
The problems can be solved naively with a straightforward solution, or much more optimally after some further considerations.
The approach is absolutely not b.s., and we observe:
* some candidates completely misunderstand the very straightforward problems posed and their solution has no hope of addressing the issue
* some candidates have a rough idea but can't write the two nested for-loops required, for example, for the naive solution to one particular problem
* some candidates don't ask for clarification or confirmation that they understand the problem correctly even when they'd benefit -- others do and make better progress
* some candidates don't consider the full range of allowed possible inputs for the problem statement and assume the happy path always ... this is plenty forgivable, but when this is suggested or pointed out, some candidates find an easy way to resolve, others jump through interminable hoops to address, and others have deficit in communication that doesn't let them discuss the issue
* when discussing perceived flaws in implementations, some understand the language used to discuss software, others have a hard time communicating or answering questions about their implementations -- e.g., when one candidate wrote a recursive function all of whose branches ended up calling the same function recursively and we asked "when will your recursive function ever stop?" in many forms, they could not see or understand the intent of our question
* some candidates misunderstand writing a solution addressing a particular test case vs a general solution
* a few candidates are able to see the more optimal solution and write a good second algorithm based on that approach
* very few candidates resort to stepping through code in their IDE, unfortunately, to see or demonstrate what happens
We've eliminated many candidates we feel would have been a poor match based on inability to understand requirements, consider all possible input situations, inability to write naive algorithms, inability to communicate effectively about interview problems. ( Note that we also let candidates continue addressing the problem as "take-home" after the interview if they show promise. )
One junior candidate we just recently hired in a South Asia office did quite well on the algorithms portion, but unfortunately did not know Linux/UNIX/macOS command-line ... and assumed she was going to fail that portion of the interview -- she'd worked only in Windows contexts before. We had pointed her to a Linux-in-the-browser site and we just asked her to Google/search for the solutions to the various fairly basic operations we asked her to perform on the command-line -- she was surprised we'd let her do that. She struggled through and found solutions, and that's exactly what we wanted to see. There's nothing that exposes a candidate's suitability more than an open environment with a problem to solve and opportunity to discuss.
Your company has become very good at finding what people are not good at.
Your company is not alone in that approach and this is exactly where the tech industry interview process fails.
It looks for the wrong thing.
I’ve been trying to cook up plausible examples but it’s a challenge.
I feel like maybe I screwed up here with the thread since it got flagged early on for having Ask HN in it, and people responded with a knee-jerk reaction going straight to the comments section. Not sure if it's possible to get a mod to rename this and remove the Ask HN, I originally tried to paste in my content directly into the self-post textbox but 9k chars exceeded 2k chars.
But, I'm now a dev manager so I get to sit on the other side. I've been doing interviews in the last week for a dev position and I don't subject candidates to coding. I talk with them about what they work on, one of my senior devs sits in and goes even deeper -- learning what they know without even really asking formal questions, for the most part. Past that I care more about them being smart enough and willing to learn new things, and easygoing enough to get along with the rest of the team. Hell, that's at least half of what I care about to be honest.
Some of this code-screen mistrust seems to come from the flood of poorly-qualified candidates; maybe there's a more diverse pool than before, but it comes at the cost of lots of wannabes.
For remote developers I do deviate slightly and I actually ask them to do a little coding, but that's because it's always someone in Hyderabad and the cultural differences make a deep-dive conversation with body language a lot harder to pull off. So I ask them to whip up a demo program that scores a game of Bowling, take as long as necessary, and bring the results to the interview and show me what they did. Not much but it works (and scoring a bowling game is actually not a bad test -- not as trivial as it sounds, but doesn't take a whole evening of work just for an interview).
Ok. We're done. Now check references.
We'll follow that up with an audition, show up on a Saturday and pair up with team members for some random hacking, see how you handle working closely with people on a problem.
Whenever people are available we can see how they'd work out. Sometimes it's a Saturday, sometimes a Tuesday. I think you're reading far too much into my comment than was intended.
I'm launching a new venture that will try to help connect great engineers with great companies. Companies with great teams, competitive benefits, and the belief that engineers shouldn't have to go through bullshit interview practices and shouldn't have to study 150 hours for a round of interviews.
The flow is pretty simple: a quick Skype/Hang out call to learn about you, what've you done, what you want, and what you like/don't like.
Next, you'll get access to a git repo. In that git repo is a project with real code and a few features/bug fixes for you to implement. You commit against that repo just like you would at work. After a certain amount of time, you'll lose access to that repo.
One or more engineers will review the repo with no knowledge of your age, gender, work history, etc. Based on that engineer's feedback, we match you with companies that we think there will be a good fit. If there is mutual interest, you meet the companies in person for a final round where you can't get rail-roaded with bullshit questions.
Email me @ interviewingisbroken@gmail.com if any of these are you:
1. You're in the Bay Area or NYC and interested in fair interviews with great companies
2. You're in the Bay Area or NYC and work for a company that wants to offer fair interviews to get great engineers
3. You're a great engineer who would like to get paid to help create interview projects or perform reviews of candidates code - we're paying up to $100/hr. Great side hustle opportunity.
I wasn't planning on announcing anything about this yet but this post got me so fired up.
We don't turn around an use the code the candidate wrote. If you consider the cost to use to write the project AND the cost to have it reviewed N number of times, we'd be overpaying drastically for that feature to be implemented.
Code reviews worth both ways.
Candidates who did the unpaid version saw the clear value: they invest 3-5 hours once, and potentially cut out 20+ hours of first round interviews, or even all the studying like the OP did.
Nobody that we've tested in the beta complained about a few hours of free "work" - they were actually excited about it.
How is this going to broaden the application pool to include neurodiverse candidates and not just pander to the credentialist notion that rote leaning to get a piece of paper makes you the best candidate.
We're only in private beta now, but the goal is that when we launch, it will be open for any person to apply to. I don't see our model ever being some that "panders to the credentialist notion that rote leaning to get a piece of paper makes you the best candidate." It's the exact opposite, really.
Everyone wins - engineers don’t get their time wasted, and interviewing companies are quite interested in not to waste theirs.
Is this from you or from the company? If it’s from you what kind of project is it? Web development? It seems like the technology stacks are so diverse it’s hard to give a coding test that’s applicable to a wide range of companies.
We don't reuse the projects across different stacks. We might borrow some of the underlying themes or principles, but they're different projects for different stacks, each written by somebody with very precise domain expertise.
Just to chime in: I'm a PhD student in CS/machine learning with a competitive enough CV to interview anywhere, and I have gone through the motions at many companies previously (e.g. for interning), and there is nothing I hate more than coding interviews.
It literally makes me hate companies, it turns me off from interviewing at a place, and it makes me deeply resent them even if I get an offer after a ridiculous series of interviews.
After you have subjected me to a terrible/demeaning interview process, I might still work for you, but I will never be a loyal or feel any inclination to contribute more than I need for personal gain, because you have established that nature of the relationship via the interview. Sorry for the rant.
I hope you succeed, so very much.
I get that many people, especially seniors, are annoyed by the general software engineering process because they feel entitled to certain positions. But for the most part it seems to make things as meritocratic as possible (aside from holding back certain populations because of bias, something people seem to be working on fixing).
So yeah I agree several day-long interviews for a company is draining and a huge time suck, but aside from that relatively minor inconvenience it puts everyone on a level playing field, and those who are “too good” for the game are free to opt out.
I’m annoyed by the process - as both interviewer and interviewee - because I don’t think that it has much to do with the actual work.
As a fun aside, I've seen people send out HackerRank evaluations for entry level positions and I think that's ludicrous.
The best interview I have had was me Skyping with one of the devs and he gave me a link where we were able to share a browser session (for the life of me I can't remember the name of the site), but he described the problem and as I typed he could see what I typed. There was no emphasis on result that I can remember, just the process of writing code, talking it out loud and seeing code evolve and asking whether certain edge cases should be considered.
What's most broken, I think, is the fact that interviewing is an uneven, broken experience that's different everywhere you go because no one really knows how to interview. I haven't seen any company take metrics on whether their process is good or helpful or filters out too many people or not enough.
Interviews where people just wait for you to make a mistake should be not a thing anymore.
(Looking at you, Google)
Sure it would be great if we found some super-magical process that didn't irritate or inconvenience incumbent and new developers alike. But you probably can't please everyone, and in the meantime we need to do the best we can because hiring still has to happen. Companies with slightly better processes will be rewarded (hopefully) and companies with more painful processes will face some penalty from losing candidates that would move their company forward. Hopefully this is enough of an incentive to keep companies continually working towards better hiring practices. But I kind of doubt it.
Of course this applies more to fresh grads.
We do an initial code review. If we have notes for improvement, we give the candidate feedback and ask if they'd like to do one more pass at it. If they don't want to, but we think that the general thinking and instincts behind the logic is sound, we bring them in for an in-person where we ask them to go over their code review on where the pitfalls are and where they think they could potentially improve.
That gives us the most realistic sense of how well a candidate is going to do at the company, since this is essentially our process.
My conclusion was that it's mostly about being able to solve novel (which means preparation matters less) problems in code and explain the reasoning behind the solution. It is of course possible that I was just lucky in having good interviewers, in which case I guess that would answer the question, more of that and less of syntax checks or finding the one true answer-type of questions.
It's not remotely scientifically-objective, but most things in life aren't. I'd liken it more to dating. Both parties are evaluating each other on innumerable criteria through their own subjective lenses, and trying to reduce it a scientifically-quantifiable measure is dubious.
If an interview process is completely detached from what it takes to succeed at a job, what good is it?
- Whiteboards are unnatural. Besides, you don't want to check their knowledge of university-grade algorithms or data structures.
- Problems to take home are an artificial setting - you can't be absolutely sure that they did the work themselves, and you have no idea how the mesh with other people. And you probably get some false negatives because they're stuck and can't ask their colleague.
- On the other hand, if you're using real code from your project, you're just asking for unpaid work. And paying every candidate is a significant expense.
- Using a proxy measure like HackerRank or a Github portfolio excludes people from minorities or who only want a job and not a mission.
- Not doing any coding opens you up for people who can talk, but not code.
Have I missed anything? Is there a good alternative to find the right people? I'd love to improve my company's interviewing process, but sooner or later we always hit one of the mentioned problems.
I think candidates should definitely code at least fizz buzz or something as a sanity check. It probably shouldn't take more than 10 minutes.
> - Problems to take home are an artificial setting
How do you feel about potential solutions I have mentioned in the post, e.g. exposing the candidate to a project during the on-site for 3 or 4 hours?
I'm also not a huge fan of take home projects unless they can for sure be time-boxed to be less than 4 hours, and it would be ideal if you actually got paid for the 4 hours of time.
> - On the other hand, if you're using real code from your project, you're just asking for unpaid work. And paying every candidate is a significant expense.
Paying every candidate is an expense, but is it less expensive than hiring the wrong candidate and then needing to fire them? It seems like other comments here have already pointed out that they pay something like 160k annualized salary hourly rate, which seems pretty reasonable.
It's certainly another interesting idea, but from the top of my head, I find several drawbacks: The first is NDAs, as not all code may be exposed to non-employees. Secondly, it will disrupt your team for a bit, because at least one developer should be available at all times and can't really join discussions or go into deep focus time. Finally, and most importantly, I think it's impossible for such an interview to have internal consistency: Every candidate will probably solve a different problem (or do you keep a bug open just for interviews?) in a very different social situation. If you're more the nervous type, you suddenly have a whole room full of full-blown developers see you sweat. Internal consistency, however, is one of the most important themes in interviews for me. Everyone deservers teh same chance.
> Paying every candidate is an expense, but is it less expensive than hiring the wrong candidate and then needing to fire them? It seems like other comments here have already pointed out that they pay something like 160k annualized salary hourly rate, which seems pretty reasonable.
But is it less expensive than other methods of interviewing? If a classic whiteboard has a slightly worse chance of hiring the wrong person, but has no additional cost, it might still be better, all in all. Additionally, the expense depends on how many you interview for a given position (is there a filter interview beforehand or do you invite every reasonable resume?). A better idea that we sometimes use it to hire uncertain, but hopeful candidates for a paid internship. If they're good, the get a full contract; if not, we haven't lost much. Also, it's an easier decision after several weeks than after four hours. However, we can't do this for everyone.
I'm going to assume you are including the costs of firing people as part of "interviewing" after they have been hired and found to be a bad fit.
I'll take a stab at this. 160k annualized per hour is about 77 dollars an hour, if we're assuming 260 work days a year and 8 hours of work a day.
$77 * 4 hours of a day = $308 additional cost per interview. I'm going to assume you're already doing some panel interviews with at least 2 people, so I won't include their cost in there. And there's potential savings if we only have 3 people interviewing for the 4 hours instead of 4 or more.
Let's say I am trying to hire for 20 positions, and I am willing to interview n=15 times per position.
20 hires * (15 interviews * $308 per interview) = $92400 additional cost, whether I get 0 or 16 successful hires out of this process.
If we think about the cost of onboarding someone with 3 other employees who are getting paid about the same over a period of 3 months:
308 * 3 employees helping with onboarding * 65 work days * 8 hours * 10% of time spent onboarding = $48048 cost of onboarding 1 employee
Let's define a bad hire as someone who contributed no value and we'll treat the salary we paid them up until the point we fired them as a sunk cost. It takes maybe 4 months (260 * 4/12 = ~87 work days) to figure out that someone should be fired if it's not super obvious?
Let's say we would have normally only hired half of our target candidates (10 out 20). I'll also be conservative and say the new process reduces the number of bad hires by 10%. So assuming 1/10 of our actual hires was bad:
~87 work days * $308 per day + $48048 cost of onboarding 1 employee + ~$5000 for health benefits = $79844 cost of firing 1 person, not including the cost of other employee benefits, the effect on morale, etc.
So here you're adding a cost of about $12556 dollars. I think the cost of firing someone is actually a lot higher than what I have calculated here, though I don't know what the actual figures are for that.
But I feel that if candidates at least enjoy the process more (which based on this thread, it seems like a lot of people here would), then isn't that a drop in the bucket in the company's bottom line for something pretty valuable?
There's obviously a lot of other intangibles at play here and we can model this all sorts of ways. Let me know if I've made some egregious error.
I recently went through a job search and I ended some coding tests because they took too much time and the questions I was asked seemed more like a random candidate filter rather than a test of developer knowledge.
My personal opinion is to make an easy coding exam similar to triplebyte or other apply once talk to many companies and randomly accept a portion of candidates.
That way people will know it's not their skill and can optimize for market exposure rather than arbitrary coding problems.
Then have an in person where you see if communication can happen. Ask them to walk you through the code used on the entrance test.
Can the industry move towards something like match day for doctors, but more ongoing?
Maybe that's the approach one takes when practicing, but hopefully they are seeing the pattern and learning algorithms to apply to solve each class of problem - or they already have learned that through their career, and such problems are just practice & exercise.
It's not about whether you can solve it, it's about how you solve it. But unfortunately, both interviewers and candidates focus too much on the former.
> – What kind of quality work does this person produce in a normal work environment where someone isn’t breathing down their neck?
> – Can I work with this person on projects that span weeks, months, or years?
No matter how well they anwser questions, how they whiteboard, how high is their IQ, or how well they do homework - none of it would tell you anything about their work quality or actual on the job performance.
Just ask them a for loop question and give them a contract to work on something. If it works, maybe hire them.
In California, it is at will employment anyway but I get the feeling that contractors are easier to fire than employees. Correct me if I am wrong.
On the other hand, candidates who are actually good will do very well and get through without any problems. I feel like it would take at least 2 to 3 weeks to find red flags, then at least 3 months to determine if someone is a superstar, and at least 6 months for everyone else to shine through.
1. Can code:
a. Gifted: b. Persistent: 2. Cannot code/haven't coded in a long time
How can you test for coding? In my view it's certainly not some academic question or the in-vogue programming interviews book. This is clearly proven to be red herring. Also at least in the bay area, there are set questions that FANG and other companies have which are shared all over the internet and amongst friends who work there. So an interview becomes a casting process for a role, where the script is the questions you have "rehearsed". This is an absolute facade and needs to stop!
I ask simple programming questions and look for "naturalness" in the way they answer and talk through the solutions. No complicated brain teasers or the smug "look at me, I know this obscure/complicated algorithm invented before I was born" questions.
Coding is one facet, the other one is design. No matter how good a coder you are if you have bad design skills it doesn't matter. How do you test for design?
1. Resume experience: Talk through technical issues/challenges, pros/cons etc. of projects they've done before
2. A simple-moderate design question: If someone lists distributed systems as one of their skills, ask them a question on it. A decent amount of determination can be made about a candidate they way he/she answers.
I'm refining my process after each interview I take and would love feedback.
I then have another test class with simple failing unit tests. They have to write the code to make the tests pass.
Then I give them more complicated failing unit tests that they also have to make pass without causing the first set of test fail.
We do pair coding and they can ask me any business question they have if they don't understand a requirement.
They can use Google if they need to during coding.
1) You don't do problems you've already solved that they have to come up with same answer to.
2) You don't do algorithm / problem testing that they're never going to use on the job.
3) You don't do whiteboarding except as you would in real life.
Engineering teams are people working collaboratively to solve complicated problems where nobody knows the "right" answer. Any interview that's going to tell you something is going to measure along those metrics, and not "can you invert this b-tree on a whiteboard."
Everyone hates interviews and how they are useless but in the the end we settle for the status quo.
Enterprise SaaS, Consumer Social and ML API companies will all have different working environments. Add on to that options of remote vs on-prem, CI/CD vs TDD, customer integration requirements, staff numbers, perks etc... and you have completely different needs for workers and teams.
A team earns their hires, and people will naturally gravitate to environments that suit their work styles.
In the context you presented this in, I agree with you. However, interviewing is a skill that must be learned or you are going to make loads of beginner mistakes. In that sense there are many right things you should be doing in an interview and whatever your company style is, you're going to want to use at least some of those best practices in your interviews if you want to find the best candidates in the shortest time.
In theory there is an optima between a new hires ideal workplace and the hiring company's style of work. As with most things it's about knowing yourself and identifying in others if there is a match.
It's the very reason I mentioned it. Too many people haven't figured out the very obvious fix: get some training in how to interview!
1. As a candidate, if you are in a strong enough position to refuse the job, then refuse to do white board exercises with no resources. No one codes like this in real life so refuse to do it in an interview. Be willing to walk out. I just say "I'm a good programmer and your interview process is broken. If you want to hire good programmers it's your interview that needs to change. We can start that right now and talk shop as programmer to programmer about coding, or I can leave. I'm happy with either option but I'm not going to waste your time or my time with whiteboard exercises that don't find the best programmers. How often do you use composition over inheritance here? I've been learning a lot about that and it's really helping me write better code. Can I show you some of my recent code where I've been doing that and get your opinion on where it could be improved?" Remember that your job on the interview is not to follow their script. It's to prove to them that you are the best candidate for the job. Go off script and take control if they are bad at interviewing. If good programmers are regularly walking away from these types of interviews, the interview process will change.
2. Unless you've been trained to interview people, you are probably really bad at it. Think about how bad someone is at coding who never did it before and has no training. Interviewing people is no different: it is a skill that takes time and effort to learn. Get trained up, and practice with role playing with colleagues. As much training as you have time for. Educate yourself in your free time on a regular basis with articles about best practice. Adding more words won't make this sink in more, but I will end by saying I can't stress this point enough. You are bad at interviewing. You need to fix that before anything else if you want to hire good devs.
3. How are you measuring success as an interviewer? This is a really difficult thing to measure because how can you find out that your interview process is rejecting candidates who would make great employees? I'm not going to try to answer this here, but it's an excellent question to think about. One thing that might help is the next point.
4. Whatever your first impression is of the candidate, spend a good part of the interview trying to prove yourself wrong. Candidate seems like a perfect fit for the team and a great coder? Instead of relaxing because you're sure you already found the right person, see if you can figure out why they might be a bad fit or if their coding skills aren't as good as first impressions. Same for someone who seems weak at coding. Maybe they are just nervous. Can you make them relaxed enough so their awesome skills can come through? Maybe in both cases your first impressions will be proven correct. But if not you might have just found a diamond in the rough that other companies are skipping over.
I hope this very incomplete list is enough to get folks interested in following up on point 2.
Does it make more sense to paste the link into the body of the text and just ask the question?