Three hundred programming interviews in thirty days | Hacker News Reader

Three hundred programming interviews in thirty days

blog.triplebyte.com

324 pointsuptown237 comments

It's great to see some objective research being done on this, and I am very interested in following the results.

They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a full-time job and a full-time life outside of work. Spending that much time to qualify for a single job is too much to ask of anyone. If it were to pass a generic proficiency certification applicable to many positions, I would consider it, but this does not scale if a candidate is applying for multiple positions.

My wife is in this position now, and it's not for a programming job. She works in pharmaceuticals, in pharmacovigilance & risk management. She does a lot of data analysis & writing (the latest report she wrote for Health Canada & the FDA was 461 pages).

A company she's interviewing with asked her to do a sample writing task as part of her application/interview. She hasn't decided whether it's worth it or not. For someone who has 15 years experience in the field, excellent recommendations and credentials, spending (her estimate) 12-15 hours doing pro bono work just to apply somewhere is ridiculous.

12+ hours is excessive even if you don't have 15 years experience.

I work in data analysis and I decline any "test" requests for a job or project.

If you don't value my time before hiring, you definitely are not going to value my time after hiring. Also, if you are incapable of judging a fit based on resume, interviews, and just talking with me, you are not mature enough, experienced enough, smart enough and intelligent enough and have failed the test for being my potential coworker and boss.

Edit: The tests make perfect sense for entry level positions and entry level candidates but once someone requesting tests for experienced positions and candidates, it shows the immaturity of the company, group, team and people who are unable to assess suitability of candidate and place too much emphasis on technical skills and tests as a crutch. The best jobs in my career were those where coworkers and boss had good relationship dynamics. Technical excellence had nothing to do with how well we worked and delivered results.

A simple solution is for the company to offer you something for the time you spent doing the sample test (money, gift card, their product,...)

How about a car? Then it is obviously worth the applicant's time and since a professional can deliver or destroy a Bimmer's worth of corporate value on any given decision it easily pays for itself in identifying good candidates and weeding out bad ones.

> it easily pays for itself in identifying good candidates and weeding out bad ones

It even easily pays for itself in just weeding out the bad ones.

I had a similar experience last year applying for a major technology company. Although I still lightly code in the devops sense (process automation) and to do typical management data analysis my day job is managing people, processes and crises.

If you have billion dollar trading systems and need a cool-headed guy who can talk to VPs or directors in one breath and SAs, DBAs, SAN, Network, etc guys in the next I'm your man.

I applied to head a global support organization, with teams in (from memory) North America, India and Asia Pacific. I'm pretty sure day-to-day would be a mix of employee career management, forward planning with my application development partners to make sure we were prepared capacity and training-wise to take on whatever's next, and crisis management of production issues and the like. Business as usual as they say.

Yet I found myself answering computational complexity questions about hash map insertions. To be fair, before I became quite so management-centric I've found plenty of badly performing code via the usual tools (dtrace, strace, thread dumps), and sql queries using analogous database tools, but at some point you have to rely on technical people to do their jobs.

Am I wrong to think if I'm going to manage a 60-100 person organization across three timezones I don't really need to be under the hood in the code any longer? Or am I clueless?

> Am I wrong to think if I'm going to manage a 60-100 person organization across three timezones I don't really need to be under the hood in the code any longer?

An organization that does this is hiring stupidly. They have clearly demonstrated via interview that they don't understand the job to be done and/or how to determine that a candidate can do that job. Asking low-level technical questions is a terrible way to answer what I'll guess was the intended "question", something like:

Does this candidate understand the technology and work well enough to make executive decisions grounded in our organization's reality?

> at some point you have to rely on technical people to do their jobs.

On one hand, I totally agree that technical tasks should be left to the technical people, preferably those who are actually working on the system. There's almost nothing worse than a manager trying to take technical decisions out of people's hands.

> Am I wrong to think if I'm going to manage a 60-100 person organization across three timezones I don't really need to be under the hood in the code any longer? Or am I clueless?

On the other, I think people want to work for people they respect and to the extent that people respect technical chops that might be what they are screening for. Though the hash map question sounds like a soft ball, so it might be screening for a baseline of software competence.

I hear you about having enough technical ability to understand what's going on and make sensible decisions. Certainly in a role like that and the one I have now a mix of technical and positional authority are required.

But at what point does it become ludicrous? When your CIO is yelling "No, No! You have to enable the EPEL repo or you'll patch to the wrong version!"?

Yep, it's clear that neither extreme is useful, and when that's the case finding the balance is tricky.

>spending (her estimate) 12-15 hours doing pro bono work just to apply somewhere is ridiculous.

Keep in mind that in today's job market, there are probably 10 people lined up who are willing to do just that.

Ah, but the willing aren't able and the able aren't willing. In any case, I'd avoid companies that recklessly fleece applicants just because they can. Once you've proven your worth professionally, begging for scraps is pathetic in any job market. For every greedy business with ten desperate applicants, there's an honest organization or untapped market ready to pay fairly for what value you offer.

The question I consider when asked to perform for free: why should anyone simply donate their knowledge to a profitable business after spending a lifetime to attain it?

Doing something because you can is the worst reason to do so.

Does she have existing samples of writing that she can point to, not locked up under copyright?

If so, then I would point to those samples and suggest those should be relevant data for assessment and I'd like to get further in the process before putting in extra work for this.

If they're prepared to discuss the matter, great.

If they just give a blank 'sorry, this is our policy,' then walk away. Why? Because if they're not willing to have a discussion with you, that means they don't care about you. It means they think they have a hefty surplus of candidates. Which means you are very unlikely to be the chosen one, and you would probably just waste a lot of your time.

Yeah, we agree. We don't plan on asking everyone to do hours of work on their own. A lot of people don't have the time. However, we're also seeing a bunch of applicants who are good programmers, but become really stressed in interviews and do badly (they freeze). These people have trouble getting jobs. What we want to do is offer them the option of doing a larger project on their own, rather than our final interview. I think this might help a lot of people (clearly we have to test this)

I personally prefer it to a point. My only request when I get this type of interview is that I'm allowed to make the project and requirements public on my github for future interviewers. If I'm doing a free project I feel it's fair.

Well, that's hard. Standardizing evaluation (critical to reducing bias) requires that everyone (or a lot of people) see a similar problem. And when someone is working on their own time, I think it's important that there's not working code easily accessible online. Perhaps this can be overcome by asking candidates to talk through their code after they are done

if you change the assessment every so often, you could permit people to release their work eventually.

I like that idea!

If someone solved a significant problem at your request, but despite this you did not hire them or otherwise compensate them for that work, I don't see how you have any right either legally or ethically to expect them not to then use the work to advertise what they can do to anyone else. The premise is one-sided enough already (though the argument elsewhere in this HN discussion that it might be preferred by nervous interviewees is an interesting one I haven't noticed before) and I think expecting any sort of loyalty or confidentiality after the process has ended is rather optimistic.

In most of the work at home projects I've been given there was a ton of available code online, they've primarily been for front-end positions so it's usually "Make a small app using your favorite library"

I can't imagine there are that many 3-4 hour projects that won't already have available code. The most important part of an at home project in my opinion is the walk through.

The best interview I had asked me to go through the project and explain the what and why I did things and asked for high level understanding of what the framework I chose or the browser/node was doing based on what I wrote.

It was an app in React and they went through why I decided to make a component for x, y but not z and then asked if I understood the virtual DOM as a concept.

To me that's the best you can do, you need to understand how the programmer thinks about problems, works through them and that they are engaged in the ecosystem at large.

I am curious why you have that requirement. What is the fairness you are striving for?

If I invest time to do something, it better be worth it for me.

Like getting a job?

A lot of companies will ask for existing code examples, in this way I can just point them to my example-code github repo and I've had companies use that instead of asking me to do a specific challenge for them.

I think this project approach is excellent - this seems to map much better to real experience in a job.

I had an enormous amount of trouble interviewing / whiteboarding initially but I got used to it pretty fast. However, I would still prefer a take home test as I feel its a far better qualifier than an in-person interview.

A lot of people feel this way, and I see no reason why it can't be an option at more companies

Interview environments are always stressful, but so are other common workplace situations. Being unable to manage stress effectively might contribute to poor on-the-job performance, even for people with great pure coding ability.

The process right now seems focused on finding people who are the best at only coding. In the future, do you intend to also consider communication skills? Senior developers act as mentors for junior employees, and often need to coordinate projects across multiple teams.

"Stress" is not some monolithic, universal feeling. This line gets trotted out frequently in these discussions, but no one has shown a correlation between the types of stress induced in interview situations and the types of stress induced in whatever situations are typical for your company. I would hesitate to even equate the stress of high-pressure situations in different companies.

Indeed. I am one of the people that freezes up in interviews. Yet I thrive in stressful work situations, such as customer emergencies (system is down), critical bugs, etc and I resolve the issues quickly precisely because of the right amount of stress.

The best way I can describe the difference in stress is with an analogy. When I visited the Grand Canyon and hiked around the mountain, my body literally froze. I couldn't move a muscle even though my mind knew I could do this and hike around. I knew it was irrational, but I was powerless. This is what interviewing feels like - a fear of heights. And with every rejection letter, the fear is reinforced.

Stress in the office is like playing competitive sports - it's a challenge rather than percieved harm to oneself.

Interview stress is of the type I call "defusing the bomb": everything at stake, essentially immediate time constraints and confrontational. This never occurs in real life.

"Unfortunately something not so different sometimes does. E.g., you're at a startup, you have a critical demo for your best-hope customer, it was scheduled very aggressively because the CEO wanted to fit the customer's availability, it's happening in one hour, and nothing is working.

Not exactly confrontational (though it could easily get that way if you aren't careful) but just as stressful.

Of course it's far better to avoid such situations, but they do happen.

You are ignoring the data.

Person after person says they do fine in the job, yet have trouble with interviews.

I've presented in rooms where the lowest ranking officer was a colonel, and most were important people at the Pentagon. No freeze up, easy peasy, because I know what the hell I'm talking about and because I know I'm not going to be judged on some bullshit evaluation. (re: "x<<8 from above). Interviews are more a crap shoot. I mostly get offers, but sometimes just utterly choke. More importantly, I usually feel miserable after.

Don't make up theoretical situations when you have real, empirical data, please.

a handful of self-reported, potentially biased posts is not data. It's barely even anecdata.

What data am I ignoring, and how do you know?

I made no comment on what kind of interview works best. I merely observed -- what is certainly true -- that sometimes "defusing the bomb" situations actually do come up in real life, where everything is at stake and you have to work under very severe time pressure.

Do you disagree that such situations sometimes arise in real life?

If not -- if your point is that that isn't necessarily a good reason for interviews to involve such situations -- then I think we are in violent agreement. I am not arguing for defusing-the-bomb interviews. I just don't think it's quite right to say that in real life you never have to defuse a bomb.

See my other comment but I think the differences are the immediacy of the time pressure, the sophistication of the thinking required and the all-or-nothing outcome.

So you're interviewing someone and basing a large part of the experience on something that would otherwise happen maybe 0.1% of the time? Also that CEO sounds like an asshole.

Usually those situations are just "grinding", you need to work expediently, and there is time pressure, but it is not immediate and you don't need to think nearly as much as in an interview. (code -> compile -> ohshit -> repeat quickly) not (oh shit, how do I solve this algorithmic question I have never seen). Most people don't freeze up in the 1-hour until demo situation.

On the other hand, the time constraints in an interview are immediate: you have a few minutes to come up with a solution to a problem (interviewer will put up with at most 10 minutes of silence or babbling), it's often a zero-to-one problem (usually there isn't a way to have a partial solution) and those factors combine into exponentially increasing sense of pressure: as time goes on you become less likely to solve the problem.

Finally, if this situation does happen at all and, contrary to my claim above, certain people do freeze up, it does so so infrequently as to be immaterial in a hiring decision in the vast majority of cases. Its practical importance is certainly disproportionate to the weight placed on it in interviews.

I think it's really on the interviewers (and us) to show that people who freeze in tech interviews would also do poorly in a job. It's probably sometimes true, but many times not.

I don't think there's any relation. Some people may be nervous just because they are interacting with strangers, whereas they would have no problem with people they already know.

Personally I don't have nervousness/stress issues in interviews, but I've found that conversation screws up the analytical mode that I use when I'm programming. It's like how people say they don't like having their managers interrupt them in the middle of the day because they get taken "out of the zone." I can program or I can converse, I can't do both without screwing up both of them.

That's just my particular conundrum. People probably have all sorts of reasons (besides the possibility of them being bad at programming) for why they may have trouble in interviews.

Couldn't agree more. When I'm trying to think through a problem I want everyone around me to be friggin' quiet and leave me alone.

When I'm thinking about an abstract concept, trying to visualize how the pieces fit together, 'talking it through' is not helpful either. It's really goddamn distracting.

The feedback I got from my last 'cs 101 algorithms whiteboard quiz for a web dev job' was, "You need to talk more and explain what you're thinking." Ugh. Do you want the code or do you want me to talk, because you're not gonna get both.

It's amazing to me that a real human who has experienced job interviews could think that common workplace situations are comparable coding in front of a stranger at a job interview.

Programming interview stress is a terrible indicator of real-life leadership ability. But communication skills are still an incredibly important of being a good programmer, often more important to success than programming ability (http://blog.codinghorror.com/how-to-write-without-writing/).

Personally, I've spent hours working on a project or problem only to find that, be it the result of miscommunication, lack of critical thinking by a manager, or lack of understanding on my part, what I'd created contributed nothing to the bottom line success of our organization.

Anyone, what kind of resources are there to objectively assess a candidates communication skills, leadership ability, and teamwork? What is a constructive way to provide feedback and resources for interviewees about their performance?

Giving an option is an awesome idea! It says a lot about the flexibility and open-mindedness of your company. Too many companies have a "take it or leave it" interview processes.

IMO it depends on the position. I did a test-project with a follow up code review for my first job and it was a fantastic. If I had not gotten the job, I still would have been happy with the experience and code to show future employers.

If you are interviewing for a more junior or senior position, many candidates are probably switching jobs and are already swamped with work. Also, they probably already have some pet projects they can share. Requiring a test project may seem amateurish.

I'm kind of in a similar boat. I've done them for companies that I like in the past, however I flat out refuse to do them for recruiting / job placement companies.

ammon / uptown, I've put off doing my screen-sharing section of your interview process (I got pushed past the phone screen?) due to a lack of time. Maybe adding a way to 'schedule' it would be helpful for those of us that live by our g.calendars....and you all as well.

What would be the best experience for you to schedule the interview? We'd definitely like to make that easier.

Just something simple, allow me to add it to my google calendar.

I did this once. After a take home quiz that took about an hour, I had a 6-7 hour "homework" problem. I did it, and sent it in. It took the company a month to get back to me, though I did stay in contact with the recruiter. The final answer, delivered by the recruiter, was "we've decided not to more forward at this time." No other feedback. I have no idea if anyone even looked at it - I never did speak to a dev at the company.

This experience left me embarrassed, honestly. I don't think it reflects well on me that I put up with it even once.

At this point, I will participate in tech interviews, but I won't do any more take-home assignments. I suppose this could change under different circumstances.

Gayle Laakmann McDowell, who wrote "Cracking the Coding Interview", wrote an insightful post her blog about this. It's titled "Companies who give candidates homework assignments: knock it off!"

http://www.gayle.com/blog/2013/09/18/companies-who-give-cand...

She mentions something I see as a problem as well - these tests allow the company to burn a lot of a candidate's time, but not the other way around. I may have burned 6+ hours interviewing unsuccessfully at google, but they parted with 6+ hours of developer time as well (though I did spend a lot of additional hours reviewing algorithms and doing sample problems). This creates a natural set of checks and balances that don't exist with homework assignments.

She does have some good suggestions about how to use a take home if a company is determined to do it. To me, a particularly insightful one was to have a high (she says 90%+) passage rate.

I agree. If you're going to ask a candidate to spend a day on a homework assignment, you should be very close to an offer. If you're using it as an early filter, you are wasting a tremendous amount of developer time.

Yep I've been burned a couple times by this and only under certain circumstances will I do a take home project. Even worse is when they also ask before hand not to post it to your github account. If they aren't going to give me feedback, then I'm going to host it and seek it out elsewhere.

Love that. If they burn too many people, they have to create new questions.

Exact same thing here. I've made it a personal policy not to tolerate these types of take-home tests. My time is way too valuable to spend hours on a project only to have it ignored by the company.

had same experience, no useful feedback after 6 hrs of work. it had to meet the provided written spec (js/logic) and visual (css/html) requirements and had to pass an internal test suite which i was not given access to.

the response from the recruiter was just "thanks, but the dev team said 'no', that's all the info i have for you"

Oh God.

Proportionate buy-in to the interview process is one of my screening criteria. If the hiring company is not willing to spend a similar fraction of their own resources on the interview process as I am willing to spend on it, I am much less likely to invest anything more than a token amount of effort on them.

Any company that does not realize that your time has value before hiring you will probably continue to waste it after hiring you.

I've seen several companies giving out such homework to me or my friends back in college. And to be honest, none of the are among the "great culture company list" people are talking about. Beside all the issues you mentioned above, those tests don't have a clear PASS line. You spent hours on that and can be rejected for no reason. Back in college I've done a code challenge from Box and my solution ranked No.1 in their leaderboard (although nearly 100 people are No.1 with the same highest score) and I was rejected without an interview. Another company I don't remember rejected me because I used (x * 8) not (x << 3). (But I'm interviewing a web company and I was writing python, not C in some embedded system) Most of the time I just don't know what the hell they're looking for. So now I reject all those HW or code challenge unless they agree to pay me for the time I spend. (Obviously no one has ever agreed my term so far.)

Even in C I would point out that the compiler will likely make such a trivial optimization for you. That was clearly a company full of morons.

Regardless of the compiler, let's say (x << 3) is slightly more efficient than (x * 8), I don't see it make such difference since my job will probably be writing code for webapp sort of things. For the things I do I actually believe readability advantage of (x * 8) overcomes the efficiency disadvantage (which doesn't even exist with modern compiler).

Some interviewers (not just the "homework") raise their bar by not allowing you to make any tiny mistake. And I just don't get it. If someone is good enough to write something like a lite version of Hacker New website in hours, I'm not going to turn him/her down because of such mistake.

And the person who could potentially write the "most perfect" code with all of the conditions applied is often not within that company's price range.

I did interviews at a company for about 2 years and there was constant pressure from management to do trivial crap like that, it finally came to a head and I invited management and one of the "rockstars" to do a mock interview.

When it was evident that the person they thought of as a "rockstar" could not solve these tests(without prior knowledge of the problems), they immediately discounted them as a worthless and stopped bugging me (about that, of course not about everything else).

Sometimes x << 3 is more readable than x * 8. It depends on why you are doing the shift/multiply.

What the hell. A developer who writes x << 3 instead of x * 8 when writing code that is logically doing multiplication is not a good developer. Written code is designed for human readability; the human should not be having to convert that mentally to its math equivalent to understand what operations are taking place. In nearly every possible case, this is at best a micro-optimization. At worst, a symptom of someone who writes unnecessarily obfuscated code to show off their knowledge in a manner that only hurts the maintainability of code by a team.

If I found bit shifting in an interview code sample as a replacement for basic multiplication, I would ask the developer why they chose to do it that way. They'll either a) calmly explain that their computer science classes taught it that way or it's a habit they've adopted after writing code for embedded systems or similar where the optimization actually made a difference, or b) their ego will make an appearance with a "because I'm so senior" attitude. The latter is not a good sign.

Or it's idiomatic for that language.

In fact GCC will virtually always produce optimal code for multiplication by a constant. Targetting AVR, for example (which doesn't always have hardware multiply), GCC produces more compact code if you use '*' than if you use '<<'. Similarly, any attempt I made to multiply by non-round constants using bitshifting and addition led to less compact code.

Uh, extremely unexpected.

Not really, simplifying simple operations (like multiply by a constant) is pretty much Compilers 101 / Dragon Book kind of stuff

The one case so far where I've seen better results using tricks like reciprocal multiplication and shifts was when there was no way for gcc to know it could throw away almost half of the bits due to input range limits. It would be kind of nice to have, say, a 19-bit integer type.

Not that does anything except to confirm the pointless of that selection criterion:

  % python -mtimeit -s 'x=1' 'x<<3'
  10000000 loops, best of 3: 0.061 usec per loop
  % python -mtimeit -s 'x=1' 'x*8'
  10000000 loops, best of 3: 0.0602 usec per loop

  % pypy -mtimeit -s 'x=1' 'x<<3'
  1000000000 loops, best of 3: 0.00103 usec per loop
  % pypy -mtimeit -s 'x=1' 'x*8'
  1000000000 loops, best of 3: 0.00103 usec per loop

(Obviously no one has ever agreed my term so far.)

My company does 4 hour work-product tests, and we pay each of the candidates for their time. We're in Portland, not SV, so maybe there is a difference in environment.

As a candidate I think the bodes so well for the culture and how respectfully people would be treated in such an organisation.

It shows a level of thought about people that goes beyond ping-pong tables and free pizza and coke (and ironically probably costs less than those things).

Let me guess ... you are happy working there right?

I have had an interview where both the (x * 8) and (x << 3) would have been frowned upon, for using a "magic number".

Good, I think? It wouldn't be a deal breaker at interview for me (or even something we test for, come to think of it), but they were right in that it is a magic number. Unless you are doing something where the context is so perfectly obvious that number should be an 8, it's better off tagged as a constant so the next dev to come along has a better chance of understanding your code.

Indeed:

  static const int eight = 8;

  x * eight;

Sometimes magic numbers are just that - numbers. Adding constant definitions is often a good thing, but sometimes just adds noise.

I think you mean

    static const int days_per_week = 8;

:-).

I like the idea of giving candidates a choice. For me personally, I would almost always take the "homework" if it meant no (or very short) in person interview. I like to code, not interview. Still, 6-7 hours is pushing it, it should be a few hours (3 or 4 at max, if you're not going to do a face to face), and a genuinely interesting problem (the kind of problem can tell you a lot about the company itself!).

When I interviewed at Apple years ago, they gave me a take home program to solve. It was interesting and I enjoyed it a lot. It required some good algo knowledge and had to be written in C. I was told in my face to face that few people ever solve it, and I enjoyed discussing it.

But the rest of the interview was unpleasant - your typical SV brain teaser riddled, algo whiteboard sessions for 4 hours, 2 days in a row. I wasn't really expecting it and wasn't prepared (my fault), so failed miserably. I think I suffered PTSD after that for a while :).

Now I feel preparing for that sort of interview is a huge waste of my own time. It has so little relevance to reality. I'd much rather put that time into building a product or learning some new PL or whatever. At least with a coding test I'm honing skills that I'll use day to day.

Homework can't replace an interview, unless you're willing to hire people who can code without any idea if they will fit in with the team or be able to work with others.

That sounds intuitively reasonable, but does an interview actually have significant accuracy in filtering out people who will do badly on those criteria? As far as I know, such data as is available says human intuition is completely wrong about this and an interview is not significantly more accurate than flipping a coin.

One take-home project I was given was very cleverly designed. Without going into too much detail, the project description initially seemed pretty simple. The actual getting-it-to-work part took about half an hour. However there was a requirement near the end that said the whole operation had to complete in less than X amount of time. The initial straightforward implementation I wrote took about 8-times longer to complete. Eventually, 3 hours later, after pulling out a lot of clever tricks I had come up with an implementation that met the performance requirements. The benefit of this type of approach is that it allows potential applicants of many levels to understand and tackle the problem while also allowing them to choose to bail out after that 30 minute mark if they can't think of any way to improve performance. Seems like a polite way to set a high bar standard.

Similar to how Project Euler works.

There are naive ways to solve certain problems (finding primes), but there is a general guideline that tasks should finish in under one minute (I believe).

> I had a 6-7 hour "homework" problem

Personally I would given them a link to my github which has hundreds of hours of work put into projects. From my experience a homework problem that long doesn't really prove anything (I could have outsourced it) and you never get feedback (this isn't the employer's fault but of laws in the US[1]).

I think a take home quiz is the only appropriate quiz if it's a basic less than an hour quiz. Harder than fizzbuzz but easier than creating a blog from scratch.

Don't even get me started on IQ tests or personality tests.

[1] - http://humanresources.about.com/od/selectemployees/qt/must-e...

I wish experiences like these were put on glassdoor as well, it'd be good to know what their hiring process is like rather than just actually working there.

Maybe enough responses about never getting contact from employers would make them start treating people like human beings

It's really amazing, isn't it?

I talked to thumbtack for data science. They wanted a 10+ hour takehome project after just speaking to a recruiter because their data scientists were "too busy" to speak to candidates.

Yeah, I'm going to get right on that.

Bet they're bitching about a data scientist "shortage (^1)" as we speak.

(1) data scientist defined as experienced data scientists who want to waste a day's labor before even understanding what they'd be working on and/or speaking to their potential boss

edit: checked my email archive and fixed company name. Apologies...

If 90% of the people pass the take home test, then why was that step necessary in the first place?

I've also soured on take-home test. I recently did a C/C++ one, I know I aced it, and I didn't even get a phone interview.

I figured out afterwards what they were doing. They were giving the take-home test to EVERYONE who applied. Then, after you submit a solution, they read your resume. After looking at my (correct) solution and my resume, they decided I wasn't worth a phone interview.

No more take home tests for me.

It's not that 90% of all developers can pass the test, it's that 90% of the developers that make it to the test part of the process can pass it. It should be a final filter to make sure they can walk the walk.

Our company does like to give a 2-4 hour take-home exercise after the technical phone screen, but (barring exceptionally bad submissions) we make a point to collectively spend as much time reviewing it as the candidate spent implementing it, for this exact reason. It's proved to be a very helpful filter so far.

"these tests allow the company to burn a lot of a candidate's time, but not the other way around "

Oh yes the other way around. At a previous employer, two of us took 3 full time months to review over 100 qualified candidates (i.e. that returned the task) to make 10 offers.

We made an offer to anybody who met our standards - whilst management initially restricted our headcount, they allowed us to expand the team as great people got in touch and the main restriction ended up being the maximum salary we were allowed to offer (which had us have to give up on half a dozen candidates). (We offered people the chance to show us their FOSS contributions to skip the task, but even those with big reputation couldn't resist tackling the problem.)

That's 3 full time months of not doing ANY work, no output, just full time recruiting, for two people who were probably in the top 10% of the company in terms of compensation.

It's not just expensive in time but also politically, as the team has no output to show for the period.

Where McDowell is correct is in the context of a large company whose hiring needs are very different. But I think that if you are building a small team where every member counts, you have no alternative.

> the main restriction ended up being the maximum salary we were allowed to offer (which had us have to give up on half a dozen candidates)

Ugh. I've been in a similar position, assisting with pouring over candidates. The worst part about salary being the limiting factor is you've quite literally wasted your time weeding through the poor candidates to find the honest-to-goodness best to hire - only to not be allowed to have the cream of the crop. It's ridiculous how much time some companies are willing to invest in finding the best talent, only to turn them down because of an extra $10-30k/year. If you're not willing to hire the talent, don't spend the time searching for them. Just post your job ads with your salary expectations, and accept the first people through the door if that's how you're going to operate.

Well, in theory, you could just put the actual maximum salary on the ad, but there's rarely a fixed budget in practice.

If someone is very junior or has some other discount factor (in practice that's the only real one), you'll probably go lower than expected for a 30 year old with a PhD, freeing budget to bump up someone else.

And sometimes, you get a stellar CV with enough brand on it to make the staunchest McKinsey reconsider his limit policy, so whilst you can't put the number on the ad, you can warn the candidate, get his number, and take a punt on management in case it might work.

The strongest factor for flexibility is when you have a chance to affect the outcomes for the company, but need to prove it before you can claim the pay raises. Then, it makes sense to take somebody at under their market rate, and 3 months later have a chat with management and bump up.

Hiring is a really sensitive topic because HR is often the biggest budget. I once seriously weakened my political position by attempting to automate away "Excel blue collar workers" in another department, which would have lowered the budget and therefore relative power of their boss internally. Those with the most flexibility in hiring (startups, which have no established internal territory to worry about) also have the least resources...

> it makes sense to take somebody at under their market rate, and 3 months later have a chat with management and bump up

Another strategy too commonly used by poorly run companies. If you feel someone will be worth $X more in only 3 months, they are worth paying that from the start. Kick the employee to the curb before their probation ends if they don't meet your expectations for the full salary. These games some companies will play ("we'll hire you at $x and then in 3 months we promise you'll get another $x") is despicable manipulation of employees' trust.

I worked for one company that did this to any new hire that was a pushover (they tried with me, I only accepted after constant back and forth for "HR approval" of my expected salary which was average market rate at the time). The gimmick is that nobody automatically got the promised raise at 3 months, and nearly everyone who went to fight for it never received it. An employee had to have already embedded themselves deeply enough into a product and be considered a "hero" in order to receive anything. The result was that the great developers would leave after being stabbed in the back, leaving the least talented people in play.

Suffice it to say, I don't even negotiate for salary or vacation time anymore. I request $x and X vacation days, and if the first offer comes back with anything less, I give them one chance to fix it. If the second offer is still shortchanged, I drop them. Unless you are desperate to find a position for financial reasons or you need your first job in the industry, these games being played during the hiring process are a strong indication as to the quality of the employer in general.

"poorly run companies" "strong indication as to the quality of the employer"

You might very well think that; I couldn't possibly comment ;)

I think your strategy makes sense and once one has enough experience and market value to make it work (after all, better be underpaid than unemployed), should be the default to adopt. It also shows strength and signals value to the prospective employer.

FWIW, most of those who came in got a raise later. I put my career on the line every time (and one too many, in the end). Didn't ingratiate me with management but it was my word (not HR's) and I stood by it. If I had one criticism to make to your otherwise accurate comment it is that it is not a company but a person you are going to work for, and one should align with people who share one's values.

I been reading your comments. You write well, normally now is when I'd be looking up your twitter.

Thank you, and I don't really have one. I'm a fan of long form, I think many ideas are complex and high dimensional and suffer from too much summarising. Plus, HN is relatively anonymous. I think I'm relatively easy to doxx, but the people I don't want reading my comments are people who don't read HN, although I think they know to do a Google search by now. Plausible deniability FTW.

Maybe another year or two and I'll have the balls to sign what I write.

Wow, you did spend a lot of time on these exams. 160 hrs a month, 3 months, 2 developers, that's 960 hours, easily. For 100 candidates, that would mean you spent almost 10 hours on average per candidate. In this case, I think you may have been spending more time on many of them than they are spending on you. However I may feel about take home exams in general, I will certainly say that clearly you are not using take-home exams externalize the inefficiencies of the interview process.

Unfortunately, I don't have any much evidence for my next statement, other than bits and pieces, but I think you are not a common case here. I don't think that most companies do anything close to what you're doing with these take home exams. My defense for this statement without much data is that companies are deliberately secretive about this - I understand it is for legal reasons, but I have no idea, none, if anyone even looked at my project.

Remember, McDowell said that this approach allows the company to burn a lot of a candidate's time, but not the other way around", but it doesn't say that they necessarily do. It's just that the equivalence (a 1 hour interview necessarily involves one hour of the company's time) isn't built into the process, it leaves the door for this kind of time wasting and abuse open, and I'm pretty sure lots of companies do abuse it this way.

BTW, if an employer promised to spend 10 hours reviewing my 6-7 hour homework project, with feedback, I'd try to find a way to make the time available, because it would be beneficial to me regardless of whether I got the job. But if they brush me off with a one liner from a recruiter a full month later, and ask people not to post the project to github (even if it might lead to interesting and novel solutions), I'm going to guess that they aren't doing what you're doing with these projects.

Ironically, the last time I complained about this practice on hacker news for exactly the reasons you described, somebody weighed into defend the practice, and an job applicant at his company weighed in to say "my application to your company was not even rejected, it was just ignored for months":

https://news.ycombinator.com/item?id=9437516

The comment referred to here is one I made, and let me expand on the incident.

Firstly, the job application that Decade mentioned in that thread was not the coding assignment, but was a resume.

We will not ignore an answer submitted to our coding assignment. Also, we do give feedback to the candidate if we decide not to proceed. We have had candidates who sent in a revised submission which incorporated fixes to our earlier negative feedback and we have been glad to proceed[1].

This meme I see often on HN that people who do hiring are some antagonistic bastards out to get you is simply not true. I am as much a programer as any one of the candidates. I have tried to design the interview process - at least in my part of the organization - to be fair and humane ("Do unto others.." etc) But I also have the other constraint that I have to make the process scale - hence the coding assignment. We are not using the code sent in by candidates in production. We have spent significant time developing the assignments - say, an order of magnitude more man hours than it would take for a typical programer to code up a solution.

I honestly believe that the "interview problem" is an essential complexity caused by the way the industry is set up[2]. I also do believe that a solution to the problem will not be handed down to us from the heavens. Each one of us programers have to help with the solution. Every time we get a chance to conduct an interview show some empathy and respect. Also, teach the professional recruiters working on your hiring team to also show the same. Make it clear that ill-treating any fellow programmer, even one which is not fit for the current role, will permanently ban that recruiter from working with you ever in any capacity. I also welcome efforts like Starfighter, Triplebyte etc from more prominent members of this community.

Now, the aftermath. After that thread, Decade followed up with me and we ended up sending him the coding assignment. Decade completed it and send us back the result, and my colleague's assessment of it was "This is the best code I've seen for this assignment. Let's take the next step". Few days later Decade came in for an onsite interview where he met various members of the team. We made him an offer, which he accepted. I am delighted to report that he started on the job the week before last!

[1] This is very rare, though. We do not hear back from most candidates. Few argue with the feedback, of course.

[2] My thoughts on the topic are at https://news.ycombinator.com/item?id=1964932. The advantages of this set up, IMO, far out-weigh the negatives cause by "interview problem".

I encourage people here on HN to take a look at your footnote [2] and consider it. We talk a lot about how programming has "low barriers to entry", but this is very misleading. It really just means that there are no legal barriers to entry.

I recently interviewed at google, and while I can't talk specifically about questions, I would say that medium to difficult level questions from "Cracking the Coding Interview" are representative of what you must be able to do to a startling (in my opinion) level of efficiency and accuracy. A question like "find the sub matrix with the largest sum in an NxN matrix" is absolutely fair game, at a whiteboard, on the spot.

Think about what it takes to get to this level of competence, where you can pass an exam like this. Granted, google is notoriously tough, but it's pretty common in the industry. And as you point out in your comment, we have to do this over and over.

I read about a guy who passed the bar with 100 hours of study

http://blakemasters.com/post/37113468298/pass-the-ca-bar-exa...

The bar exam (in California) is considered one of those brutally difficult, high barriers to entry that at least you only have to do once (there are training requirements to remain active in the bar).

Now, think about how much time it takes to get up to speed with data structures and algorithms, operating systems, and combinatorics, to the level where you can solve tough problems in 45 minutes to a high level of accuracy with a dry market at a white board, under the additional pressure of an in-person interview. It's hard to even think under those conditions, in my opinion.

I really don't think it's too far off to say that if you do three interviews, at difficult tech companies, you've done something approaching passing the bar.

I've heard a few people here on HN say that while they aren't in favor of a formal barrier to entry like the bar exam, they would happily take an exam like this if it meant that they didn't have to keep doing it over and over. Where they could properly prepare for it, and take it, and pass it for once and for all[1], with a proper study path, under conditions that would be consistent and fair, and get the results back (rather than a mysterious "we don't see a match at this time").

[1] As with many formal credentials, I have no problem with further education requirements to stay up to speed. The problem here is the capriciousness of it all.

I was reacting in surprise, because all I saw was the Jobvite job listing, and its echo listings on the other job boards. I had a much more positive experience with the code exercise.

My experience is that what you hear is true, that you shouldn't spend significant time on job listings. The problem is that every other technique available to me had long odds. Professional network? Non-existent. MeetUps? Most people I met there have similar problems. Lunchcruit or similar gimmicks? No response, not even a lunch. I feel like I just got lucky that mavelikara gave me a chance, and I intend not to waste it.

I've had a different experience on the hiring side of the table. The coding exercise that we provide as part of the interview has been very successful in surfacing the skill and talent of the programmer. By using that screening device, we have assembled a team of very sharp programmers. It's totally worth filtering out people who choose not to spend the time. Even though we end up losing some potential candidates, it's worth it for a 0% chance of a mishire for technical reasons.

A take-home test has a 0% chance of being cheated on? Are you sure you're talking about the same thing here?

Our exercise can't be cheated on because we discuss it with the candidate during the in-person interview and even ask them to change the code in a simple way.

Also, the exercise is designed to be original and so a solution can't be Googled.

We're talking only about the "in their own time" take-home types of tests. The type of cheating I'm talking about is outsourcing the problem or even just having a friend do it for them.

We're currently hiring and for the first time, we should have started doing this years ago, given a simple coding exercise.

I estimate it would/should take any programmer/problem solver no more than 15, maybe 20, minutes to come up with the solution if they have experience in the language we request they solve it in. The exercise doesn't require specific domain knowledge but should quickly assess whether a programmer, particularly with experience in a different language, is able to learn and apply the syntax and semantics of another language. If they are completely unfamiliar with the language then maybe an hour. They can use whatever references they want. We will check for complete copy/pasted code and alter the requirements and/or test data over time.

The other great benefit is we get to see what/if/how they ask questions to clarify the problem and solution.

EDIT: This exercise is geared towards fresh graduates or developers that don't have experience in the language we use. Can it be gamed? Yes. Might some people not bother? Yes. Are we okay with that? Yes.

See a simple one like that where if I need to spend time researching the question will only take up to an hour is one thing. But the last one I got from one of the video game engine companies required building a full crud app with live db, documented api, and multiple feature sets. all for a front end job. Some places just take it too far and sour the idea.

The problem is that you don't actually have a clear measure of who you're rejecting. Every company uses a similar argument to support their (very different in every case) interview process. The fact that it allowed you to build a great team does not mean that you could not have built a better team by giving other options as well.

True, I see your point, but you also can't just hand-wave away our results. I've been on many development teams that did not have a strong technical screen like this, and those teams suffered from having some technically weak members (people would would always be breaking the build, would write unmaintainable code, or would just plod along and always need help). Our team of six has none of those problems: every one of us is a strong and disciplined programmer.

The other benefit of the coding exercise is that it might filter out a certain level of impatience. We don't have coders who can't be bothered to follow [insert team process convention here].

Yeah, I can see that. I did not mean to be disrespectful. I actually generally like take-home coding exercises (and used them at my previous company). I'm just arguing testing a bunch of approaches.

My rule for these - only do these tests in-person, on-site, and after after the company has invested a significant amount of their employees' time to interview you in-person.

Tons of companies hand out these "homework" tests to hundreds of people without having even the slightest intention of hiring them. F* that.

I think it depends on the project. I'm looking to switch roles, and some projects are very small: solve a simple algo question. Should take less than an hour, which is less than the average interview, anyway! So no problem.

Other projects are enormous. 40-80 HRs. of work, easily (write an entire web-app!) I did one of these... did not get the offer. I would not want to complete another.

I feel that if you are asking a company to invest $100k/yr on you, it is not unreasonable for them to ask for a few hours of your time.

The point is that they pay you the $100k for the time you give them as an employee. They don't pay you a cent for the interview project, so you shouldn't give them a single second in return.

They don't pay you for the interview, but that usually costs the company several man-hours worth of work just for the time an applicant is on-site. Should applicants be reimbursing companies for failed interviews?

Maybe. I've always thought it would be interesting to make applicants pay an application fee. This would cut down on people "spraying and praying" with their application, and would lessen the workload for companies. It would also justify spending more time and effort in reading applications, since the company isn't just wasting resources reading bad applications.

Even under the current system, at least the waste is symmetric. I give up a few hours of my time, and the company gives up a few hours of its time. There's equity. The mutual work that the company and applicant do offset each other.

A take-home test model skews that balance in favor of the company. An applicant can spend 10 or more hours working the project. The company can run it through an automated testing suite and have a recruiter spend five minutes looking it over. The system is designed to waste more of the applicant's time and less of the company's.

> Even under the current system, at least the waste is symmetric.

But it's never symmetric. Someone in HR spent time setting up the job posting and managing the process. Someone in management took the time to respond to HR and review your resum, then approve the interview. At least one person at a time is in the interview with you. Then the entire team will spend time afterward breaking it down.

A single hour spent by a bad candidate wastes at least 3 man-hours of work by the company, and most likely more.

> A single hour spent by a bad candidate wastes at least 3 man-hours of work by the company, and most likely more.

So? Companies need employees. They have to do what it takes to get them. If it weren't worth it, they wouldn't do it.

It makes sense to me that more time in aggregate is spent by the company than the candidate, because the company has dedicated recruiting/HR/coordination people that handle the process. I, as an already full-time employed developer, don't have as much time to burn with interviewing.

There is no way in hell that you will get someone to pay an application fee on top of having to burn a vacation or sick day.

That interview costs me at least $500 cash -- my take home pay for the day of vacation I would have to give up. I value my vacation days at rather more than $500 actually, because I have so few of them.

You aren't asking them to invest $100k/yr. You're asking them to bring you in to an on-site interview. A few hours of their time for a few hours of your own is much more equitable.

It's not unreasonable for the company to ask for a few hours of the person they actually end up hiring. It's wasting the time of the dozens of other people that will never get reasonable consideration that's objectionable.

Wasting candidate time should never be used for screening. At most, steps that take a significant amount of time or effort should be reserved solely for determining the order in which the acceptable candidates receive offers, and the amounts of those offers.

And if you're worth $100k/year and they're asking for a day of your time, it's not unreasonable for you to ask for $500 from them for turning up. How many employers do you know who offer that to all of their interview candidates?

Also keep in mind that applying to jobs is frequently throwing your application into a black hole and just praying. Sending a resume into the black hole is one thing. Sending a large codebase or project you worked on as part of the application is entirely different.

I remember a few months ago, I was applying for an internship with a company and they requested that I write a web app to test coding proficiency before they thought about granting an interview. I spent about 12-15 hours writing the thing, deployed the app to Heroku and put the code on Github. I sent a link to the app and the repo to the recruiter. I never got a response of any kind.

Although some companies do try to make the application experience as pleasant as possible for the applicant, the majority of companies don't give a shit about the people who want to work for them, and make no attempt to respect the time of their applicants. Placing greater demands on the applicant is an easy way to shift more of the workload from the recruiter to the applicant. The applicant is the loser in this situation.

My last company had a screening coding problem that was designed to take around a day (although it didn't have to; when I did it as a reference point, it took a couple of hours). I was always kind of astonished how many applicants were willing to do the whole thing, given, as you say, that they often already had a full-time job and/or were applying for multiple positions.

We always gave really extensive feedback on it, partially because one of the things were were looking for was how people would react to that feedback (I was generally surprised at how many people got defensive about fairly objective observations/questions).

Overall I think it was quite a good data point. It was nice to have a more low-key and extensive complement to the more stressful live whiteboard interview (yes, we did have one of those, and I still think it was useful).

When you get a coding assignment, you have no idea if they're going to give honest feedback if you submit a solution, or if they're just going to say "Thanks for applying, but we decided to go in a different direction."

If you gave me a coding assignment that would take a day to complete I just wouldn't do it and move on to the next company. I want to work for a company that respects my time and knows that my code is worth money. A couple hours is all I would ever do.

I agree. I had an interview ask this, and it was essentially solving a problem that one of their clients had presented to them. And they needed it within a week.

It felt very much like they were just exploiting applicants for crowd-sourced ideas.

I often see complaints on HN about the silliness of whiteboarding in programming interviews, would you whiteboarding to a take-home assignment? Are there constraints you would prefer on one or the other? What amount of time spent qualifying for a single job is too much?

I personally pushed to make our interview process more centered around a take-home style assignment, as I feel it's a much better approximation of programming skills than an in-person, high stress whiteboarding session, or a quiz on some programming trivia. We try to keep the assignment tightly constrained ("Here's a broken app with x requirements. Fix it.") in order to keep it to a few hours max.

This I feel has been tremendously helpful for us, as I've never seen someone pass the take-home assignment and subsequently do poorly in the technical portion of our interview. Previously we had many people scrub out of our (admittedly pretty easy) technical interviews. Obviously asking for the take-home has resulted in a few false negatives, but I'd much rather an overly strong pre-interview filter to wasting hours of my entire team's time on a crappy interview.

we do this at my current company. I think it is a great alternative to whiteboard tests as long as it is reasonably short. For mobile devs, we ask them to write a simple app solving a problem they should be familiar with and that we have had to deal with in the past.

I'm sure it's true that a non-trivial number of good candidates will reject a job opportunity when asked to do a work sample. These, however, will generally be those with plenty of attractive alternative offers - the ones well known in the community, with a good network etc.

At the other end of the stick, there are those still trying to break in to the field. They may have moved to a new city, they may have worked in a totally unknown firm, they may have taken an "alternative" path through the educational system (and just to be blatantly obvious, the above describes me, circa 5 years ago) - their CVs don't jump out at employers, and as even phone screens are fairly expensive in engineer time, they are not very likely to be given a chance.

Online coding challenges and take-home tests etc are a quick and cheap way to extend an opportunity to more "risky" candidates. I think that's an important element.

(FTR, the company I ended up with, I got through to primarily via a "Who's hiring" thread on HN, however they did operate a "take home" test that took one hour).

It sounds like your process went ok. But one hour tests are one thing. Projects that may take 10 or more hours are totally different. It really is amazing what some of these companies will demand of their applicants before even granting an interview.

And although this system might be more accessible than traditional recruiting, it is far from perfect. It is the job of the community to demand better and more respectful hiring practices from companies. Paying the applicant for the time they spent working on the project or offering a traditional interview for people who work full time and have families would be a good start.

I think the take-home project can be successful, but not as most companies implement it.

A basic rule of courtesy is that a company shouldn't ask the candidate to invest more time in the process than they are willing to invest.

Most companies will spend an hour or half an hour talking to you on the phone and then give you a project that takes at least 8 hours to do. That's a ridiculous ask. It assumes that the candidate already knows they want to work for your company and is willing to invest that much time to get the job.

They forget that an interview is a 2-way street. The candidate is also deciding if the company is the right fit for them, and being asked to invest a ton of your own time when the company doesn't seem willing to invest theirs is a huge turn off.

There are a lot of advantages to a take-away task - such as being able to write actual code with a computer on your own stack as opposed to just writing on a white board (which some people have problems doing).

However, this type of ask should only come after both the company and the individual have already invested significant and equal amounts of time - enough time for the candidate to be sure (or fairly confident) the company is a place they'd like to work. This means the take-away may only be useful after the on-site, or possibly right before it but after a few rounds where the candidate gets to understand the company better.

Spending that much time to qualify for a single job is too much to ask of anyone

How can you state that as something absolute? I certainly understand your position, but I would be _glad_ to do something like that. My reasons

- I hate white boards and quizzes. I would fail those, probably.

- I'm not interested in travel time, just because. Give me 'homework' and only meet if you're prepared to make me an offer (other issues notwithstanding - it's okay if you reject me afterwards if I show up without wearing pants) without any more bullshit

- I value my free time, but I also love to work on Things™. Any such 'homework' assignment would be a tiny pet project to drag me away from YT/HN/Awesomenauts and force me to learn something that I might not have picked myself

In fact, I even thought about posting an Ask HN sort of thing to offer (limited somehow, day-job and being a dad and all) working for free in exchange for the input. I just haven't done it so far, because of fear of failure/rejection.

So - you state a good point: People might not want to invest the time. But please speak for yourself only, not for the industry. I very much like the idea of presenting work done at home.

That said, I probably wouldn't do both. If someone asks me quizzes first (and I manage to pass for weird reasons, stars align) I wouldn't invest more hours. You, the employer, just wasted hours with crap while you could've gotten ~reasonable~ accurate ideas about my work. If someone asks me to do one of these assignments, invites me to come over and THEN starts the quiz BS I'd probably get up and leave.

Employers: Pick one way to test my skills. My preference: Judge skills in a remote/async process, use the face time to check for a cultural fit / negotiate / talk about the company, position and job: "We invited you after seeing your work, here's why you should work with us, this is the offer and these would be your coworkers - let's chat and do the tour"

Imo, this post did not organize its data and findings into a coherent presentation.

For example...

>The fizzbuzz-style coding problems, however, did not perform as well. While the confidence intervals are large, the current data shows less correlation with interview results. [...] The coding problems were also harder for people to finish. We saw twice the drop off rate on the coding problems as we saw on the quiz.

I read that paragraph several times and I don't understand what he's actually saying. If those candidates "dropped off" on the fizzbuzz, were they also still kept for further evaluation in the following extended coding session? A later paragraph says...

>So we started following up with interviews where we asked people to write code. Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks. Conversely, people who spoke about very trivial sounding projects (or communicated so poorly we had little idea what they had worked on) were among the best at actual programming.

For the fizzbuzz failures to be non-correlative and counterintuitive, it means he did not reject them for failing fizzbuzz and they later ended up doing spectacularly well in the larger coding sessions. If that's what happened, then yes, that is a very counterintuitive result. What were the topics of the larger coding sessions?

I assumed it was just a quick summary, but yeah, I'd love to see more data too.

Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated.

By dropoff I mean people who left during a step, and never logged back into our site.

>Doing well on the fizzbuzz problems was not very correlated.

If you mean "correlation" to only refer to the population that passed fizzbuzz, then it is to be expected that the final positive/accepted interview evaluations don't correlate. Fizzbuzz was never statistically designed for that. It was designed for early rejection and not for predicting ultimate success at the end of a multi-step interview cycle.

>By dropoff I mean people who left during a step, and never logged back into our site.

And the population mentioned in this sentence is what I first interpreted to be included in your "non-correlation". It looks like you don't include this population. The quitters that never logged back in were not further tested by you for later stage evaluations. That's where my confusion was and now it's resolved.

Agreed, it would be more useful to know the false negative rate (i.e. those incorrectly 'rejected' by fizzbuzz). The small / negative correlation could just be caused by a high number of false positives.

Doing well on the fizzbuzz problems was not very correlated.

I think you're mis-using fizbuzz. You cannot really 'do well' on it. You can basically pass it or fail it. Failing means you're probably no good, but passing it doesn't prove anything.

Yeah, I just mean "short programming problems of a difficultly similar to fizzbuzz"

That's the same thing though. If it's the same level of difficulty as fizzbuzz, it serves the same purpose: you can fail at it, but you can't really do well at it. All you can do is not fail.

I appreciate that you guys might not be statisticians, but if you're going to try and analyze data like this, you simply must address survival bias. As it stands, these data are meaningless unless you assume dropouts are completely unrelated to your screening.

You claim doing well on the Fizzbuzz wasn't correlated with interview performance, but you also said "We saw twice the drop off rate on the coding problems as we saw on the quiz."

An alternate explanation for your finding then is, more low-quality candidates drop out of the process when given FizzBuzz, leaving a relatively homogeneous pool of higher-quality candidates for the later interview. This effectively reduces the ratio of meaningful interindividual differences relative to noise, which will reduce the correlations.

In all likelihood, both of your correlations are low, but the idea that coding is less predictive than a quiz could be purely a statistical fluke due to survivor bias.

All candidates did both screens (quiz and fizzbuzz). The correlations were calculated against the same population. Now, I agree that survivor bias could affect the quality of these results (we know nothing about the significant % of people who dropped out). But it's not really possible to solve that problem outside of a lab. I don't think it's an argument to not do analysis. For now we're simply trying to minimize the dropoff rate, and maximize correlation. The quiz was better at both.

Well, the candidates doing both screens is better, but it doesn't totally solve your problems.

It doesn't address the survival bias issue, and when you say a significant percentage dropped out, that's not reassuring. But it's not the case that you need a lab to solve the problem. Even a basic questionnaire of programming ability self-assessment might tell you if there are meaningful differences in the population that quits your process. At the very least, you should understand and talk about survival bias in your article to indicate you're aware of the issue.

Even if you still want to claim a difference between the quiz and coding exercise, you're not yet in the clear. For example, did you counterbalance the order you gave them to people? E.g., if everybody did the quiz first and the fizzbuzz second, that meant they were mentally fresher for the quiz and slightly more tired for the fizzbuzz, which could again create a spurious result. And this definitely doesn't require a lab to test.

Don't misunderstand me, I appreciate your attempts to quantify all this, and I actually think you guys have roughly the correct result (given the limited nature of fizzbuzz-style coding), but when you step into the experimental psych arena, you need to learn how to properly analyze your data. Given that your business is predicated on analyzing the results of how your hires do in the real world, you need to really up your analytical game.

I have to agree with kingmob. It very much sounds like survivor bias. My first reaction is that anyone who drops out due to a test has a high likelihood of dropping out because they can't do the test, which would leave you with a low correlation test when compared with the survivors, but a very high anti correlation with the total population.

I read a blog post a couple years ago by a game programmer/designer who outsources a lot of work through places like odesk/elance. Basically his thing was to weed out the fakers, he'd offer anyone ~5hrs at their bidding rate to finish a predefined programming task expected to take ~5hrs. He says this will usually drop his pool to less than 10 out of the hundreds who may apply, and he can usually use at least one of the people who complete the task. It's hard to say how many of these people go away because the task looks too big, and there's risk of not getting paid, but it's clearly a good filter for him.

As far as measuring this survivor bias, you might gain some insight by randomly altering the order of the testing. You could measure when people tended to drop off. You might even find that people all tend to drop off around the same amount of time, or maybe after some certain amount of effort. It might even be worth paying people to see if that would improve completion rates (while introducing it's own biases).

>> I appreciate that you guys might not be statisticians

and apparently very few on HN in general, seeing how this piece was upvoted.

Google has been studying their process in the last few years and are reaching different conclusions than these guys.

Leave it to the reader to ascertain which group has more validity.

This is not the kind of comment HN needs more of. A better version would (a) drop the snarky putdown, and (b) actually say what Google's conclusions are. Then readers could decide for themselves to what degree those findings contradict these, instead of being told what to think.

Asking the author because I'm curious...

Are you tracking longer-term hiring outcomes too? They'll probably take some time to become meaningful, but they're far more important. The data you've compiled is useful since it helps to filter earlier in the process, but it still presumes that your in-person interviewing process makes the correct decision. If the final filter is letting bad candidates through or screening out good candidates, all the correlations you've found could be reflecting only the ability to pass the interview, not the ability to do the job successfully.

Hopefully you're continuing to follow hires 1, 2, 5 years after being hired to tie it back to the data you collect about the interview process. It would be awesome if you could find predictors of candidates that are likely to quit less than a year after being hired or candidates that will receive less-than-stellar ratings from their managers. By doing this, you'd help hiring managers deal with the blindspots in their hiring, not just streamline the existing process.

"as soon as we started doing them however, I saw a problem. Almost everyone was passing."

Why is that a problem? Maybe almost everyone is decently good (as evidenced by having a string of jobs, and presumably, references), and your interviews are creating tons of false negatives.Or heck, vice versa. You don't know.

You are presuming your conclusions. You have no basis to make conclusions yet, you just have incomplete data. It's iteresting data, and I'm gleefully happy that somebody is looking at this in the context of programmers (too many studies are very broad, across many different career/job type, IMO). But I think all you have right now is data. Fishing for correlations at this point is nearly bound to lead you astray.

With that aside, I'm very interested in the eventual correlation with test performance and job performance. I'm biased - I dislike IQ tests, but I must admit there is a lot of research on them out there. For me personally, I perform spectacularly on this sort of test, pretty poorly in whiteboard tests, so-so in pair program to get a job, and generally top of the heap in actual job performance. It would definitely help me personally if these tests were true. Yet, still, I wonder, do they measure "get things done"? Do they measure "don't piss off the CEO/customer" skills? There's a ton of things that I think are important beyond pure cognitive skills.

If they want to market themselves as being great at screening candidates so that they send only the best to interviews at client companies, then a test that 90% pass isn't of much use to them.

Here's the part that really seems to matter the most.

    This does create some danger of circular reasoning
    (perhaps we're just carefully describing our own biases).
    But we have to start somewhere, and basing our 
    evaluations on how people write actual code seems like a
    good place.The really exciting point comes when we can 
    re-run all this analysis, basing it on actual job
    performance, rather than interview results.

Absolutely. Results on the earlier screens and results on the later interview aren't exactly independent variables, and neither is the one that really seems to matter - subsequent on-the-job success. There are all sorts of biases and confounding factors likely to be shared between them, especially since there's no indication that the later interviews were even done blind w.r.t. the earlier screens. Until then, we're just measuring correlations between different interview techniques, and it should be no surprise that two different kinds of code-focused interviews show the highest correlation.

Author here. We do do the interviews blind to earlier screening results. That's clearly vital. But you're totally right that job performance is the real thing that matters

Cool. Might want to mention that in the article, since I'm unlikely to be the only person who thinks of that.

So, the jury is out until a followup. How to measure productivity without bias? Applying Demarco and Lister's "Coding War Games" approach may work between companies, but how to apply within a single company? What would be the standard error? A dozen new hires is a lot different than 300 interviewees.

What kind of skill set? My previous company contracted with Galye Laakmann McDowell to coach the engineers before an acquisition due diligence. I thought it was unnecessary because we rarely used O() notation in our day-to-day. Since then, I'm designing a machine learning product and find sections in McDowell's book extremely relevant. My takeaway is that it all depends: are you hiring an architect or a hiring a construction crew? How do you measure such different skill sets? Or, how to measure with a job that combines both?

Hi Ammon. I'm certain that we talked on both the 45-minute and 2-hour video conferences, so those are not blind. Do you mean that you and Harj didn't discuss applicants throughout the process?

I got a phone interview with Harj and wasn't considered for anything further.

I'm not sure what kind of hackers they were looking for, but I've been directly involved with creating the infrastructure used in marketing campaigns with the likes of CNN, McDonalds, Infiniti, and more. I've turned an idea into a company with 8 full time employees and have investors seriously interested in one of my side projects. I'm currently involved with leading a project that integrates with a large bank.

I'm a full stack ruby dev learning clojure in my spare time and heavily involved with self improvement. Anyone who watches me for a moment can see that I can solve problems very quickly. I didn't care much about being selected, I have a solid job and offers coming in.

Would anyone who got selected by Triplebyte care to list their credentials/achievements? My main motivation was to see how I compare against others at my current level.

I fell through at that stage as well, which I think was surprising only because their value prop is connecting you to YC startups. I'm a former early employee at a YC startup having contributed to nearly half of its codebase and other YC startups constantly attempt to recruit me.

Oh well, I'm at a non-YC startup now and very happy. I guess the discrepancy lies with Triplebyte selecting engineers that Triplebyte wants, not what is actually representative of what YC startups want. Overall it's still probably a decent signal—you'd have to expect some false negatives here and there (and I'm not a rockstar dev either).

I made it through the phone interview and honestly, you sound like a better candidate than me. Momma always said I should be a lawyer.

I have 8 years experience and talked about my personal minimal JavaScript framework I built.

I didn't pass the next interview, which in my opinion their provided reasoning was kind of atrocious. In fact, I got the "wrong" feedback sent to me, which for the most part I feel I refuted, only to then get a follow up that basically they sent the wrong feedback and I fiddled around to get things working too much for their liking, despite ultimately getting them both running.

Very interesting methodology, but it would be very nice to correlate this data with long-term job performance. Interview decisions (of which of course you get more than long-term results, and they are clearer to quantify) are hopefully, but not necessarily an indicator of whether an employee works out for your company. Otherwise you run the risk of optimizing the quiz/screening process around metrics that influence your interview (i.e. test for how you personally weigh performance indicators, not for how these performance indicators actually affect performance)

I 100% plan to do that. It's a very exciting idea

This is exactly what I came here to say.

It's all well and good that certain things (quiz) correlate more to performing well in a long-form remote coding task.

But part of the problem of hiring is that these test-style code questions are themselves not necessarily good indicators of future job performance.

The only way to truly analyze whether a given technique is working or not is to follow the candidates through into their jobs and see which ones actually become good long term employees.

One of the most enjoyable and mutually effective interviews I had included a one-hour pairing session which was language agnostic that required me to describe how I'd implement a basic data structure. The interviewer drove and implemented. This was then followed by a day long session of pairing on real problems with a number of different interviewers. Lunch was spent with some of the other team members, where we discussed basic things like culture, day-to-day affairs, and each of our histories.

This was a great approach to me, because it didn't particularly focus on anything outside of the present. We worked on solving real problems, and contributing to the project. It's a great, low-stress, method of gauging if someone has the chops for what is typically the "day-to-day" life at the given shop.

> In our first 30 days, we've come up with a replacement for resume screens, and shown that it works well.

What's the metric that shows it works well?

> The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results.

And how precisely do you measure job performance? If this is achievable, I've got a line of companies out my door that would love to pay for a service that systematically measures job performance.

Measuring job performance is hard. A good analogy is measuring intelligence. The IQ test is clearly bullshit (it does not come close to reflecting the complexity and high dimensionality of human intelligence). That said, it's a useful research tool. Across a large number of people in aggregate, it can be used to learn things (like that leaded gas was a really bad idea). We'll be doing the same sort of (very imperfect) measurement of job performance.

That is an excellent analogy. (Not being sarcastic.)

I'll be very interested in hearing what your measures are, and how you'll account for things like employees being in a bad environment/having bad bosses. It's a complicated thing to measure.

Are you guys planning on publishing any of this material in a journal or research venue, or will you keep the results to blog posts?

For this purpose, it shouldn't be too hard. You don't need the evaluation results for deciding raises or promotions, and you don't need to compare employees to each other. You just need to determine if the hiring decision was correct. We can do it with a binary score:

0. Oops, shouldn't have hired.

1. AAA would hire again.

I suppose you could make finer distinctions (need to fire this person/we regret hiring but will work on it/meets expectations/exceeds expectations), but my point is that the golden standard for evaluating hiring decisions is whether the hire is adequate or not for the job. I mean, if someone gave you an oracle or a time machine that allowed you to evaluate the employee on the job for a while before hiring them, all your hiring problems would be solved. Now, if you cannot evaluate if someone is good enough for the job, then you have a bigger problem.

I really think technical, or any other kind of hiring is broken. As you showed best indicators are questions (like your quizzes) or by showing "live" you can rather than a pretty CV, certificate or a fancy university name (I'm talking about tech not medicine, construction engineering or others that really require those).

I have been rejected many, many, many times because the first screening (CV check by non-technical recruiter). My last example was at a well know tech startup were I had to hack my way to get noticed in order to get the first interview. The funny thing is that I was the fasted candidate to get hired + I won a company-wide award for my work at the company just 4 months after joining.

I haven't finished a degree because I thought was boring and I was learning things I already taught myself before, but this fact makes my resume go down the list very fast. Because interviewers don't have time to lose and thousands of candidates to check I'm sure they will find very useful the use of technology on getting those good prospects in front of everyone else.

Something I've seen many times at my past jobs is having good technical applicants, some of them are even referred by one team member and are turned down later because culture. I don't know why but engineers and technical people are more likely to fail at those than others. The surprising thing is that they check culture as the last step because those who can run those type of interview are a few and can't become full-time culture keepers. This is an enormous waste of time and resources for the applicant, the interviewers and the company itself.

For people without impressive CVs networking is key. Go to conferences, hackathons, and meetups to meet people. This will get your CV past HR almost always.

>Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks.

This indicates to me that either the "simple programming tasks" are not well-designed, or the the discussion about the past projects was not long enough. It still sounds like this interview process is only identifying candidates who are good at coding while someone is watching over their shoulder.

However, what I find to be the bigger issue with this article is that "success" is considered to be "passed the interview". Ultimately, all this article tells us are what currently correlates with qualities Triplebyte likes to see in candidates, not what correlates with good hires. To be fair, they do mention this at the end of the article.

The skill of the interviewer in interviewing candidates should correlate just as strongly with everything else. How are you capturing that? I ask because our #1 determinant after an at home code sample is the #3 thing you find are not predictive.

Are they asking critical questions on what decisions and trade-offs were made? Their past projects, can they explain well the reasoning for choice of tools used? Can they talk about what types of improvements they wanted to see in the pipeline process of build-test-deploy?

I'm just surprised that this question is singled out as "poor."

Preamble: not middlebrow dismissal, I really like what they're doing here and will pay attention to them going forward. This is just picking a nit that my hypersensitive self just can't resist:

"Fizz buzz style coding problems are less predictive of ability to do well in a programming interview"

I'm sure this is 100% true, but I thought the point of fizzbuzz-type problems were to weed out people who couldn't program at all? It's not to identify good programmers or even competent ones, it's to identify blatantly incompetent ones, which are surprisingly common even when hiring in SV.

I've never personally asked fizzbuzz when interviewing because my company's hiring process seems to do well enough to not require it. However, based on what I read here it's also very good for filtering out narcissistic divas (i.e., the occasional HN posters who pop a monocle when they get asked fizzbuzz: "how dare someone ask a dumb question that is beneath me?!? Needless to say, I walked out of the interview immediately! Harrumph!").

Maybe Triplebyte's article is using the term "fizzbuzz-type problem" to refer to any contrived programming problem, but in common usage fizzbuzz-type problems are bozo filters that serve no higher purpose than filtering out bozos.

Yeah, we're just using it to mean a asking a simple programming problem (of fizzbuzz difficulty).

I just can't imagine any hiring process in which someone could fail fizzbuzz (or something of that level of difficulty) and still be considered a good fit for any sort of programming job. Failing fizzbuzz means someone can't program professionally, but passing it doesn't mean anything other than meeting the lowest possible standard. Likewise for anything of that same level of difficulty.

But if you're not evaluating the ones who fail, then what's the point of it? you're just grading for style points?

What I take away from your article is that being able to solve a FizzBuzz-style problem isn't correlated with being able to do a more intensive programming task later, which surprised me. What I take away from your comments here is that among people who were able to solve a FizzBuzz-style problem, some variety of style point scoring isn't correlated with being able to do a more intensive programming task, which... okay, that's totally plausible, and much less surprising/interesting.

It would be interesting to separate out the sensitivity (how often does the test give a positive result on good candidates?) from specificity (how often does the test give a negative result on bad candidates?) for each of these tests. I'd guess a fizz-buzz-style test has good sensitivity but terrible specificity.

I've done over 1000 interviews and my experience agrees with their findings that talking about a project is not a good predictor of coding. I usually left detailed resume questions for the end since so many candidates would bomb the coding part of the interview.

I'm surprised they didn't get stronger results from fizz buzz, but I noticed among the candidates I saw that the percentage of 'non-coders' is substantial but not a majority.

One thing missing from this investigation is a measure of solution quality. A good portion of candidates who actually finished coding questions with me ended without thoroughly understanding how their code worked and/or had code that would be hard to maintain. Other candidates would write top-notch code but were unable to explain their thought process to some extent. These are critical pieces to the interview that contribute much more 'color' than 'score' and are important to note.

I really don't mean to cast aspersions on your interviewing abilities, but the common factor in all of the interviews you've conducted is you, and how you talk to a candidate abut prior projects can really matter. At one extreme you have "tell me about your last project" free-form. At the other you have highly-specific drilling down on a relevant technical issue (e.g. flow-control mechanisms in Ethernet networks) or behavioral characteristic (e.g. openness to collaboration). Which of these have you tried? Have you observed any difference in effectiveness between them? My own experience, from interviewing people who conduct interviews themselves, is that people who say "asking about projects doesn't work" just don't know how to do it right. Again, I'm not saying that applies to you personally, but what information can you provide to refute the observation?

I totally agree with you that the "ask about projects" question requires care. And, yes, I did see differing results depending on how the discussion went. When I did the question, I've run it as both a 1-minute quickie and a 15-minute conversation. I tried my best to dig an solid technical exposition out of the candidate if I could.

Typical cases where the project discussion was not helpful:

* Senior engineers could often talk in great detail about things they'd built, but then would most often bomb coding questions.

* A lot of MS/PhD-level new grads could give great talks (about almost anything) but again would bomb coding stuff.

* Lots of ESL candidates struggled tremendously to convey basic project details, but nevertheless were solid coders. From my TAing experience in grad school, I noticed that there are a lot of ESL CS students who have poor oral communication skills but are orders of magnitude stronger at written communication. (This made strong textbooks and written references crucial to the courses I TAed).

* Sadly we hired quiet a few senior / PhD-level people who could give great talks, were highly responsible, but were eventually fired for poor coding. (There were also some management disasters associated with those cases, but stronger coding skills would have helped them nevertheless).

Interviewing has both subjective and 'objective' parts. The most objective data one can collect are things like completion times, the actual code written, and (perhaps) raw contextual data like a measure of the candidate's mood (were they sweating like mad?). Project discussions rarely recover solid objective data. So, I'd definitely recommend against outsourcing those questions to a service like Triplebyte, and junior employees should probably also not be spending 15 mins on projects for /every/ candidate.

The only time I've found talking about a project to be a good predictor is when it's a type of project or tech stack that I'm familiar with. In that case, I can ask very specific probing questions about problems I'd encountered with that problem or tech. In answering these questions whether and how often they talk about specific solutions they came up with versus saying something along the lines of "well I dunno, I wasn't responsible for that piece" or "I'm not sure about the details, I was more involved with the higher level decision making" can be very telling.

It seems like they're only evaluating phone screen methods against their pre-designed coding interview problem? But what if there are issues with that problem?

There seems to be a big assumption that "our programming questions are going to be good and predictive, even if everyone else's are bad." What if being able to describe in-depth a past (real) project correlates just as well (or better) to on-the-job performance as being able to design and code one of their artificial ones? Or what if those artificial ones just don't correlate that well with on-the-job performance in the first place?

It is definitely harder to BS-detect/grade, though.

They want to re-run against actual job performance in the future, that's nice, but it seems like they're throwing ideas out awfully early, then.

while this process feels like it's hitting the sweet spot for finding out who can write brilliant code, that's half or less of the battle in hiring people. personally (and as a hiring manager), i feel like a majority of the hiring process is dependent (obviously) on the environment you're hiring into.

hiring for that small startup? you'll want multi-hat wearing people first, brilliant programmers second.

hiring for a large enterprise team? you'll want to hire for "plays well with others" first, and brilliant programmers second.

that's not to say you should hire schleps, for sure. they should at least be competent programmers. i guess what i'm saying is (despite how it sounds), hiring someone who can program brilliantly is important, but not as important as hiring someone who can navigate your company's software-making requirements successfully.

firing the brilliant engineer who thinks he's more talented than everyone else in the small company so he keeps demanding to be put in charge? yup, that's a thing. firing the brilliant engineer who fights tooth-and-nail over some inconsequential feature the product team wants to change? that's a thing too. assigning a brilliant engineer to crap, meaningless work because no one else on the team wants to work with them? yuppers -- seen it.

in any organization, you are either the only one in charge or you're following someone else's orders -- both of which require different aspects around working well with others.

Hear hear, the "no assholes" rule is always good to apply, regardless of programmer brilliance.

> hiring for a large enterprise team? you'll want to hire for "plays well with others" first, and brilliant programmers second.

Aka, the reason why virtually all enterprise software is pure shit.

Neat idea, especially for someone like me who's trying to get out of support and into development.

From Triplebytes' FAQ:

"When do I meet companies?

If we decided to work together after giving you feedback post our technical interviews, we'll start introducing you to the companies and guiding you through their hiring process."

So, just to be clear, first you quiz/screenshare/interview with Triplebyte, and then you still have to go through each company's search process? Or do companies partner with Triplebyte to fast-track candidates who've already been vetted?

I interviewed with TripleByte and it was different from a normal job interview in some ways, but fundamentally also the same. There is just no culture fit aspect to the interview. Keep in mind that they are not addressing the problem that coding interviews do not predict on-the-job success. They are basically addressing the needs of a broken system.

Correlation between testing and interviewing success tells you nothing about the correlation between either of them and actual work performance.

From the article, in bold:

The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results

How will they gather data about the job performance of the people who did not pass the interview? Without that, the study is only halfway done.

That's true of any hiring system though: how do you gather data on the job performance of the people who don't pass a resume screen? A coffee date? A phone screen? Missing counterfactuals everywhere.

To run a full analysis, you need to hire people randomly - both people who fail and pass your hiring process - and assess their subsequent performance. This never happens in real life.

However, you can still run an informative statistical analysis based on the variability in interview scores and performance scores. For example, the people who scored 5/5 on the interview should perform better than the ones who scored 4/5.

They said they would be correlating that at a later time. This is the beginning of what will hopefully be a much larger study.

Even so, an automated quiz that could predict with high confidence that someone would fail an interview would be a big win, saving both interviewers and interviewees a lot of time.

It's very nice to see hiring advice based on data rather than anecdotes. But I wonder if the process described in the article is pre-selecting for people who are out of work and desperate, rather than currently employed and casually looking for something better.

From the article: > Our process has four steps:

> 1. Online technical screen.

> 2. 15-minute phone call discussing a technical project.

> 3. 45-minute screen share interview where the candidate writes code.

> 4. 2-hour screen share where they do a larger coding project.

Then later:

> ...we can't afford to send people we're unsure about to companies

Does every applicant in this system really have to go through four rounds of screening before even talking to someone who works at the actual company? I can't imagine doing that unless I was desperate.

>> ...we can't afford to send people we're unsure about to companies

>Does every applicant in this system really have to go through four rounds of screening before even talking to someone who works at the actual company? I can't imagine doing that unless I was desperate.

And from the candidates' viewpoint: That headhunter made me waste 3 hours on screening and didn't even get me a phone interview. Why am I wasting time on this?

I realized another problem. You're trying to predict WHICH PEOPLE WILL GET HIRED BY YOUR CLIENT. That's a completely different outcome than trying to pick the people who are the best workers. If the employer's process is defective, your pre-screening is just reinforcing that bias (albeit improving your "efficiency" as headhunters).

Wait, how is this telling us anything? They don't seem to be predicting what determines the best candidate by looking at how he/she performed on the job. All this is doing is predicting how well they will do in a different part of the interview. Am I wrong?

I personally hate the "take home test" approach to interviewing. I've had multiple such tests that take anywhere from 10-25 hours to complete because simply answering the question isn't enough; you need to give textbook correct answers and your code must be formatted perfectly with the requisite comments and documentation. In short, it's pretty similar to an upper-level college course's final exam; however, in college, you can get a good grade with a few mistakes; in interviewing, you get rejected for a few mistakes. I'm done giving a company 15 hours of my time just to get to a first interview; this is arrogant, condescending, and completely devalues my time.

The reality of hiring is you're going to make mistakes, like every other part of running a business. Even in an extended "interview" such as dating for a potential life partner, people make mistakes so I'm not sure how the hiring process can be quantified to remove said error. The interview process is so excruciating these days I often hate the companies I'm talking with.

While we're at it, the skills requirements listed with jobs today are astounding. My experience is that a company wants to hire a programmer with at least a journeymen's level of expertise in 6-8 skills. If you have 5 and are comfortable you can learn the other 3, you're dead in the water. Let's be honest, the latest Javascript framework isn't that complicated. The latest NoSQL database isn't that hard to learn.

The truly hard parts of joining a new company are learning how projects are managed, getting the political lay of the land, finding a sherpa to answer your questions in the first couple of weeks, and learning where you fit within the organization.

If only there was some way that people could spend a couple of years proving they had basic competence, so they didn't have to prove basic skills on every interview.

Why did I get a CS degree if every interview starts with the assumption that I'm an unqualified loser?

Because, unfortunately, a lot of people receive CS degrees but are completely technically incompetent.

Because lots of people with CS degrees don't have basic programming competence?

Why are those people getting CS degrees? Shouldn't they get flunked out?

When schools allow students to do coding projects in groups, it's possible to graduate with ZERO ability, provided you can find someone competent to partner with.

Right, exactly. That's what I'm saying, I don't know what you're disagreeing with. Having a CS degree does not guarantee programming competence. Some people get CS degrees who shouldn't get CS degrees. Some students probably should get flunked out who don't flunk out. Some students graduate from perfectly fine colleges with good grades but zero ability. It happens, hence fizzbuzz. Some people are just naturally good at marketing themselves, and show up with multiple degrees from top-tier schools, vast lists of publications, with years (if not decades) of industry experience on their resumes in important-sounding positions doing impressive and difficult-sounding tasks, and can't code their way out of a paper bag. Hence fizzbuzz.

Why are universities graduating people who can't pass programming 101? Why are they so grossly negligent? They diluted the value of my degree, by graduating too many clueless people.

I did lots of problem sets, homework, and coding projects in school. Why do I have to do it again FOR EACH JOB INTERVIEW? Especially since the projects are usually less on-topic than the ones I did in school.

I.e., I do a homework project for a class, and if I don't get an A, I usually get a reason why not. I do a coding interview project, it goes into a black hole and I never hear from them again.

I think it's because CS and software development are different things with some overlap if you Venn diagrammed it. It doesn't necessarily mean the university is negligent or incompetent. It means that what constitutes the content of a good CS education doesn't map to in-the-trenches software development. It also probably means the schools reward and incentivize different things than a software development firm does.

The blog post kindly shared here reports on a startup's experiments with offering a new kind of hiring screening service. "We launched Triplebyte one month ago, with the goal of improving the way programmers are hired. . . . Well, a little over a month has now passed. In the last 30 days, we've done 300 interviews. We've started to put our ideas into practice, to see what works and what doesn't, and to iterate on our process." They are currently validating what they are doing on the first steps just against what happens at the later steps in their process: "For now, we're evaluating all of our experiments against our final round interview decisions. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases)."

I agree with the blog post author that current hiring processes mostly show that "too many companies are content to do what they've always done." And the idea of a standardized, automated quiz of programming knowledge sounds interesting. But what has to happen next is to an actual validation study and find out if programmers hired by this process do better as programmers in actual workplaces than programmers hired by some other process.

Regular readers of HN are aware that I have a FAQ post about this topic of company hiring procedures.[1] Company hiring procedures are the research focus of industrial and organizational psychologists, who have almost a century of research to look back on with long-term, large n studies to provide data on what works and what doesn't work for hiring capable workers. It's a shame that most company human resource departments ignore basically all of that research.

[1] https://news.ycombinator.com/item?id=4613543

"The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results"

I'm not sure how this gets around the circularity arguments though, since you never get to evaluate the job performance of someone you selected out already. Only the tiny fraction of coders that make it past the initial test get evaluated, which could serve to reinforce the potential biases rather than ameliorate them.

The one case in which this would work is if they hired a number of coders that didn't work out well, and could add or update a feature as a negative predictor of job success.

I'm assuming that they're not at the scale of a larger company with thousands of engineers, and that the observations going into a regression model are relatively sparse. If this is a startup with a 20 hires, I'd be surprised if there was much to do to refine the model after a round or two of evaluations, but would be excited to learn otherwise.

Lack of insight into false negatives is (I think) the major flaw in most studies of hiring engineers. For the analysis in this blog post, we just passed everyone thorough the screening, to avoid the issue. Going forward, we'll be passing 10% of those who fail the screening, to be able to track our false negative rate. My dream is to reach a scale where can do the same thing with the final hiring decisions (well, less than 10%, clearly). Someone needs to do this.

The chart is kind of dumb IMHO. It's nice that they show correlation between different factors and their final hiring decisions, that doesn't indicate the actual quality of the people hired. All it does is reveal an unwritten formula behind their hiring decisions. In other words, the time to complete a programming task is being ignored in their decision process and so should be removed from the process. That's independent of how well it indicates a good candidate. Or am I missing something?

Most of the problems of coding interviews can be solved by good reference checking. Code problems in isolation will never expose how a developer will do when faced with real-world issues, users, QA, and teammates.

Why are people so afraid to pick up the phone and talk to references? I'm always happy to give out my references, and always delighted to talk about the good devs I've worked with, with specifics about what they've done.

Standardized tests don't work for schools and don't work for jobs.

> Why are people so afraid to pick up the phone and talk to references?

First, references are chosen by the candidate, so they are almost universally expected to say good things.

Second, many companies explicitly prohibit their employees from giving meaningful references about former employees. Relying on a process that needs people to violate company policies or to have moved on from the role wherein they worked with the candidate is not really a good idea.

Not to mention that in the industry I am in people will tell you the exact opposite about people. It makes sense, you want your worst people to work for competitors, and your best people to not.

I found the article very interesting, but it seems to me that the metrics are the wrong ones. You cannot treat hire and no hire outcomes equally. Let's not forget that the goal of the interview process is to actually hire people. A successful hire is worth millions, a correct no-hire decision is just avoiding further losses of time and money. On the other hand, in terms of efficiency, a candidate that wasn't rejected early is the most expensive error. Well, not as expensive as actually hiring the wrong person, but it usually takes months to figure that out. Whereas the fact that a whole man-day was spent interviewing the wrong person on site becomes obvious in a few hours.

Here are the important metrics in my opinion, in order of decreasing importance:

- How many people were hired? Or what percentage of positions were filled? How long does it take to fill a position? Nothing in the article mentioned how many people were actually hired.

- False positives (people making it to the most expensive stage of the interview, typically a day-long on site interview, and being rejected there). What percentage of people that went to on site interviews got offers? Personally, I have always advocated processes that eliminate as many false positives as possible, even if it comes at the cost of some false negatives. Of course, you have to be careful not to filter out people too aggressively, because then you're just not going to hire anyone.

- False negatives (incorrectly rejecting good candidates early). By definition that's impossible to measure exactly. However, if you are not hiring fast enough, then maybe you have a problem with your screening process. At this point you could do an experiment and relax the screening process for half of the candidates and see what happens. But it could be just a sourcing problem, that is, you are not getting good candidates in the pipeline to begin with. It's very hard to tell whether you are being too harsh and not believing enough in people's abilities (or not willing to develop talent), or you are just not attractive to the kind of people that you want to hire.

Of course, all of the above is from the employer's point of view. If you are also trying to provide job seekers with a good service, then you can devise other metrics for success and for efficiency from their perspective.

(edited for formatting)

Maybe I glanced over it but who did the candidates interview for? Were those just simulated interviews or are they working with companies that actually hire people? Or are they hiring for themselves? I'm very confused by what "success" means. The last paragraph indicates that it wasn't actually used for a hiring process? I see a commenter on their site has the exact same reaction.

Either way I applaud every afford to improve the hiring process. However I'm a tad bit skeptical. They should release the dataset (unless I missed it) because it's pretty convenient that the results seem to indicate that hiring can be improved by a quiz which they could build and sell.

I'd be interested in the following screening filter: Have a programmer at the company read through the projects the candidates supplied (as a replacement to "read CV") and then come to a conclusion of yes/no. No projects = no job offer by default. You can always think of a different approach for people with no projects if you feel like you should hire from that group. Possibly have multiple programmers read the code and discuss it.

Most candidates can speak well about interesting projects they'd worked on in the past, but a significant percentage of those can't pass a coding test.

Also, not being able to speak well about a past project is highly correlated with doing well on a coding test.

Sounds like one or the other should be thrown out. (Or maybe only the small percentage who do well on both will go on to do well on the job?)

One or other should be thrown out. Or both thrown out. Or neither thrown out.

Maybe only the small percentage who do well on both will go on to do well on the job. Or maybe large percentage will do well on the job. Or maybe a percentage who do bad on both will do well on the job.

In short: there is no data at all about doing well on the job, so we don't know what to throw out and whom to chose.

Where I work, we ask candidates to do a take-home task after they've passed the first in-person interview. The take-home task is trivial, something most developers with a few years experience should be able to knock out in less than an hour. All components of it are easily googleable.

The advantage is that we're building a corpus of solutions to the same problem that we can compare against each other, which is interesting. More importantly, we're building a corpus of solutions that we can then pick from to have the candidate analyze in-person, and talk us through what they see, what they'd do differently, what they like/don't like, etc.

In short, we familiarize them with the problem via their own answer, and then ask them to analyze someone else's (anonymized) answer. Our sample set so far is too small to draw definitive conclusions from, but it feels better than our old ways of doing things.

This is a noble project but I'm concerned that it may not be a methodologically valid blind study.

Specifically, in order to be a blind study of the relationship between the screening exam and technical interview performance, the technical interviewers should not know the results of the screening exam before they make their decision. While they do not state this clearly, it seems possible that since the same 2 people were conducting all steps themselves, that they were not properly blinded.

Thus we cannot rule out confirmation bias in the interviewers themselves, i.e. that they were impressed by good performance on the programming quiz, not that it was an independent predictor of good performance in the technical interview.

Now, maybe one person did the screening and the other did the technical interview with no information sharing in every case, but this would need to be clarified.

Many fields require serious professional certification. You can't become a doctor unless you go through a board certification that includes simulated patient interaction. Likewise, you can't build a bridge or a dam without engineering certification.

IMO, this article demonstrates the need to certify software engineers; using a process similar to the interviewing process described. Therefore, when hiring, we can skip most of the "do you know how to code" and get down to cultural fit and mutual interest.

Professional licencing processes establish minimum standards of competence, training and education. They do not guarantee excellence and a license represents that its holder is no worse than the worst legally allowable practitioner. Half of all surgeons are below average. You don't want one of them if you have a choice.

Depends on what the average and standard deviation are. If the average is shitty in the absolute sense, I probably want one well above average. If the average is stellar, I probably don;t care as long as I am not getting bottom of the barrel.

And if you had a choice between a below-average licensed surgeon and an unlicensed surgeon?

There's no process that can guarantee above-average practitioners for everyone, it's a paradox. That doesn't mean minimum standards are pointless.

Half are below the average of practicing surgeons. But the average surgeon is much better because you weed out a lot of incompetent people who can't get certified.

Surgeons are not number 10 heating oil commodities. The difference in quality between one surgeon and the next matters when it comes to an operation's outcome. Likewise - and this is the gist of my comment - licensing software engineers might raise the minimum bar, but it won't make hiring any easier because licensing doesn't establish relevant expertise or pertinent experience. Obstetricians not thoracic surgeons are what you want for Cesareans just as you don't want a J programmer writing your CMS.

What do you mean licensing doesn't establish experience or expertise? That depends entirely on what you use to qualify people for licenses.

Naturally, you won't hire a heart surgeon to do a brain transplant, but then you don't have a generic "surgeon" license, you have separate "brain surgeon" or "heart surgeon" licenses. You get a Cesarean from someone licensed to do it, and that won't be a thoracic surgeon.

And it _would_ make hiring easier, because you have a much smaller pool of candidates to look at. The problem with licensing is that you end up having to pay for a license, and technology changes to fast to keep licensing relevant. And so much of tech education is expected to be done without guidance or experience.

Again the license means a person is minimally qualified. It doesn't mean they are good at whatever they are licensed to do. As a licensed architect in real life, I am minimally qualified under the law to design a State Penitentiary. I lack the experience and expertise [and inclination] to do it well in accord with contemporary standards...i.e I'm not qualified to do that type of project beyond a general legal minimum for practicing architecture.

So when there's a new penitentiary in the works, there's an RFQ for design services.

This is interesting, but are they using "performance on a sample coding task" as the truth? If we knew that was a good proxy for being a good hire, we'd already be done.

I've had mixed reaction to this take home exercise. I've done that some 4 times and 3 of them resulted in getting called in for an onsite (Amazon). If problem is interesting and there's something to learn. I tend to go for it. But couple of them were lame. Neither it was a good use of my time, nor it was an indicative of my skills.

In near future, I'll be requesting exercises that can be treated and used as a micro library. They are free to use it and I get to post it on github.

> The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results.

Yes, this is when you have something real.

"suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks."

Unsurprising. Turns out talk really is cheap and doesn't indicate one can do. I've even seen people able to maintain jobs over a period of years by talk alone without ever really doing much. Or even being able to do much.

>Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks.

I feel like this should be listed among the least surprising things in the world. Being a good programmer is about MAKING that hard project look easy, by approaching it in the correct way!

There are different kinds of jobs and different kinds of work environments. Grouping all of the situations together and trying to find just a single way of interviewing people flies in face of diversity in human nature. While this research is great and startups in this field are doing well, I see this huge gap that no one so far is trying to address.

I'd really like to see how references correlate with the other factors. Did they not call references?

I should apologize to Triplebyte in the name of those of us, including me, who immediately created fake profiles and took the test just to see if we would do well against this new rubric.

OPEN LETTER ================

Hi, I had a reading of the conclusions you made and I felt as if you defined a process of hiring machines to code rather than humans. So I took a few moments to read your manifesto(https://triplebyte.com/manifesto) (the premise on which your entire conclusion is made), and here is my take on it.

1. /"Whiteboard coding and algorithm questions aren't good predictors of how effective someone will be at writing real code." Whiteboard coding show how someone really thinks. It illustrates the though process of the person and that helps the interviewer to judge him on his rational thinking and his logical approaches. Algorithms add to this by illustrating the problem solving ability. A person may not be able to solve an Algorithm actually, but the attempt on a whiteboards speaks more than his implementation on a online platform.

2./ "Candidates deserve a consistent experience and consistent evaluation". The entire USP of an interview is the diversity which allows the interviewer to judge if someone is able to adapt to new situations and come out of his comfort zone. What you are suggesting is to change the interview process into a GRE exam which will only in-turn develop the culture among developers to prepare for that exam for 2 years.

3./"Hiring decisions should be made using a clear scoring system, not gut feelings" Most of the companies have a 3round or 4 round interview process. It is obvious enough to remove the gut feeling factor. If you wanna argue that it may be possible that a candidate got selected based on the gut feeling of all 4 interviewers then my counter argument is that he is worth being selected if he could generate that gut feeling in so many people.

4./"The hiring process should be focused on discovering strengths, not uncovering weaknesses" Agree to the point. However, the irony is that you are trying to define a particular process to hiring. I wonder if it could actually perform the "discovery" part.

5./"Candidates should be told exactly what to expect in an interview, and be allowed to prepare in advance. " So basically you want to hire the guy who has studied the most over the smartest guy in the room. From my experience, I can surely say if companies like "Google" and "Fb" used to follow that practice, I wouldn't be even writing their name here.

6./"truthful feedback on how they did so they know how they can improve" Agreed. Something that should be adapted by all companies in their recruiting process.

7./"Good programmers come from all types of background." You enforce my point with this statement. Good programmers need not be just people who could quickly write a program for a search in a large of data using hash maps but can also be people who have brilliant problem solving ability and are slow in transforming that into code, or people who are amazing in thinking software design and scalability and probably cannot remember code syntax so well. A company needs a good blend of all these people. Then only a good ecosystem to growth is created rather than just making a team of 10 machines who could transform a pseudo code into Java in 10 minutes.

8./" The software industry needs to experiment more with hiring processes and figure out what really works." I think many are already doing that by doing Tech Hackathons, online challenges, weekend projects, open source contribution references etc. So, not something new which you guys figured out.

9./"Candidates are at a fundamental disadvantage in salary and equity negotiations" Not sure what kind of companies you have surveyed. I think most well known companies maintain clear standards of salary and compensation plan. Though people will surely be flattered reading this. :)

10./"Companies should not have to make recruiting a core competency" Now you are just trying to open the market for yourself by yourself. No comments. :P

Would love to hear your counter arguments. Mail me. :-)

Great. So where can I do this quiz and these small programming tests? :)

Gosh I hope they are controlling for time-of-day, day-of-week, and weather considering how closely packed together those 300 interviews were.

Without any demographic data about who the interviewees were, this is basically useless. Are they all under 30, white and Asian men? Where?

I can see that that would be interesting information to have, but why is it absolutely necessary?

Fresh graduates are often better at programming / algorithm interviews than older experienced programmers who have been out of the interviewing game for a while. It's actually pretty important to know which demographic of users this data applies to.

Another approach: http://ortask.com/a-better-way-to-hire-developers-and-tester...

Reply on news.ycombinator.com