Yeah, I totally agree. We actually asked YC Companies about their fire rates. The fire rate at most of the companies was under 6% (documenting a high false negative rate approach)
Precisely. It's a flaw (shortcut) in the human brain, with effects far beyond programming interviews. This is why structuring interview evaluation is so important
The cost of a real work test is much higher, both to the company, and to the applicant. Some companies take that approach (weebly for one), but it means that some of the best programmer will not apply, and it only makes worse the problem of having to aggressively pre-filter (based on credentials, or something else).
You are 100% correct that most research into interviewing has show it to be far less predictive than companies believe. However, I'll add that no one is doing the false negative studies that would really be required to show this. For example, Google has shown that among people who pass their interview GPA is not correlated with success. This does not at all mean that GPA is not correlated with programming ability. In fact, it almost certainly is. People with low GPA who pass google interviews have a lot of other things going for them. To really analyze this, you'd have to give a job to a random sample of applicants. No one has done this. My goal is for Triplebye to get large enough that I can.
Well, they sort of are, just not explicitly. The standard process is to have 2 to 5 interviewers, each asking their own question, and pass everyone who get at least one strong yes, and no strong noes. This ends up with a random factor, but ultimately favoring people at least one real strength. The interviewer who sees this strength will fight for them. We've seen people with high skill variance (great at one thing, bad at others) pass interviews at a higher rater than people who are just OK at everything.
Author here. Ageism is real, and we'll be analyzing it in a future post. But in this case I don't think it's ageism. I think it's big-company-ism. Startups define themselves in opposition to big companies, and take signs of big company culture as strong negatives (which may or may not be justified statistically, I honestly don't know). But there are great programmers at big companies in any case, and startups should figure out how to hire them.
Author of the post. We did not distinguish by experience level in the question that we asked about tech / product. But experience is VERY helpful in a job search, so an experienced ML/AI person would almost certainly have more interest than a product person fresh out of school. We've not seen much weight given to CS PhDs. Someone fresh out of a CS PhD program is viewed much like someone out of a BS program. Industry experience is a big help.
This is definitely true. The filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers among MIT grads than SUNY Potsdam grads. But this fact does nothing to help the good programmers who look bad on paper who everyone is ignoring. This is precisely the problem that we're trying to fix with Triplebyte. We want to make actual technical evaluation (not credential evaluation) cheap and accurate, so we can just interview everyone.
Author of the post. I think that this is exactly right. I don't know what motivated the companies to put that policy in place (they just told us that they had this preference). But I can speculate. There is an epidemic of interest in ML. Four out of 5 college grads we speak to list it as an interest. I think that interest has grown to the point where it's no longer any kind of signal about technical strength, and perhaps a signal that the candidates will not be flexible about what they work on.
Yeah (author of the blog post here), I think it's a powerful idea. I don't know of any other public attempts to do this (most ideas about hiring stay locked up inside companies). I'd love to hear any ideas you have. Email me at ammon@triplebyte.com
Less experienced programmers are definitely over represented in engineers adversely affected by stress. This could be because people get better at interviewing at they gain experience, or because the people who can't perform under interview stress drop out of the profession
This only applies if the take-home interview is obligatory.
I agree that obligatory take-home interviews are unfair (I feel the same about trial periods). They work (they're probably more accurate than a standard interview), but they take too much time from the applicant (and will thus scare away many of the best people).
As an option, however, I totally disagree. There's a significant percentage of good programmers who are ill served by standard interviews. Those people want this.
1 - We don't expect anyone to give up this much time. I realize that it's a big ask. However, a lot of programmers we've spoken too really want this option. Anyone who does not want to is totally free to go with the (faster) regular interview. Either they let is watch them code for a short time, or they code for a longer time on their own.
2 - I can't say for sure (we're only just launching this), but I hope (and think) that talking with someone for 45-minutes about the project that they did will be enough to make sure that they actually did it themselves
Triplebyte founder here. I can't be entirely sure until we give this a try, but the hope is we'll be able to tell if the candidate wrote the code by talking to them for 45-minutes about what they've done.
Yeah, we've been very busy. Sorry about this. We're focusing more on learning and less on throughput right now. We'll have more spots opening up next week. Send me an email at ammon@triplebyte.com, and I'll make sure we get you in.
We got most of the data (more than half) from the founders of the companies directly. The rest we scraped. I bet there are a few on an MS stack, but it's a small percentage
Interesting post. Talking and writing about yourself well, even when not matched with programming well, probably helps get job offers (and in many cases helps be a good employee). It's difficult to tease these two thing apart. I imagine looking at speaking and writing can still miss great people (as can every filter), but I can believe that it's much better than looking at resumes.
Lack of insight into false negatives is (I think) the major flaw in most studies of hiring engineers. For the analysis in this blog post, we just passed everyone thorough the screening, to avoid the issue. Going forward, we'll be passing 10% of those who fail the screening, to be able to track our false negative rate. My dream is to reach a scale where can do the same thing with the final hiring decisions (well, less than 10%, clearly). Someone needs to do this.
Yeah, I can see that. I did not mean to be disrespectful. I actually generally like take-home coding exercises (and used them at my previous company). I'm just arguing testing a bunch of approaches.
I think it's really on the interviewers (and us) to show that people who freeze in tech interviews would also do poorly in a job. It's probably sometimes true, but many times not.
Measuring job performance is hard. A good analogy is measuring intelligence. The IQ test is clearly bullshit (it does not come close to reflecting the complexity and high dimensionality of human intelligence). That said, it's a useful research tool. Across a large number of people in aggregate, it can be used to learn things (like that leaded gas was a really bad idea). We'll be doing the same sort of (very imperfect) measurement of job performance.
All candidates did both screens (quiz and fizzbuzz). The correlations were calculated against the same population. Now, I agree that survivor bias could affect the quality of these results (we know nothing about the significant % of people who dropped out). But it's not really possible to solve that problem outside of a lab. I don't think it's an argument to not do analysis. For now we're simply trying to minimize the dropoff rate, and maximize correlation. The quiz was better at both.