Hacker News Reader

Comment by ammon | Hacker News Reader

I mostly focus on consistency in this post (how to make a group of interviewers more consistent). Of course, what actually matters is accuracy (predictive utility of the interview). Obviously those are not the same thing (you could run 100% consistent interviews by just having everyone always grade “strong no”). However, I think (based on running the interview team at Triplebyte) that inconsistency is the primary obstacle to accuracy in practice. So I end up spending the majority of my time focusing on how to make interviewers more consistent.

View on HN

In psychometrics, the field that studies the same thing you're trying to do, the concepts you're referring to as "consistency" and "accuracy" are known as "reliability" and "validity".

It's somewhat striking to me that you seem so worried about these concepts but you don't seem to be aware of the normal terms for them. How much does TripleByte try to inform itself of the existing research in this field? To what extent does TripleByte seek to incorporate psychometric results about what kinds of tests are likely to have high reliability and construct validity?

And one more more specific question:

> what actually matters is accuracy (predictive utility of the interview)

What is it that you're trying to predict? You could be trying to find employees who will be good employees, which would put TripleByte in the business of credentialing, or to find employees who will pass interviews at other companies, which would make TripleByte a recruiting agency. In the past, Harj has been explicit that what TripleByte wants to predict is whether a candidate will successfully pass the hiring process at another company, regardless of how well that hiring process performs. Is this still true?

I’m guessing they know the terms but ‘consistency’ and ‘accuracy’ are just more common easier to understand terms without sacrificing meaning. I’ve talked to some TripleByte engineers and been very impressed by the sophistication of the statistics and experimental methodology they use.

Using an experimental methodology isn't something you should be impressed by in itself. Experimenting means you don't know what you should be doing. That's great if no one knows what you should be doing and you're trying to figure it out; it's less great if everyone else knows what you should be doing, but you don't.

The psychometric literature is pretty robust.

One specific conversation I had was how they read the literature on the best adaptive testing systems and then developed improvements tailored to their specific data and the advantages they had as a real time online test.

Another was on specifically the psychometric literature, and the big meta-analyses of the predictiveness of different testing factors on job performance, and how that influenced the experiments they did early on and how they honed in on what they do today. As well as downsides they discovered of various methods people commonly suggest they're ignoring.

I came away from those conversations extremely impressed with TripleByte's employees and competence as an organization. They definitely think about this stuff.

Well, consistency and fairness might have the added benefit of goodwill towards the company from those folks don't necessarily get hired.

Were you able to measure the outcome of this process? I.e., was there a significant reduction in inconsistency, inaccuracy, or similarity bias after the exercise?

Ses ms reasonable, after all, precision isn't the same thing as accuracy but you can't get accuracy without precision.

Reply on news.ycombinator.com