Hacker News Reader

Comment by rahimnathwani | Hacker News Reader

Interviews seek to estimate long-term job performance. Since they cannot measure this directly, they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations. This doesn't mean the interviews are measuring the wrong thing, or that people who cheat on the interviews will meet the performance bar in the actual job.

If someone uses AI to defeat a CAPTCHA, does that mean the CAPTCHA was poorly designed because "the bot is just as valid a user as any human"? Of course not - the CAPTCHA's entire purpose was to establish that correlation between "can solve this" and "is human," and breaking that correlation through automated tools defeats its purpose entirely.

View on HN

First off, yes, actually that's exactly what it means. If bots can solve CAPTCHAS then the CAPTCHAS are poorly designed, that's the whole point.

I don't think this sort of CAPTCHA type step should have a significant place in a good interview process in the first place, if it is important to you to put your applicants through this sort of gauntlet you're not treating them with the dignity they deserve.

If you're just looking to fill cogs in your machine, sure, but in that case you're just as likely to get kids who studied a bunch of interview questions but also don't know shit. I'd just as soon have someone who's good at ChatGPT.

Let me put it a different, perhaps simpler way:

Let's agree on the premise that good human engineers are much better hires than good AI users. If your hiring process fails to distinguish between good human engineers and good AI users that's evidence that your process is not producing good data.

In the exact same way that if your CAPTCHA is getting solved by bots it's a bad CAPTCHA.

Right, so the problem with the interview process isn't that it tests the wrong skills, but that it fails to prevent the secret use of AI such that a human can appear to possess those skills when they do not, in fact, possess them.

It seems there are at least two different perspectives:

A) If a human can use AI to pass a remote interview that they would otherwise fail, then the questions are unsuitable for use in interviews, whether remote or in person.

B) If a human can use AI to pass a remote interview that they would fail if it were face to face, then the questions may be reasonable, but the safeguards for remote interviews are insufficient.

I disagree with A.

The types of questions that an AI gets right are still bad questions to ask in 2024, regardless of being remote or in person.

If you're allowing your programmers to use AI on the job you're not getting particularly useful information if unaided someone fails a question that they would answer correctly with AI assistance. Sure, being able to answer the question in person may still correlate with being a better applicant, but that doesn't mean it's a good question.

Interviews take time, you should make good use of the time by asking questions that tell you a lot. Questions easily answered by AI (this also goes for questions easily answered by studying interview questions by the way) are not good questions. Measure the things that matter.

   (this also goes for questions easily answered by studying interview questions by the way

I agree, but it’s easier said than done.

> they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations

Not necessarily... You just get the people that are technically competent enough and are self-corruptable enough to pass the test. There is still a correlation with performance but it is a weaker signal and you are more likely to miss out on people with equal ability but higher morals.

Reply on news.ycombinator.com