If your hiring process is broken by people using AI then the process is either not evaluating the right things, or maybe it's fine because realistically speaking you're hiring a dev who's going to be using AI anyway.
If someone uses AI to defeat a CAPTCHA, does that mean the CAPTCHA was poorly designed because "the bot is just as valid a user as any human"? Of course not - the CAPTCHA's entire purpose was to establish that correlation between "can solve this" and "is human," and breaking that correlation through automated tools defeats its purpose entirely.
I don't think this sort of CAPTCHA type step should have a significant place in a good interview process in the first place, if it is important to you to put your applicants through this sort of gauntlet you're not treating them with the dignity they deserve.
If you're just looking to fill cogs in your machine, sure, but in that case you're just as likely to get kids who studied a bunch of interview questions but also don't know shit. I'd just as soon have someone who's good at ChatGPT.
Let me put it a different, perhaps simpler way:
Let's agree on the premise that good human engineers are much better hires than good AI users. If your hiring process fails to distinguish between good human engineers and good AI users that's evidence that your process is not producing good data.
In the exact same way that if your CAPTCHA is getting solved by bots it's a bad CAPTCHA.
It seems there are at least two different perspectives:
A) If a human can use AI to pass a remote interview that they would otherwise fail, then the questions are unsuitable for use in interviews, whether remote or in person.
B) If a human can use AI to pass a remote interview that they would fail if it were face to face, then the questions may be reasonable, but the safeguards for remote interviews are insufficient.
I disagree with A.
If you're allowing your programmers to use AI on the job you're not getting particularly useful information if unaided someone fails a question that they would answer correctly with AI assistance. Sure, being able to answer the question in person may still correlate with being a better applicant, but that doesn't mean it's a good question.
Interviews take time, you should make good use of the time by asking questions that tell you a lot. Questions easily answered by AI (this also goes for questions easily answered by studying interview questions by the way) are not good questions. Measure the things that matter.
(this also goes for questions easily answered by studying interview questions by the way
Not necessarily... You just get the people that are technically competent enough and are self-corruptable enough to pass the test. There is still a correlation with performance but it is a weaker signal and you are more likely to miss out on people with equal ability but higher morals.
In my experience if a problem simple enough to be completed in an hour, it’s simple enough for AI.
I recently had a question where I was just given a bunch of tuples and told "let me know anything you can glean about this data in two hours". I think you learn much more by seeing how someone fails at a complex nebulous task than whether someone can implement a doubly linked list from memory or whatever.
Real life programming is like 80% failing at complex nebulous tasks, picking yourself up and trying again (by volume). Interviews should simulate that. If AI helps you, so be it.
That said, I also don’t know how to adequately test, other than to carefully watch people’s eyes during questions; even that isn’t foolproof.
You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one.
A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!
Currently the cheating mostly aims for coding tasks because they're pretty simple to do and self-contained. Much less for helping with a good dialog.
Over the years I've interviewed many people for technical positions and there are many ways of getting to the core of a prospective employee's competence.
Of course there are questions about training, past experience, qualifications and those about the work you expect them to perform but other matters also enter the equation and they're almost as important (the person's demeanor, confidence, etc.). By letting the person do most of the talking—explaining things about themselves and their work—you can learn a great deal.
Stating the obvious, interviewers ought to be on top of the job they're hiring for but from this story and from other anecdotal info I know that's not always so. In my case as head of the department I was pretty much up to speed on most aspects of the work, so it was clear to me when an applicant was spinning me BS. If they knew more than I did then they almost certainly got the job (life's easier when you've smart employees, for starters one can ask them questions instead of vice versa).
What's that leading up to? Well, when interviewing it was obvious to me (and to the other interviewers) within just the first few minutes whether the person was worth considering. It's uncanny how good one can become at doing this (and my record speaks for itself, I was pretty pleased with those employees I hired).
Let me illustrate with an example: some decades ago I had an applicant apply for an electronics job. He spoke only Chinese and a few broken words in English and none of us on the interview panel spoke Chinese. BTW, he turned up with a Chinese/English dictionary in hand.
I dispelled with many of the questions I usually asked and showed him a somewhat complex circuit diagram with many subsections that performed very different functions (if I showed this at all then it was usually towards the end of an interview).
Within seconds of looking at the diagram he progressed logically through the processes from beginning to the end (note the processes/stages were not drawn in linear progression and consisted of many branches). He did this mainly by pointing.
It was immediately obvious to both my independent technical assessor and me that this guy knew his subject very well. The only point was a minor objection from the nontechnical employment officer who questioned if we'd manage to communicate adequately with someone who didn't speak English. I didn't see that as a significant issue.
He turned out to be an excellent employee who knew his work very well. And within six months his English was good enough not to be an issue.
In summary, if AI is a problem now then it will be more so in the future and that needs to be tackled. That said, I find it hard to accept that an experienced interviewer can't quickly cut to the core even when an interviewee puts up smoke screens and distracting noise.
Like, Popper's falsifiability it doesn't take much to knock a BS or inconsistent argument off its pedestal when one has enough data, refutability usually works well.
That said, one must always keep in mind that just because an applicant can get some things wrong it doesn't mean that he/she isn't the best person for the job. All that tells me is that I have to probe a little deeper.
Edit: asking awkward or trick questions usually ought to be avoided. An interviewee is usually already under considerable stress and can easily be flummoxed when asked them (even if they know the answers they can bind up and not spell them out).
Usually, it's immediately obvious when an interviewee doesn't know an answer to a question or is struggling with it because of stress—body language alone usually makes that abundantly clear. It's then best to gently segue to another question. If the question is really relevant to the job then one can return to it at the end of the interview when the person is more at ease. By then, it's pretty clear to the panel whether the applicant is in the running or not. If not, then do not return to the question.