The same way we did before. Very very simple code submission (most people using AI use it even though we call out we're going to ask them later to modify later without AI tools writing code for them) then pairing interview where we ask some basic "are you actually at the level you say you are" question, then ask them to extend their program submission with:
- engineers on the call as pairing assistants
- google, ai tools, whatever for libraries, syntax, etc. we tell the candidate directly that it's impossible for us to gsther signal on how they think about problems if they ask Claude to just whip them up a solution
- the expected output
- their own unit test suite
Every candidate that has submitted an ai submission thus far has failed because they have literally no idea where to go. I've interviewed dozens at this point. I'm not saying "they're unfamiliar with the structure", I'm saying "they cannot actually break down the problem even verbally".
It doesn't matter their pedigree or past experience on their resume, if they used AI to generate they don't seem to be able to resurrect the skills that actually matter for the thing, engineering and product work.
Note because I know folks hate code submissions. It's not hard. We give a CSV with 3 columns, 10 lines. Do some basic mapping and some structuring, some basic data modeling. We only expect about 1 actual class or struct. Then unit tests and it should run in the terminal. Max submission length with verbosity has been a java program at something like a hundred lines total if that, most folks complete the submission in an hour or two. Extension is that we modify one of the rules and extend the CSV by 5 lines.
I seem to still be getting good signal from this, since the engineers that I've hired off of this have been fantastic with or without AI tooling immediately in their hands during the day.