The setups I've seen produce the strongest signal aren't the ones where the candidate can't see the answer that AI produces easily. They're the ones where there is no single answer. Architecture decisions. How you would investigate an outage. How you would tradeoff one constraint against another. The candidate can spend an hour with Claude on it beforehand and it doesn't matter, because the test isn't "did you get the answer," it's "can you talk through the reasoning" and "can I trust your judgement."
Hearing someone explain their thinking on an open problem is a much harder thing to fake. Even if they used an assistant to structure their initial pass, the live unpacking reveals where the thinking is theirs and where it isn't.
Funny because I do the same; I don't commit to doing any work/assessments before at least an honest interview round. Businesses have to understand that _they_ are the ones who have to solve for the filtering problem. Most great candidates that you are hoping to snipe are already in a stable job and do not _have_ to jump through hoops to get past a filter.
I've also flat out refused to proceed with least 5 jobs when recruiters have contacted me, tried to tempt me with a job and then sent through a 3 hour coding test. I've got better things to do with my free time than toy programming puzzles.