Unfortunately this adds quite a bit of overhead and would make everything take a lot more time. It might be worth it though.
I've been planning forever to do something similar with length, duration, power, etc.
This is called "number sense". I'm pretty sure we do have evidence under searches for that term, it was well-enough known as a concept when I was in school decades ago and is the reason we don't use calculators when first learning math.
For math we ask the whole reasoning and not just the answer, but here what do you do ?
What I resorted to is a live code review with the student, and it becomes very quickly apparent if they know the code, know what the AI said about the code, or didn't bother to read the code at all.
But it's way more time and effort.
It works quite well. I do the math lessons during bath-time daily with my 6 yo. He's up to the point were he can add multiply pretty much any number by 2, 3 or 4 as long as the product is under 3 digits.
Going from adding single digits to multiplication of random 2-digit numbers by 4 with lessons only during bath-time (no paper or whiteboard) gives a child a great deal of confidence with numbers.
Computer science teaching really needs to be prepared to do the same progression as math, but even harsher because LLMs are not even deterministic.
You learn math without a calculator, then you learn how to do the boring parts with a calculator that you would still be able to do (slower) if needed.