How about the other way. I define a few test cases and the AI writes code for a generalized solution. Not just code that regurgitates the test cases, but that generalizes well to unseen cases. You'll notice this is simply the machine learning problem restated.

The next step could be to have the AI write code that describes its own reasoning, balancing length of code and precision.