Heh, dumb idea of the day: do a "flipped classroom" in a different sense -- instead of a student taking a quiz or exam, you provide a clueless AI and the student has to train it on the subject at hand, and then the AI takes the test.

Lest anyone think this is a brilliant idea: an LLM would likely be much better at doing the ~~RLHF~~ RLMF, so you haven't actually gained anything. But I feel like there may still be the kernel of a good idea somewhere in here.

Perhaps your assignment is to iteratively train the AI, which is pre-prompted to seek out every reasonably possible failure mode when it generates solutions. So the learning mechanism is to train an adversarial AI on the subject matter as a way of learning it yourself. This does not solve the problem of cheating with an LLM, it's an alternative pedagogical approach.