> The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays

This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.