Regarding [2], dang, I am working on exactly this! I mean, it's not that novel of a technique once you start controlling the sampling process directly, but you beat me to the punch.

This technique generalizes to pretty much any grammar one can specify. I weakly hypothesize that by making it impossible for the LM to output syntactically invalid text, the model's task performance improves not just because all of its outputs are valid, but also because part of the model's "processing power" gets "rerouted" from trying to understand and follow the grammar it's writing, to applying improved reasoning overall.