In other words, measure LM perplexity on the ground truth words, then on the predicted words, and minimize the difference in perplexities. Ideally with a general model like GPT2 or BERT or something that you aren't using anywhere in your actual ASR.
This may even be more tolerant of errors in the ground truth transcription than raw WER
Exactly. Errors with proper nouns are usually more problematic than errors with stop words, yet they're weighted equally in the WER calculation. Ie, deleting "Bob" and "but" both count as a deletion of the same degree according to WER, but we as humans know that deleting "Bob" is potentially a lot more problematic than deleting "but".
Theory being you don't want to add or remove confusing words, but common stop words are less of an issue.
I'm not sure how this interacts with a multi word replacement, where the new words together make sense but independently make no sense to the LM.