You could have the models output a confidence alongside next-token then weight the penalty by the confidence.