Assuming you're doing local inference, have you tried setting a token filter on the model?