I do similar to you. I wonder if not rating is limiting your gains relative to giving (gasps) four or three star ratings.
I would prefer not to harm the driver, unless he was a dick or was driving a garbage can, I would then rate it 1-star with a clear conscience.