Yeah, I often read the subtitles before the words are spoken, which sometimes ruins the surprise when they say it.
If it's a comedy, there's not going to be any sort of quiet sounds, so I keep the subs off. If it's an action that usually has gunshots way louder than dialogue, I'm going to adjust the volume to those gunshots and turn the subs on.