I base my decision on a genre I'm watching.

If it's a comedy, there's not going to be any sort of quiet sounds, so I keep the subs off. If it's an action that usually has gunshots way louder than dialogue, I'm going to adjust the volume to those gunshots and turn the subs on.