I didn't realize they finally added a way to sort of control this behavior. It's under Languages in the YouTube settings. Similarly there is a control for captions languages under Captions/Subtitles.