The trick is to actively tell YouTube what you don’t like, it work quite well for me.
And god help you if you watch even one off-topic video - like another comment mentions, YouTube algorithm then concludes you want to see a million more. Reminds me of a Weird Al lyric about TiVo: "I watched Will and Grace one time, one day. Now TiVo thinks I'm gay."