The youtube algorithm is very aggressively trying to get you to watch dumb shit, and you are correct that there is no way to opt-out.

It's best at present to not engage with recommendations at all, and instead start following the RSS feeds of the channels you enjoy.