This is the only downside. You really need to curate good content.
I'm quite aggressively removing videos I don't like from my watch history, or flag "don't recommend" channels I know won't be for me. If I'm not careful it'll recommend crap for a while.
Other big complaints include no ability to prevent it from substantially using my cell data despite telling it to do everything over wifi. I've taken to just removing network permissions from the app unless I want to add something.