I've done that for a while too.

Often if I want to watch a one-off video topic, I'll view it in private mode, so it doesn't get added to history. Then I don't really mess up the algorithm.