You can make the same argument that like/upvote buttons also get misused and abused, but the consequences are not the same. You might disagree with this and justify your opinion by saying that misinformation is dangerous. But that's why you have a brain, so that you can make that decision for yourself! When you censor an opinion by downvoting something, you are making an unfair decision for future viewers of that content who would rather make that decision on their own.
Moderation/censorship of content that doesn't break the law should be illegal.
Edit 1: I should clarify that I wasn't necessarily referring to YouTube here. Likes and dislikes on YouTube are just for measuring engagement and don't reduce access to content. On other platforms, such as HN, hitting the downvote threshold removes it view (or makes it otherwise more difficult to read). To me, this is a weak form of censorship, making it harder to view the content, if not impossible for some people who have poor eyesight or mobility skills. If you have to explicitly toggle on a setting (i.e. showdead) or use JavaScript hacks to make this content visible, I think that's a big problem.
Edit 2: I'd love to continue contributing to the discussion, but ironically and unfortunately, I'm rate limited on this website by the moderators to 5 comments per day. So maybe now you can understand where I'm coming from with my seemingly absurd ideas. I don't even know why I continue to use a website that limits my ability to comment and use it effectively. Maybe I'll request deletion and become a permanent lurker, watching all you model citizens engage from a safe distance.