I don't know of any definitive cases where someone made a comment that was appropriate, got vast numbers of down-votes, and then got banned because of it. I know of many people who have complained about being banned, only to discover that it wasn't the case.
I do know of many people who present dissenting views, but present them carefully and constructively, and who do not attract down-votes because of their comments. They work hard to get people to think first, and not to react with a jerk of the knee (so to speak). It is possible. I've seen it. It's not easy, and I suspect not everyone can do it.
In short, it's unclear that your beliefs about the situation are accurate - there are many ways to get these things wrong. I'm not saying you are wrong, I'm just saying that your view about down-votes may not be accurate.
And no, I don't think down-votes should be removed, although I frequently don't understand how they are being used.