It’s the opposite. Being contrarian is free karma, generally. The downvotes come from picking unpopular positions, whether you’re wright or wrong. I used to get downvoted a lot when talking to people about privacy, but now those sorts of posts get upvotes. Times and prevailing opinions change