At some point you have to cut your losses; you stay open to persuasion by people who consistently exploit your willingness to listen, you're being played. Downvotes do serve the excellent purpose of saying 'I disagree with this and am not willing to waste further time on it.'
Downvotes can also be abused. But an easy way to get around that is to make the system fully transparent and then do cluster analysis. If a downvoter or group thereof only ever puts out negative votes or habitually downvotes everything fromeone they don't like, that will show up after a while and people can weight away those negative opinions. Absent transparency, then the incentives to game the system go way up, and system operators usually wildly overestimate their own ability to guard against that.