Yeah, this is how I generally approach it.

I don't care if I agree with someone. I just want them to provide more information and add some value to the discussion. If the information is invalid, I just do what I can to provide counter resources.

I also generally try to approach downvotes the way you're saying as well, though that brings us back to downvotes being "mini flags". I don't know that I disagree with that.

Mostly just bringing up that upvotes and downvotes aren't really defined. They represent the general unspoken opinion of the people voting, which I think is fine.