Obviously, the most interesting, useful comments are often going to be controversial. In cases where people downvote stuff they don't like (which is a proxy for, does not fit my worldview), these comments will not get the most points. This means you shouldn't care too much about points except that they tell you where the conventional wisdom lies.
Personally I downvote something if I think it's dumb. Since there is more stupidity here now, it seems, there are more downvotes. That's good.