I care because it's a measure of whether what I say is agreeable or disagreeable. It's not about fitting in or following the herd, but if I say something that loses me karma I want to understand why. Maybe I'll end up with the same opinion anyway or maybe it teaches me to think more before I talk. Mostly, karma helps me reflect on my opinions and question those opinions.
The only thing I would do with HN is to _force people to comment when they downvote_
If you agree with me, then an upvote with no commentary is fine. If you downvote me, tell me why. Maybe I could learn something, maybe I get the chance to explain myself better, maybe an open disagreement on a point enriches the community.