As I have had it explained to me by someone else when I asked the very same question: People view the number as part of their self worth. If someone downvotes you, you are suppose to care about the fact that an arbitrary number, which has no real meaning in the greater scope of things, went down. For instance, I am suppose to care when a lot of people come and downvote my response to your question.

The only thing I use karma for is to figure out if I have said something factually incorrect or if I have made someone upset. I only get upset about it if someone starts voting me down, when they merely disagree with what I have to say. Negative scores tend to prevent other people from actively engaging in conversation, even if the downvoted post is well thought out and meant to encourage constructive dialogue.