Here's the way I think they should be used:

Up == agreement

Down == disagreement

Flag == violates community standards

I generally try to add a thoughtful response to anything I downvote, but I am guilty of downvoting and moving on a couple of times.

I think requiring a response to downvote would be better than removing downvoting entirely.