Everyone is an asshole in someone's mind. You'll never please all people, and when you don't, they'll think you're an asshole. That's life. Don't let what they think bother you.
I mean, if thousands of people know you, 3 thinking you are a jackass isn't bad.
For example, I hate it when people spend a long time deciding where to eat. It drives me crazy. So if it goes on for more than 5 minutes, I'll say "I'm going to (insert restaurant here.) Anyone who wants to join me should come" and leave. Some people love this, some think I'm an arrogant prick, and most are just happy they don't have to wait 45 minutes to eat.
So really, you should ask why people think you're a jackass. I know some people hate my tendency to unilaterally make lunch decisions, but I have no intention of changing it.
That's a nice one, have to remember that :-)