If one person calls you a jackass, ignore him. If two people call you a jackass, think about it. If three people call you a jackass, buy yourself a saddle.
For example, I hate it when people spend a long time deciding where to eat. It drives me crazy. So if it goes on for more than 5 minutes, I'll say "I'm going to (insert restaurant here.) Anyone who wants to join me should come" and leave. Some people love this, some think I'm an arrogant prick, and most are just happy they don't have to wait 45 minutes to eat.
So really, you should ask why people think you're a jackass. I know some people hate my tendency to unilaterally make lunch decisions, but I have no intention of changing it.
I mean, if thousands of people know you, 3 thinking you are a jackass isn't bad.