If I act like an asshole, then when people get mad at me for being an asshole I'm to blame. If I reject people for not being smarter than I am, then if they get mad at me I'm to blame.

I agree. You are responsible for yourself but nobody else. You seem to imply, however, that others' actions dwell in a void entirely uninspired by your own actions. I hold, on the other hand, that everything influences everything else, and so nerds should be responsible for their social hostility because that's what's influencing their persecution.