If you two were friends, this might come across as playful ribbing ('taking the piss' as they say in the UK). But since I don't think you are friends, it comes across as mean, on your part. And you demonstrate (usefully, I think) the mistake of confusing complex and/or erudite speech/vocab with arrogance.

But actually, I think the "attribution of arrogance" is a post hoc rationalization. In truth, some people feel utterly threatened by complex, erudite speech. It is hard to be confronted with a person who is having a thought that, without a lifetime of reading with intellectual curiosity (and a good memory), cannot be fully appreciated.

Speaking intelligently can be (often is) like serving a plate of delicate, complex food in front of someone who isn't interested in this "pansy bullshit" and just wants a hot dog. And it hurts the same way, because what you prepared for them really is amazing, and not only is the effort not appreciated, but the recipient interprets the offer as an insult. This is a very painful experience, and you learn to take great care to learn who is expecting a gourmet conversation and who is not.

The insecurity is strong with this one. You're projecting that the person's literary references are "name dropping things in largely irrelevant contexts". There is no basis for this assumption, other than your obvious bias to jump to the conclusion that this person is elitist and goes out of their way to seem elitist. Why? Are you sensitive that you didn't get into a school like MIT? Neither did I, but nothing he is saying comes across as even remotely elitist to me.

You're really just coming across as a jerk.