Ya, I think it's more that people assume that if you choose not to drink, you look down on people who do drink, and especially in one-on-one settings that makes them feel uncomfortable drinking around you
It's pretty easy to spot in people, and I go out of my way to avoid people that are that morally and intellectually stunted, so that may contribute to me seeing less of this reaction.