I can believe that the social piece is harder. I am not even vegetarian, but I eat less meat than I did growing up. Yet, I have had my meat-and-potatoes family of origin label me
vegetarian because not all meals contain meat these days.
However, I do think one reason vegetarians, especially vegans, get so much social flack is because they typically position themselves as morally superior to meat eaters. I know one incredibly toxic vegan whom I wish would just make the world a better place by kindly dropping dead. In addition to being high handed, she is inconsistent, hypocritical and generally assholish to the max. She is the vegan version of the hellfire-and-brimstone Christians I knew growing up in the Bible Belt who made me feel like "If my choice is to sit next to you in church or go to hell, I will take hell." On top of all her other BS, she isn't even a committed vegan. She is a convenient vegan who promptly adds cheese to her diet periodically for REASONS, all while looking down her morally superior nose at everyone around her.
If you want to be vegan, cool beans. Good for you. But you might get less social flack as a group if you took a different approach (as a group) to how you talk about that choice and how you treat other people who aren't vegan.