Other pointed out to costs, but I think the issue here might be substitution.
An "organic" apple is a substitute for "GMO" apple; in fact, it's so good a substitute most people from the indifferent majority wouldn't be able to tell organic from regular apart if not for the label. There's plenty of types of products where removing gluten also doesn't affect taste too much. In these cases letting the minority impose their desires is essentially free.
Vegetables are no substitute for meat. If you like meat, if you crave meat, you won't accept a tofu replacement. This makes it more difficult for vegetarianism achieve total domination. However, the effect still manifests itself clearly. Where I live, 15 years ago, vegetarianism was this weird thing from movies that nobody otherwise thought about. Today, every restaurant I can find has vegetarian options available and marked in the card, and whenever friends or I organize a get-together, the number one question is always "do we need a vegetarian menu?".