For instance, whole chickens provide about about 1200 calories/dollar, whereas leafy green vegetables all provide under 100 calories/dollar. That means there is a 12x difference in the amount of resources required per calorie provided. You can go on about the externalities of meat all you want, but a 12x difference almost certainly makes up for them.
So if you switch from eating lots of meat to eating lots of leafy green vegetables and berries, you probably aren't making much of a difference. The better heuristic to follow is to just pay attention to how much your food costs. Less money spent on food = lower environmental impact.
If you're interested, I go pretty deep down this rabbit hole here: http://www.thinkfundamental.org/why-kale-is-actually-terribl...