Probably partially because there's stuff (think vitamin B12 for the most extreme example) you can't get or can't get easily from a vegan diet, needing (oftentimes expensive) supplementation.

I'm guessing that it's a luxury of choice. Poor people can't chose what to eat.

Except that's obviously wrong because plenty of poor people have strict dietary requirements.