The simplest way to think about it I guess is that regardless of wealth, everybody needs a certain number of calories to live. If you are poor and can only afford subsistence, you are paying a much larger portion of your overall wealth just to stay alive.
That may be ok with you, but for a lot of people it's not.
Something like a wealth- or income-based tax credit would possibly correct this, although poor people are also less likely to know how to take advantage of such tax breaks.