> If no one is dirt poor, does it really matter if some are much wealthier than others?

Yes, because the wealth distribution has a self-reinforcing inertia, especially over generations. Absent mechanisms for redistribution of personal wealth (e.g. confiscatory inheritance/estate tax over a certain amount) it will only concentrate into a kleptocracy.

Too much inequality is bad, even if the lowest 5% lives very well, because it is not a sustainable equilibrium. On the other hand, too little inequality is also not sustainable or desirable.