Inequality is inherently bad for dynamic market economies. Often the argument is that increasing inequality is fine if the economy is growing and the lower classes aren't losing income but inequality also slows growth as more of the investment goes to consumption for fewer and fewer people. So economically I agree that inequality is not zero sum but it seems like inequality lowers the total productivity of the economy in the long term (so it's net-negative sum).

This is in addition there are effects on the civics and reduction in welfare of people. Like many things in economics, inequality is a hard thing to do good experiments on but the data suggest that inequality itself has a host of effects that you're ignoring.

This article is a decent primer on the effect of inequality and what we know and don't know (and also some of the difficulties studying it): https://en.wikipedia.org/wiki/Effects_of_economic_inequality

Edit: there's a lot of literature on this. If you use Google scholar you can find tons of articles talking about the various effects of inequality (though as always with economics, the papers struggle to get good data). Modern economics definitely doesn't view it as an isolated and benign thing.