How can you say that wealth is not a zero sum game when resources are finite?

It's bad because wealth provides power, and power is currently being wielded to take away the autonomy of those who are poorer. Economic means are used by the obscenely wealthy to do some pretty horrid things, from buying elections to covering up their child sweatshop operations. Wealth inequality rapidly leads to general inequality in the eyes of the state.