The problem isn't wealth; it's segregation of the wealthy from the rest. There's a common social trend toward lowering the variance within groups and (as a natural consequence) increasing the variance between groups, and a host of studies assert that this is generally a bad thing by causing long-term instability in society. It certainly causes deep divides that are very easy for demagogues and authoritarians to exploit.