The issue isn't their economic power by itself. It is their outsized political power that derives from that economic power. The more political power the oligarcs have, the less power all the other citizens have.