The power imbalance will ensure that this is unlikely to happen, in much the same way that people generally don’t trust that massive corporate entities have their best interest at heart but might have some regard for their colleagues wellbeing.

How so? Each human is a different "model", and its constrained to the physical world. What are we going to do? Put whole corporations in jail, shareholders included? Limit what they can do?

The "it went sideways" scenario for 100k agents spawned across the world using the same bad model is completely different from humans going sideways.