Surely proof of work also requires trust in a group of miners, trusting that 51% of them don't collude?

As mining complexity grows, surely this problem is only exasperated, since you need a bigger investment to be able to mine effectively, which would presumably result in an even smaller group of more powerful miners?