To me, mining has always been about having resources, to buy and maintain hardware, server space, electricity, etc.

What’s the real difference, other than lag time for PoW?

Mining costs money and those with more money can have more miners and more power over the rules.

Is it actually true that stakers can collude to change the consensus rules of the Ethereum network?

Miners/stakers do not make the rules and never have.

Except, that is not how PoS works.