Yes, consensus does mean that the group "in power" can change the rules in ways that can harm those outside the consensus.
This is kind of scary, but on the other hand I think it won't be exercised in too strong a way - or at least in a way that harms a large number of people. The reason it won't happen is that aggressive moves that harm too many players threaten the whole game. So those in power have to consider whether their actions could ultimately undermine their own value, since the value is agreed upon by a larger market than just the people with consensus.
Now, if you have a pile of nerds with more interest in ego or "correctness" than in financial value, then wild changes can occur. I don't mean to suggest that's always a bad thing either. It's a bit like choosing when to evolve in a backward compatible way or when to make changes that are good for the long term but which annoy or frustrate some people in the short term.
For me, the takeaway of most interesting events in the blockchain world is that it's still quite young. There's a lot to learn, and it will take time to stabilize and become boring and reliable (or at least more predictable).