I would trust my manager to not abuse that power. If he would start using it, I'd first try to change team or, if not possible, just change employer. Hopefully, for the sake of the organisation, people quitting would reflect badly on the manager over time.

The point being, it's much easier for me to manage the relationship with my manager than with some anonymous committee that doesn't even know me. The manager is already important enough that I would not keep him if he seemed more interested in personal agendas than his subordinates.