I work on a team that lost about 66% of its members in the last year and a half. Every single one of them liked our manager, but the way the organization works at a higher level was making everyone's life miserable. People absolutely leave companies because of the company.

I've also had toxic managers at other organizations, so I agree a manager CAN drive an employee away, but that's not always or even usually the case.