That quip doesn't necessarily mean your direct manager, though. If the company doesn't support employee growth, or is ethically dodgy, those are management decisions.

Anecdotal, but I've definitely had a situation where my team and direct manager were fantastic for two years until the director (my manager's manager) got replaced. The new direcror had a big meeting with all the managers re: "things are going to change around here!" Within a week, there was a palpable change in the office atmosphere and everything started going downhill as the managers were forced to implement policies that I know for a fact they had fought against. It wasn't my manager that was bad; but he was put in a bad position by his own management who clearly had no interest in hearing from the plebs. So yeah ... I did leave because of a bad manager; it just wasn't my direct manager.