I've been a manager for some years. Without sounding too up myself, what made me good at management was the fact that I knew how to do the things I asked people to do. It allowed me to critically evaluate the work, set realistic timelines, and champion the contributions up the chain. I also maintained the ability to perform this work (by design and happenstance sometimes, since we could be understaffed due to sickness etc).

The good managers I saw also had this ability. The managers I didn't enjoy working with / saw struggle were ones who either never had that ability, or had lost it a long time ago.

Note I'm speaking about line management. There's a world of difference between managing a dozen or so individual contributors vs managing senior managers / directors. However, managing LLMs is analogous to line management.