If you're a top manager, a founder or C-level type, you still have to deal with loneliness (since the power you have over the people below you in the org chart inevitably crimps any relationships you form with them), but you at least get the compensating benefit of being able, to some degree, to determine your own destiny: you make decisions, the company pivots to follow them.
Middle managers don't even get that; they're sandwiched between the people above them, who actually get to decide what the "company line" is, and those below them, who just have to follow it. Middle managers have to not just follow it, but enthusiastically agree with it, even if they think it's stupid or counterproductive. (Or at least they have to do so whenever the people they manage are watching.) Failure to do so, to participate eagerly in whatever fad thinking the people at the top are high on this week, is what I used to hear described when I worked on an Air Force base as "a career-limiting maneuver."
And on top of that, they have to work in an environment where even the limited power they do have to make a difference can be thrown out the window at any time. All it takes is for someone higher up the food chain to take an interest in your team and suddenly all the decisions you've made start getting overridden. Their reasons for doing so can be completely arbitrary, but that doesn't matter. If you're lucky, the person doing the overriding is at least generous enough not to do that to you in front of your team. If you're lucky.
This is part of the reason why the traditional career path of promoting senior developers into middle management is so destructive. The usual critique of that model revolves around programmers often not having the kinds of skills and experiences that make good managers, and that's true. But it's also true that middle management just generally sucks as a job, regardless of who you put in it.