Let's not forget that companies aren't real sentient beings, do not really take decisions and do not really have their own interest. They are legal fictions. Trying to understand the behavior of a company based on its self interest can therefore be misleading.
Why do bureaucracies develop? Why is Google launching yet another chat app? Why are so many companies disrupted by new techs instead of adapting? All this is against their interest after all.
Companies are ultimately led and controlled by many individuals, in pursuit of their own individual interest (mostly). Some groups of individuals within a company have more control than others, and will consequently bend the company toward what's in their own interest.
I'm not versed into sociology of organisations, but from a cursory look at some big corp internals it seems to me that this easily explains the frequent growth in numbers and importance of the coterie with the more agency over hiring, regardless of whether it's good or bad for the company as a whole, ie the growth of middle management cancer.
It's not that management is dumb, it's that every layer of management is self-interested. Every hierarchy has to contend with principal agent problems. Organizations can do a better or worse job of it, but the incentives are always there and they lead to predictable inefficiencies which are frequently observed. Everyone knows this, but unless you know how to get someone to manage without self-interest, you aren't in a position to solve the problem. Nobody is. That's why it lingers.
You can play semantic games about whether or not it should be called inefficiency -- I don't really care what you call it -- but the RfR / Dilbert dynamic is very real and very prevalent.
Middle management becomes necessary.
But the question is around pay. Should they always make more because they sit higher in the org chart?
I would suggest they make more because of power structures not utility or because they are not as easy to replace.
Because of first class shares, senior leadership shares percentages shareholders with a vote are often the people in power. Common shareholders don't hold much power.