> Then why don’t companies adopt a flatter structure with higher compensated engineers? The goal of a company is to make money after all.
Let's not forget that companies aren't real sentient beings, do not really take decisions and do not really have their own interest. They are legal fictions. Trying to understand the behavior of a company based on its self interest can therefore be misleading.
Why do bureaucracies develop? Why is Google launching yet another chat app? Why are so many companies disrupted by new techs instead of adapting? All this is against their interest after all.
Companies are ultimately led and controlled by many individuals, in pursuit of their own individual interest (mostly). Some groups of individuals within a company have more control than others, and will consequently bend the company toward what's in their own interest.
I'm not versed into sociology of organisations, but from a cursory look at some big corp internals it seems to me that this easily explains the frequent growth in numbers and importance of the coterie with the more agency over hiring, regardless of whether it's good or bad for the company as a whole, ie the growth of middle management cancer.