Pay grades are telling us things we do not wish to hear:
- engineers are more replaceable than their managers
- management staff are consistently difficult to find
Look at the acquihire prices engineers are getting these days...
Your master's degree in management appears to be much less valuable than experience.
The same market that produces for more failing companies that successful ones?
The Market, is not some omnipotent, omniscient force. Much like the forces of natural selection, you actually don't have to be "the best" to survive, just "good enough", and most of the time organizations and organizations survive despite themselves rather than because of themselves.
Or said most succinctly, I think you may have bit too much faith in both the accuracy of market forces and our ability to correctly qualify and quantify such forces.
Markets will never magically produce the socially beneficial outcome. They efficiently allocate resources along lines imposed by incentives on market actors, and the resulting supply and demand. If the demand is insane, so are the market-allocated outcomes.
I didn't make the argument that our current structure is "the best" or that the status quo is perfect. Only that sufficient supply/demand for managers exists vis a vis engineering staff to drive management wages up.
In short: it's clear that managers are more highly-valued.
In software, one source of irrational ideas has been the management models that were developed for industrial and manufacturing organizations. Since building software is a design process, not a manufacturing one, these models don't apply. The industry is slowly figuring that out, but that has left plenty of room for an inefficient market in the meantime—middle managers being perceived as more valuable than programmers and all the rest.
It could well be that good managers are much more common than good engineers, but that the essential problem is that it's hard to find good managers and the consequences of hiring a bad manager are much worse than that of hiring a bad engineer.