It's because of the principal agent problem.
As organizations grow, people inside it become less and less oriented towards the organizational goal. The rigidity appears to fight that.
But many of our current problems are more directly related to Taylorism and an intentional separation of design from production.
GMs failure to learn from Toyota at the NUMMI plant is a well documented modern example, with Japan intentionally targeting the limits of Taylorism to basically destroy the US industrial market is another.
The centralized command and control model also ignored the finite congitive ability and assumed the relational omniscient actors.
The funny thing is that multiple Nobel prizes have been awarded related to some of the above, and we know that Taylor faked some of his research and most business tasks are not equivalent to loading pig iron onto train cars anyway.
Even TOGAF and ITIL recently made changes after the Feds changed the Klinger act, moving away from this model and every modern military uses mission command vs C2, but management is still teaching the pseudo scientific management school of thought and not addressing the needs modern situations.
The incentive models are largely a reason for this and recent developments like 'impact' scores push things back even more.
You can still have a principal-agent relationship, but delegate and focus on intent and avoid this trap, but it requires trust and bidirectional communication.
Really IMHO, or comes down to plans being safe feeling, high effort and compatible with incentives.
Those plans never survive the real world, because of the actors bounded rationality and knowledge.
A book that is potentially a useful lens into this is 'The art of action', but it is just a lens.
Organization 'goals' are often poorly communicated and useless because 'planning' is insufficient and not durable enough.
Being way past any horizon that can be planned for, actionable concepts of shared intentions and purpose are not communicated.
Toyota gave teams concrete goals to obtain, allowed them to self organize and choose how to deliver.
GM meticulously copied what those teams had done and forced Detroit teams to follow those procedures and it failed.
It was allowing the teams, which understood the context and constraints of their bounded problems that worked, not the procedures themselves.
Amazon's API mandate resulted in a product mindset and scaled better than almost everyone until culture erosion killed that.
Delegating works, but centralized process needs to be restricted to the minimum necessary.
Unfortunately the negative aspects of bureaucracy seem artificially successful in the short term, but the negative aspects of setting things in concrete are long tailed.
The growing disengagement problem is one of those long tails IMHO.
Taylorism is actually an attempt to make organizations flexible given that the more subordinated people are all completely unaligned with the organization goals and the management is in closer alignment. It's a very direct consequence of that.
Of course, the irony on it is that reality is often closer to the other way around.
All too often the front-line workers do not have a direct voice that will be responded to from above in anything but a generic way.
So without guidane from above, or in spite of it, the lower echelon ends up aligned most closely with the macro vision of what they think the company is supposed to be like traditionally or as a unit, and it's often quite a bit different than some leaders toward the top who are in position to identify and gravitate to directions that are more lucrative for themselves than the company as a whole.
It could all be represented by a graph but it is a huge measurement problem.
Minimize the variance of what with a business unit? It is not clear to me there is one measure that would apply across business units. I suspect that is part of the problem and something that gets amplified with scale.