The problem is that popular theories of management are about control. The CEO controls everything. His immediate in-group controls slices. Their in-groups control smaller slices still. If you control nobody, then you are lowest in the primate status structure.
Remote working undermines that, because it removes you from direct control. Managers and execs are in large part unable to usefully evaluate the work of the people under them. (The whole theory of the MBA is that it's a universal management degree, that understanding of a domain is irrelevant.) Which is why we have to spend so much time on control-oriented systems like plans and trackers and reporting. Managers, unable to actually judge the work, fall back on the obvious: physical presence and conformance to ritual.
To fix this, we need an entirely different theory of management, a shift from controlling to supportive. There are plenty of options here (Lean Manufacturing, Servant Leadership, "Teal" organizations, what the Zingerman's businesses are doing, even our pre-MBA business past), but approximately nobody is adopting them because the people with control like having control. And because we have these cultural assumptions that somebody is supposed to be in control.