Your last sentence pretty much summarises 90% of reasons why some companies won't go remote route.
As a manager, one needs to understand what his team is working on and how they do it. If a person can't understand underlying work or processes,then hi shouldn't be a manager.Also,while one doesn't necessarily need to be a dev to be able to manage devs, ability to understand problems and time required to solve them is crucial.