You forget the business owners who fear they'll lose direction of the work.
It's not so much whether you put in the hours at home, but whether your work is directed towards a common goal.
Remote work requires a higher degree of communication, and some business owners don't possess the skill or interest to maintain out-of-office communication.
However, the opportunities are also massive. For one thing, the owners/managers get to work from home too, which I certainly wouldn't give up! You can also hire from anywhere, which is huge in terms of finding talented people. Most importantly though, I think, all the things you need to do to manage a remote team effectively are things you really should be doing anyway. Yes, maybe some of the communication would be face to face instead of Slack or whatever, but for the most part if you're managing well, it's not going to be too hard to translate that to a remote environment. It's about actually knowing what people are working on, not just making sure they're putting in the hours. Knowing the goals of the team, and the individual goals of the team members. Fostering communication and collaboration (whatever the medium). And then continuously reviewing and refining the processes by which you do all those things.
This is a negative for many businesses: their employees can get poached by competitors far away that offer better pay or benefits or working conditions.
Yes, but what about the crappy companies? It's a big net negative for them, so they're going to oppose it. They don't want to compete with companies around the nation; they want to be the only employer in town so their employees feel locked in and unable to change jobs without selling their home and moving.
Years before covid, I pushed my company out of an office to full remote for many of the reasons you listed. It allowed us to keep a few great employees who wanted to live elsewhere, and save substantial money on office space (which matters a lot for a small company). At the time Slack had just become big and we embraced the async, transparent communication style fully.
Early on one of the other owners asked me, 'if no one is in the office how do we know they are working?', and I responded 'how do we know they are working now?' That's when he 'got' it. We had never relied butts in seats, but it can still be a little scary going into an unknown.
Remote/in-person both have unique challenges, they are just different. After years of remote, I think the advantages outweigh the challenges vs in-person.
It's certainly been a continuous process learning how to effectively manage a remote team (and manage in general), but agreed that at least for us the advantages definitely outweigh the drawbacks. It's actually been nice as remote work has caught on elsewhere, because though it means more competition, it also means much better tools as well as a broader knowledge base to draw on.