Effectively managing an organization that is at least partially remote is more difficult, and my experience in the Bay Area is that management skill is often overlooked and underdeveloped, particularly in startups.
I've seen several companies with very very competent technical founders, very capable engineering team, and not one person in the company that can manage their way out of a brown paper bag. The only way for that kind of structure to work is having everyone in the same room.
A culture of remote work takes a lot of discipline, as it's so easy to have an in-person conversation, make a seemingly small decision at the time, and immediately the remote workers feel out of the loop.
Remote work is actually /hard/ to support, and a company has to make it a priority to do it well and be disciplined about writing down all communication. Otherwise, they're setting up their remote employees for failure, which nobody wants.
Also, there are things a remote worker just can't do; stand at a whiteboard and evaluate paths forward, for example. If a company values that sort of collaboration above anything else, it is certainly not "insane" to want people in the office.
While it's true you have to have software and processes in place to communicate, it is doable.
In my experience, meetings like this are more often than not a huge waste of time. Since I started working remotely I am less distracted and more productive, and have no trouble collaborating with other devs.
Edit: I just got out of a meeting with someone in eastern Europe. An extra $20 for a camera pointed at the whiteboard solves the collaboration problem 90% of the way.