A culture of remote work takes a lot of discipline, as it's so easy to have an in-person conversation, make a seemingly small decision at the time, and immediately the remote workers feel out of the loop.
Remote work is actually /hard/ to support, and a company has to make it a priority to do it well and be disciplined about writing down all communication. Otherwise, they're setting up their remote employees for failure, which nobody wants.
Also, there are things a remote worker just can't do; stand at a whiteboard and evaluate paths forward, for example. If a company values that sort of collaboration above anything else, it is certainly not "insane" to want people in the office.
Edit: I just got out of a meeting with someone in eastern Europe. An extra $20 for a camera pointed at the whiteboard solves the collaboration problem 90% of the way.
In my experience, meetings like this are more often than not a huge waste of time. Since I started working remotely I am less distracted and more productive, and have no trouble collaborating with other devs.
While it's true you have to have software and processes in place to communicate, it is doable.
Effectively managing an organization that is at least partially remote is more difficult, and my experience in the Bay Area is that management skill is often overlooked and underdeveloped, particularly in startups.
I've seen several companies with very very competent technical founders, very capable engineering team, and not one person in the company that can manage their way out of a brown paper bag. The only way for that kind of structure to work is having everyone in the same room.
These companies need to realize that the market rate for the kinds of engineering and talent they demand is set in the bigger cities, not their little slice of the Midwest. I understand why they're doing it, but for all the issues in the Bay Area the opportunities are still better than almost anywhere else and so is the pay.
Housing costs are eye-watering from my perspective, but cheap for someone from SV or even Seattle.