First, though, I'll tell you that I'm currently working remote for an SFO startup and before that, I spent six years working remote for a large cloud provider based in Texas. Before all of that, I was an on-site worker for the first 14 years of my career.
I don't like remote work. It's not because I'm not efficient or don't get shit done (I am and I do). My chief complaint about remote work is how it socially isolates the worker. No matter how many Google Hangouts or morning standups you do, you are losing out on a huge part of the social experience of working onsite. One of the greatest things about working around smart people is the chit-chat that you get by working right next to them. I'm talking about the conversations you have on the way to lunch or standing around the soda fountain. This stuff is spontaneous and fast-flowing and it's precious. As much as I wish this could be replicated over IRC or IM, it just doesn't happen that way. There's something magical about the conversation from four geeks riding together in a car on the way to lunch.
The other aspect to social isolation is personal happiness and the desire that most of us have to be in the presence of others. Over the last six years of working remote, I've gotten really tired of being alone all the time. Working from a busy coffee shop helps, but you're still not chit-chatting and you have to contend with shitty Internet connections and poor security (pack up your laptop to use the restroom, etc).
Working remote is not all that it's cracked up to be. It took me several years of actually doing it to realize this. I love my job at this awesome company but if I could work onsite and still have my family here with me, I'd do it in a heartbeat.