In short: drinking coffee/beer/koolaid with the rest of the team is of vital importance, even if the actual work can be done remotely.
Also, the people that don't get this aren't good hires for remote work anyway, because one primary skill needed for a remote worker is knowing how to compensate for not being physically present.
This is especially true when you're looking for workers with very specific skill sets. It's generally easy to find a generic Java developer locally, but if you're looking for, say Cassandra experts (like we are at DataStax, with an office in Austin), then you're probably willing to hire someone anywhere in the world if they're good enough.
I do agree that it's not optimal in every situation, and your culture has to adjust for it. I'm just saying, if there is a talent crunch, companies need to be willing to take steps to handle it. Distributed teams are one such option.