Well, I worked at a place where most of the team was remote. It absolutely sucked. I barely knew what the coworkers were doing (sure, my boss had a better idea than I did, but I am certain even he would have preferred to have employees physically around him for routine interaction), what kind of people they were, and the reverse was true with respect to their knowledge of myself. The time zone difference only exacerbated the problem. This "lack of people knowledge" problem at remote workplaces causes everyone to under-invest in building long-term relationships with everyone else (because nobody really knows what everyone else is really like), and the culture nosedives. [1]
Now, clearly, remote employment can be necessary (consulting offers a typical pattern), but the common attitude that there's no reason at all for not just having everyone work remotely is incredibly annoying.
[1] Speaking of culture, remote workplaces offer a clear explanation via counterpoint of the vaunted but often misunderstood term "company culture"---culture is what you don't have when everyone is working remotely.