When I hear folks complain about the pains of working from home, what I often hear are their stories about how they were needlessly disruptive to their coworkers.
Having impromptu hallway meetings mean that your peers, and subordinates, must keep an ear open for such events should they risk being left out of important decision making processes.
Having watercooler conversations means that your office-mates must take effort to overcome their human desire for social communication in order to focus on their task at hand.
And so on.
Generally, I find having a team that works wholly remote means that we schedule communications clearly and are able to maintain the focus on our work before us.