Distributed teams work really well.
One-person-out teams are a surefire recipe to crash-and-burn.
With distributed teams, you get into a flow where Hangouts are as natural as going for a walk, where everyone has a proper headset, where there is appropriate shared whiteboarding software, where people communicate with tools which leave a digital trail (mailing lists, wikis, github, etc.), and so on.
My distributed team worked better than local teams, hands down.
With one-man-out, there's one person who wants this stuff to work, and the rest who, well, don't care too much. There will always be the dial-into-a-conference-room where you can't hear someone, the one-douchebag-doesn't-use-a-headset echo, the everyone-but-the-person-out-goes-to-dinner-together, and so on.
It. Just. Doesn't. Work.
It can work depending on the mindset.
You're right, if the single man team is looking to socialize and feels a lot of FOMO not being in the office then it won't work.
You obviously don't hire a person like that for your single man remote team.
I've found that in the vast majority of cases single man remote teams are remote because they're optimizing for a particular way of life that they enjoy; for example, the country side lifestyle with their partner and kids.
Then it usually works out fine; we will still organize for a on-site every 2-3 months for a week just to give them a bit more of sense of belonging.