What I've seen work, which I think still hits your goals of occasional face time, is to have a quarterly offsite for the whole company. Everyone works remotely, but once a quarter you find a nice location that is easy for everyone to get to, pay for their flights and hotels (and if you're really nice, their family too, or at least child care at home while they are gone), and then have a couple of days of quarterly planning, hacking sprints, and general social activities that don't involve alcohol or strenuous physical activity (but if people want to organize that for themselves they are welcome to do so).