Having started remote work in 2006, I always found the biggest benefit from actually meeting in person for a week at the time away from everyone's home, 2-4 times a year.

When a company is just starting up and on the smaller end, all-hands type meetings once a year or once every two years are also nice.

The important bit is that it's a location remote to everyone so people are likely to spend time together after work too. This also makes it easier to look for cheaper locations.