The people I work with are not my friends. I despise most of them. This is true for every place I've worked. It's probably true for most people here. You don't chose your co-workers. And now you're forcing me to take my vacation with these people, who I put up with every day at work.
You may be puzzled reading this. "But my company is like a family." Some companies are really like this; everyone's your friend. But if every company was like this, we wouldn't have shows like "The Office."
It's also hard to expect everyone to go on a vacation. What if your kid has to go to school? What if your spouse has a job, and has to stay home. Can you ask people to leave their family behind for a week so they can bond with their co-workers?
Then you might say "fine, just make the vacation optional." You won't be fired for not going, but when it's time for performance reviews, who's gonna have "not a team player" on their report? The one person who didn't attend.
But you make a point. We're people. We need contact. My solution? Work at a co-working space a few times a week. Work out of a library or cafe every so often. And get everyone on your team to use video conferencing.
And if your team lives in a close vicinity, it's totally reasonable to schedule some day trips.