You make some good points. Time difference is definitely a factor, but the thing is that with people WFH, you presumable are already dealing with people working in different timezones. So you organize those trips to always be somewhat within +-4h of the most of your employees. And those who can't join, will join into another destination. It would surely be one of those things you'd have to have a team dedicated to organize. But that's a team that would replace a bunch of existing, office-specific operational positions.
Food and bacteria – well, that's true. Although not everyone is affected and these come mostly from poor hygiene of either cooks/staff or the employees themselves. You make sure you choose a trusted and tested locations and educate your employees over tap water usage. I guess not much more can be done.
> There’s also the issue that you’re usually there for work, not just fun.
I think it's a matter of the vision. I absolutely would have made it to be more fun than work. First of all, in my example those would be frequent, so there would be less or no FOMO and not everyone would go to each of them. Which means you could definitely delegate the line support to those absent. If you ended up getting called in for help after all, that's just life. But, that's also unlikely to happen each time you getaway.
> you’re gonna be doing 40+ hours working
Well, no. I understand it may have been like this in your case, but it shouldn't be. I see those getaways more as a concentrated leisure/team-building activities you otherwise do throughout the year in the office and still get paid for. Except here you have an added benefit of actually being able to hold some really cool, out of the "cubicle-box" brainstormings/hackatons. Plus all the team building games, parties, etc. – again, all those things you'd normally do in the office to actually bond with people.