My personal opinion, may be off, vacation etc rarely keeps retention up. Noone stays saying "wow off-site 6months ago was amazing, I should stay".
If anything, this actually speeds up some people. "Wow, I can't stand these people at the work and now I need to bear them NOT at the work?"
Oh some absolutely do, based on the bonds formed during that off-site.
Especially for a remote company, these team get togethers can be make or break. An annual all expenses paid visit to an exotic destination is a life changing benefit to some people and adds a physical dimension to relationships with coworkers that Zoom doesn't. It translates directly into more trust and retention. It's a significant cost but well within what you can afford if you lease little to no office space.