To me, the downside of remote coworkers is we've already seen a dynamic at many companies that start with "we'll allow remote workers" straight to "if we allow any in-person collaboration, then remote workers will be second-class citizens, so to pre-empt that, we will actively discourage any in person collaboration."
For example, Coinbase didn't just allow remote but shut down the SF office for this reason. Twitter is re-opening their office but in a crippled state, where the food options are massively downsized, and employees are actively discouraged from eating with any teammates.
If you're the type of personality who gets energized by collaboration with teammates, if you like the real teammate relationships that more easily develop with facetime, then it's not a matter of allowing remote coworkers but whether those remote coworkers now get to advocate for actively destroying any office culture.
Again, I understand why there's many advantages of remote work, but let's not pretend the people who didn't want to go remote are unaffected.