I think the overall theme here shouldn't be "I'm sure we will lose ppl and that's fine".

Instead you should be an advocate for mixed teams. Why can't your remote coworker just zoom into the meeting with you onsite?

We can all be accommodating to each other.