And occasional face to face team meetings are part of that work at many companies. If you don't want to participate you're probably better off finding a company that doesn't have such an expectation.

But what if you also like the people you work with?

Or, to flip that around a bit, what if you hire people who are both competent and interesting/pleasant to be around?