Why don't you become remote then?
Remote all the things!
The upsides for me of being in the office massively outweigh the downsides, but only if my colleagues are there too.
In practice I feel like you could just aim to have some teams be mostly people who like office and other teams be mostly remote. Would be awkward hiring wise though to send someone to a different team just on there office vs remote view.