I only prefer working remotely as an alternative to an office with open-air seating. Give me an office with a window and a door, and I would quite willingly work from an office. My employer has, for over a decade, refused to build anything but open-air seating. So everything thinks I prefer to work remotely, because their own feelings taint their perceptions of my actions, but they don't ask my actual motivations and take for granted that I support their views.
Please take care not to frame everyone who prefers working from an office as people who would "impose" their views on others. I work from home rather than impose my views on others, but your argument makes a very good point. I should probably step up and begin pressuring for better office design, since that would remove a crutch that the remote-work movement is unaware it's being propped up by.
Yes, some will still prefer to work remotely, but I've done offices, and I've done remote, and it's clear there are pros and cons to each that vary per project and per team. It's time for office design to be given due credit for how it drives people into remote work, so that we can stop confusing a desire for a healthy working environment with a desire to work from our homes.