The people who say this are the people you never see doing any work. They use the office as a social gathering place. They seemingly have no real job, despite a lofty title and high salary. They are the bane of existence for high functioning employees.
This is one of the few hills I will die on. I've never heard of a productive critical employee complaining about not being in an office. Every single one loves remote work because they can actually get work done.