Less productive, compared to what though? In most offices I've seen cubicles and open-plan spaces reign supreme, along with flex-desk policies. Which means that even when all team members are in the office, they are not working face to face. So while you may be correct that teams could be more productive if they're all in the same space, the typical office is not optimized to achieve that.

Also, teams never need to be face-to-face for the entirety of the work: in my experience, only design and planning benefits from face-to-face coordination; for deep productive work the benefits of quick verbal checks is negated by the loss of focus from quick verbal checks from others.