One argument I heard is that junior team members benefit from others being in the office.

So there might be a long term benefit to the team, even if individual productivity of high performers goes down slightly.