Some people just can't handle working from home. A lot of people, actually. Heavy-handed management can somewhat keep them in line, but then you're tying up extra manager time just to get certain employees to do their job.
That's not to say people can't slack off in the office, too. The problem is that someone prone to getting distracted has 10X more distractions at home and feels 1/10th the peer pressure to work like everyone else in the office.
But if you start picking and choosing who has to come back and who gets to stay home, it's a disaster. The people forced to come back to the office will resent the people who get to stay home, creating more divisions in the team.
The real issue, IMO, is that companies hired way too many people who couldn't handle remote work. At this point it's not an option to apply more management pressure or lay them off because they make up large parts of the employee base. It's not exactly fair to people who can WFH, but at some point bringing everyone into the office starts to look like the only practical solution to addressing widespread problems.