That sounds fine in theory, but who decides which teams are remote? How do you avoid a situation where someone comes on a remote team and gets the rug pulled out when someone else decides that team is now in person?
As for getting rug-pulled: that possibility exists with any management decision. You could always get an asshole boss, the company could get sold, etc. Not to fully discredit your point, but it wouldn't be my biggest concern as an employee.