It's much easier if we're talking about a new company. IMO companies formed after the pandemic have a much easier job of figuring this out. You can set expectations from the beginning about which teams are remote and which are not. Some teams should be remote. I know of a hybrid company that has always had SREs be remote. They operate a global business and want people covering all timezones.

As for getting rug-pulled: that possibility exists with any management decision. You could always get an asshole boss, the company could get sold, etc. Not to fully discredit your point, but it wouldn't be my biggest concern as an employee.