I once had a short consulting engagement with a VR company that spun out of Second Life and had to be "in world" the entire time. They had made a VR office space for the dev team to congregate in. The first time I beamed into a meeting room and saw a dozen blank faced avatars staring back at me I got filled with anxiety. I've been working remote for 13 years and I can't remember another time I felt anxious like that running a remote meeting.
The people were great and the spatial audio was interesting, but I found it really distracting to have to be in a VR world when really I just wanted to be in emacs coding. I felt mentally drained by the end of every day, worse than being in an office working a full day. I felt "always on" in a way I don't in an office (and certainly don't when remote). Maybe there's a niche here, but it's really not for me.