For many people I know this is the case, so I don't think pure remote or location work are "it".
I work 100% remote with semi-flexible hours and a big part of this is the freedom to leave my house and do things during the day, like meet up with friends, or go exercise while the sun is out in winter etc.
You can still do that. My friend rented an office a five minute walk from his home - it costs him 3% of his salary.
That's a completely different issue from remote vs in-person work. You should learn to be comfortable being alone/by yourself, especially as a man.
You guys can go to office together, and leave the rest of us out of this?
That's nice for you.
From personal experience on the other hand {statement of preference}