In my personal experience, after going fully remote, my manager implied that it could limit my ability to advance, and that ended up being the case. At least in my company, face time with leadership seems to carry weight when it comes to promotions. For me, the benefits of remote work were worth the potential tradeoffs in career advancement. Regardless of your stance on remote work this is the reality.
I've also been on several teams post-COVID where the majority of members were remote. In my experience, when issues arise on remote teams, it's usually attributable to other underlying problems, not the remote work itself.