The problems with VR revolve more around the real potential for VR being highly intense experiences which you just can’t do for 8 hours a day like you can reading, watching TV, surfing the web, futzing with apps, playing video games, etc.
It is easy to dismiss the claims of people violating other people’s personal space in Horizon Worlds as people being over sensitive but actually you can evoke very strong feelings with proxemics in VR. If a character gets too close to you or stays uncomfortably far away it can make you feel chills up and down your spine particularly if these are contrasted. (I think of how I felt in the real world when somebody I’d flirted with the other day came close enough to to feel their body heat the next day.)
Small businesses can use Facebook effectively to reach their customers but doing the same with VR might be a lot more expensive. Things like Horizon Worlds might be too close to real life to really seem like an escape. People mock the low quality graphics but the real problem is an absolute lack of story. It is the artistic and business problems that could keep VR niche.
As for AR there is a real problem with the glasses, Karl Guttag’s blog has gone into great detail how difficult of a problem it is to overlay images on the physical world even if cost and size and looks aren’t an issue. AR has some great applications such as overlaying construction plans over the thing being constructed but whether anything will justify the cost to the consumer is something else.