* Motion sickness is still a really big problem
* If you're short-sighted and wear glasses, then the picture will be blurry for you. The original PSVR worked around the issue by pushing the lends out so you could comfortably wear glasses and the headset, but none of the modern headsets afaik include this affordance. Wearing contacts isn't an option for a lot of people.
* Way too expensive. I can afford one pretty easily, but most people can't. For roughly the same price as a Quest you can get an iPhone SE which is going to offer significantly more value to the average person
* They're too heavy! People complain about a 240g iPhone being too heavy in the hand. The Quest is 503g on your head, and this headset in particular doesn't distribute weight very well. PSVR is slightly heavier, but is at least slightly more comfortable than the Quest due to its weight distribution.
I think there might be some other factors at play, too. Throwing on a VR headset feels a lot more antisocial than watching TV or tinkering on your phone in the living room because you're wearing a giant headset that separates you from the real world. Even if the hardware does get really good, I'm not entirely sure how easy it is for manufacturers to convince families that headsets are a good purchase.
I really enjoyed using my friend's VR headset to slowly play through Half-Life: Alyx. I could manage about one level at a time before needing to take the headset off and take a break, because of motion sickness + how uncomfortable the whole thing was. I think there's still a lot of R&D to do before headsets are really in a position where regular people start buying them up en masse.