The Quest 2 has only been out since October 13 of last year. 4m units in the US in less than a year is a great number.
It's likely that most of those 4 million users have never had a VR headset before.
Where as I have a supercomputer in my pocket that’s a pleasure to use, always accessible, cheap to expensive, and directly manipulatable. Even the blackberry (aka crack berry) was far more popular than VR ever has been. I just don’t see it outside of certain games and niches, and I’ve tried.