The problem is society. I personally don't want to hang out with anyone who has a camera always on, possibly recording everything I say.
I don’t disagree at all with your privacy concerns - but I disagree that it will be a problem. I think the population of people that care about not being on other people’s cameras enough to complain and make those people stop using their camera-laden devices is very, very small.
there are so many other issues like reliable object detection, slam quality, waveguide or display optics, rendering quality, battery life, latency etc…