I think you nailed it: you have to relax a certain amount and let your brain do it's own thing—like falling asleep. Some people have more of a tendency to latch on to the fact they're looking through a device and don't 'let go' and the magic doesn't happen. It's also probably situation dependent; your mind is less likely to let go of your actual environment and focus on the virtual if there are distractions or anything you're concerned about in your surroundings (e.g. it will be less likely for some people while in stores and at conferences etc.).