For me it's areas a few blocks around Times Square. The only thought I had (and maybe the Oculus is the same) is that the sheer number of people also using Airpods in the area means everything gets confused.
Same, in London, walking down Oxford Street at 6pm can mean almost constant ‘diruptions’ over my AirPods.
In short, it's the combination of no walls + lots of signal congestion. Times Square + Oculus both fit this mold.
That's the latest ridiculous annoyance in a long, sad series of ridiculous annoyances that I've experienced with portable audio, to the point where I've just resigned to not listen to music anywhere other than at home and at work. The hassle and distraction of managing either an audio cable or a Bluetooth connection are just too much.