I've accidentally wandered onto a film set before. When it's in a public space it's not always obvious what's going on or that you shouldn't be there. I had no idea why a generally quite busy part of town was almost entirely devoid of traffic that day. There were no clear signs or ways for me to know what was going on until I talked to a random guy and he told me.
Perhaps your ability to detect and understand context is simply vastly superior to mine.