Your ability to not listen is dependent on the speaker's ability to jailbreak your mental sandbox. So you may have deluded yourself into thinking you're not being manipulated by anyone, when in reality, you're only not being manipulated by those whose steering you can recognize.

The steering cancels eachother out if you listen to many people at once, so it's not really an issue. Kind of like how averaging many noisy still photos of the same object denoises it.