It always felt like "a load of bullpucky", but the defining moment for me was one day riding down a quietish-street with a legal-but-still-obnoxious Akrapovic pipe. In front of me were two or three girls walking in the middle of the road, talking. They only realized I was there when I sounded the horn and seemed really surprised, judging by their jumpiness.
If someone walking in front of the bike can't hear that noise, I really don't see how someone inside a sound-insulated box, in a generally noisy environment, probably blasting the radio, is going to notice the loud pipe.