That's not quite it. It's that there are some people crying wolf who see something as erotic material when it's not. Do you see the difference there? Getting to the heart of why they see it this way when others don't is an interesting question.

There's an ugly implication of the "I know it when I see it" standard -- something in the individual's brain has realized that what it's getting from the eyes is supposed to be exciting. How could they otherwise make the claim? That's got to be an unsettling experience for a lot of people. The self-disgust this causes is likely to be so visceral and panic-inducing that it will be hard for someone experiencing it to understand that the reaction is not universal.