You suspect wrongly. Having seen the attitudes to my father change when he wore one. Ranging from outright verbal abuse, to assumptions of stupidity & senility.
(I have no idea what is the correct term for "behind the ears", I hope it's understandable.)
Off-topic: While I was there, they asked employees to submit ideas for a new hearing aid marketing slogan, with the incentive of a free vacation to Vegas going to the person who submitted the one they used. For some reason, I did not win the vacation with my suggestion: "Stick It In Your Ear!"
At the low end when eating out in restaurants occasionally having wait staff ignore him and asking other folk at the table "what would he like", or people assuming that he couldn't hear and talking about him — to at the high end having a guy shouting "deaf fuck" at him repeatedly on the street for no obvious reason.
I'm not trying to say that this happened every day — especially the outright insults. But it was enough to be noticeable.
I suspect, as @hibbelig commented, age had something to do with it.