As a Jewish man, I come from a long-line of hat-wearers. And I get a lot of complements from people on my hats when I'm out and about. Men are curious about them, as if the want to wear them, but fear looking silly.
My experience wearing a hat on my college campus was similar. I found a faux leather fedora (same style as Indiana Jones) and I'd wear that for sun and rain. I don't think anybody said anything negative about the hat, but I got several compliments on it. People like hats, they just worry too much about what others think of them.
Some of the orthodox hats are just so cool. Black. Rakish. Almost streamlined.
Wear one! It's ok!
Huh, the Borsalino. But damn, those are expensive hats!