I'd bet my money on no. To my understanding, those hallucinations are basically brain-level disturbances in perception, where the brain does its best to fill in ambiguous activity with known objects. So I guess if you've never seen hats (or men), your brain would just interpret the signal as something different.

Extreme tangent but interesting (to me): people in different cultures experience voice hallucinations differently. In the west, "hearing voices" is often frightening because the voices are hostile. In other cultures, the voices are friendly!

https://news.stanford.edu/stories/2014/07/voices-culture-luh...

That's an interesting question, but the answer probably isn't worth the trauma you'd inflict on a child by intentionally raising them to be completely unaware of the existence of hats.