People wore fedoras to identify them as part of a subculture (Reddit gentlemen).
Most people who weren't part of the subculture pretty quickly figured out that removing their fedora meant people didn't assume they were a part of it, so they took it off.
Same thing with hipster glasses, mullets, dreads, swastikas, list goes on...
Is that where that came from? It sounds right.
I used to wear nice hats regularly including a couple of fedoras, but I had to retire them because of this.
How many times have you met an actual douchebag wearing a fedora? How many times have you heard people saying that all douchebags wear fedoras? I bet the ratio is at least 1:50. It's like how people who insist all vegans are smug outnumber actual smug vegans by a huge margin.