I think a good chunk of it was the CDC randomly changing its recommendations with no real explanation.
At the beginning of the pandemic in the US, from January to May of 2020, there was a big campaign by the CDC to discourage people from wearing masks. I remember seeing various guest "medical experts" on the news claiming that people who wore masks were actually MORE LIKELY to contract COVID. At the same time, virtually every other country was encouraging or mandating masking.
Then June of 2020 came around, masks quickly became encouraged and then mandated. Fauci only made vague and nonsensical statements like "the science changed!" and "we're moving at the speed of science!" to explain the abrupt change in policy. As if there was ever any scientific data to suggest that COVID-19 was not capable of airborne transmission or that masks wouldn't help limit said transmission.
I get that there was a shortage of masks early on and the CDC was probably just trying to prevent panic buying (which occurred anyway, there were N95s selling on amazon for $40 per mask). The ridiculous claim that "masks make you more likely to get COVID" followed by a sudden requirement to wear said "COVID causing" masks completely destroyed public confidence and fueled all kinds of conspiracy theories.