Most of these people are passersby on the street who would have no opportunity to touch something I've touched.

I heard from someone who wore a mask recently who was flat-out told, "You're overreacting." It's not just caution, there is some kind of social insistence that the masks are not necessary and even to be discouraged, which I even sense in myself when I'm out and about without a mask and see others wearing them. It's natural for whatever reason, but we should be cognizant that it's just better than if people don't have protection enough to fight that tendency.