As already mentioned it can interfere with constant checking habit for extra safety, as well as cold weather appropriate headgear. Personally I gave up cycling once compulsory helmet laws were introduced in Australia since for sun protection I had an already heavy enough wide brim weather proof hat. It's probably not that common, but I have a weakened neck from a poorly treated bad whiplash (forced over the edge of road on a bicycle with a wooden road marker being right at the spot my head / neck area would land.) Additionally trying a helmet (not all helmets are the same and possibly there are brands out that have addressed certain problems) I found the straps sitting proud interfered with other distant road noises, not an issue in town traffic, but it would certainly drown out vehicles approaching from behind driving on non urban roads, with many drivers not looking for cyclists sharing that road.
I whip my head back and forth over my right shoulder constantly to check the state of traffic behind me.
Before I maneuvre, I check, check and check again. Always checking. Always fully aware of what's behind me.
Having something attached to my head makes it harder and less comfortable to do these rapid checks. I think this makes me less safe overall.
Because I like my head not freezing when it's ten degrees below freezing? When I bring this up, sometimes people tell me about thin membrane caps that fit under the hat, but they don't work when it's actually really cold out. Helmets absolutely do not fit over warm-weather hats (or any hats, really). (Other people have said "why would you be biking if it's below freezing", which shows that they don't understand the entire conversation.)
The helmet itself gives some wind protection which helps as well I think.
Here in the Netherlands, I guess that my risk for getting involved in an accident, is not any higher than walking and other forms of transportation. I am not using an e-bike and I usually bike at a speed of about 15km/h. I do not use a racing bike, but a bike where you sit in an upright position allowing you to look around easily. I also take care of using routes that are safe. Here in the Netherlands, where are more bikes than cars, and biking is an important means of transportation for daily commutes and doing shoppings. We have good infrastructures for bikes.
I live less than a mile from my former cubicle, and my old commute was mostly through a park with two road crossings. There was no reason to wear a helmet.
If I'm on a longer road ride, or riding off-road, I religiously wear a helmet. If I'm riding lift-assisted downhill, I wear a full-face helmet.
They’re not GOOD reasons but let’s not pretend there aren’t any at all.