It's definitely a weird kind of embarrassment, I won't pretend it doesn't exist. But it's one of those hurdles that are so small in retrospect, you wonder how you ever let it stop you from experiencing all the benefits of a cycling rich life. I think it's because until you've experienced it, you don't realize how much is on offer on the other side of the fence.
In my own city I wouldn't ride without one, but in a more shared space friendly city, like famously in many cities in Japan, I wouldn't be as inclined to wear one. If I'm riding at 30kmph right next to traffic, you bet I want a helmet. But < 20kmph in and out of foot traffic, or properly shared slower roads, then I wouldn't worry about it.