Anecdotally I've been using a combination of open-ear headphones and IEMs for over a decade now. Also a huge live music fan.
Despite listening to a lot of music, some of it quite loudly, I've no perceived hearing loss. Can still hear whispers, distant conversations, and feint sounds that many people can't even hear at all.
Everything I've read on heading loss indicates that managing listening duration relative to the loudness of the source is critical. There's charts for this that'll tell you exactly how long you need to listen at a specific volume for damage to occur.
You can crank it up for a song or two, provided it's followed by a break or significantly reduced volume. Similarly you can very briefly remove ear protection close to the stage at live venues. It's all about breaks, protection, and managing duration.
If you listen to music all day at the same volume, I'm sure it's very possible to unknowingly build tolerance and do some damage. Fortunately my mind tends to burn out on music long before my ears do.
Slightly off topic, but there's a similar risk with live music. One of my first live shows, I forgot my ear plugs and ended up too close to one of the speakers. Didn't really bother me until the next day when I awoke to my ears ringing. Fortunately it cleared fast with seemingly no permanent damage, but I was lucky. I'd say 95% of people at live shows have zero hearing protection.
On that note, if you ever forget your ear plugs at a show, just periodically plug one ear at a time with a finger for 30+ seconds every few minutes (or whatever keeps you comfortable). You might look stupid, but it'll save you.