Wearing headphones every day will permanently damage your hearing. If you really want noise, put on headphones. Respectful quiet should be the default.
If you instead use a good pair of over-the-ear headphones so that you can get the sound you want at a low volume and without the assistance of any active noise cancellation, it's not quite the same story. Probably still shouldn't be doing it all day just to be on the safe side, but from what I've been able to find in the literature it look like it's probably fine.
I don't use expensive headphones (mostly in-ears as they are more comfortable) but I went through a few different sets til I found some that worked like ear plugs.