The anxiety is what I mean by general stress. I'd bet you probably suffer more from the anxiety about the sensory distress more than you do the sensory distress! I had a problem with this as well - the anxiety totally took over, and was much worse. Over time, I managed to keep the distress local to when I was actually feeling overwhelmed, and then could start to work to mitigate the sensory overload, by taking frequent breaks outside, wearing noise cancelling headphones, sunglasses, or finding quiet corners with fewer people.

Its not much of a problem for me anymore, and over time my senses have become a bit more fine tuned and won't be piqued by much more than very loud/unexpected sounds.