People choose what to worry about. You can say it's unfair because they could've worried about a different set of things, but at the end of the day it's their choice.
The relentless demand for justification is indistinguishable from a demand for others to stop doing things you don't like.