You're missing a few words. To be correct you would have to have said 'if nobody's capable of hearing the difference every single time beyond a statistical doubt, does it matter?'

You can say that and be correct, while also sounding a little more silly than perhaps you'd like.

edit: rather than go even harder, I'm instead going to suggest it's perfectly fine to care about things you don't hear every single time, but still like or dislike :)

My pet example is sand in the lettuce for a salad. If you dislike that particular cronch against your teeth while eating salad, it has a spectacular ability to ruin your enjoyment of your salad, even though you don't perceive it every single time. Digital distortions are like that for some of us, things like wow and flutter and vinyl surface noise are like that for others. People vary. (which is also why not to generalize about what 'people can hear')