Well of course it wouldn't work if the input is vastly different from what it's been trained on. Why would it?
In so far as we can understand people with different accents it's because we have been trained on them. Even if they are not common around us we've had some exposure, from occasional visitors, travels, or media. When we hear an accent we've really never been exposed to we aren't likely to understand it. A good example is foreign speakers trying to speak our native language... even if they've learned our language in school for years, their even slightly off pronunciation can make it very difficult to understand what they are saying.
So the benchmarks say how well model X does on this exact transcription taks given this exact training data, and no other knowledge.
Even basic things, like female/male voices in train vs test set don't match.