Even people who've spoken English daily for decades will do a double take if they're not used to it.
If you listen carefully, the range of vowels is quite wide, for the same word in different accents. The probably throws off a lot of people.
When they visited Texas for a family reunion, several of my cousins who have not spent much time around non-Texans honestly asked about the language she was speaking.
I think this diversity, while large, is still less than in many other languages. As one example of many, just within Europe there are accents/dialects of German in Switzerland that are largely mutually unintelligible even to some other Swiss German speakers, and nearly entirely unintelligible to most Germans. This is also true for dialects of Italian, or even Dutch, according to friends from those countries. It's generally even more fragmented outside of European languages.
Unlike English, some of these cases don't even have easily standardized registers. In my experience, people in Birmingham or Glasgow, for instance, can fluidly switch to a more standardized accent and dialect that's natively or "intuitively" understandable to a speaker from North America or South Africa.
The thing is though, you can ask people to speak High German instead of mundart, eg at meetings. They admit it's different enough to warrant explicit statement.
You can't be telling your Liverpudlian or Brummie to speak the Queen's English, that would be rude.
Dialect continuums between languages are basically the norm around the world. You can even connect Sicilian to Portuguese for example.
I would assume the primary reason English isn't part of any major one is because Britain is an island, otherwise English would probably connect to Dutch and join the West Germanic continuum. It would then appear just as fractured as any other language.