I have this distinct memory of being picked up this one time by an Australian friend who was taking my cofounder and me to lunch after we'd been in the US for a few weeks.
As I sat in the car and we started chatting, I immediately had this sense of relaxation come over me, knowing that at least for the duration of that car trip, I'd be surrounded by only my native accent.
I'd never noticed it before, but it was a stark realisation that deciphering foreign accents is hard work and seems to put your brain into a much higher processing rate.
And as Australians, we've grown up being exposed to USA accents all the time through television etc. Yet even then it had been exhausting.
So yeah, I can empathise with the notion that deciphering foreign accents is something that takes extra effort for all of us, and whilst it may be a factor that promotes prejudice, it's not an indicator of poor character, but just one of those things that we have to accept is there and be more mindful about overcoming.