For example, many English speakers (and Quebecers!) I know have stories like this to tell:
https://www.youtube.com/watch?v=kaFf2kWL45U (45s clip from The Monuments Men)
Now, as an English speaker, I feel like I need to do better for the ESL people I know. It's like having guests in my house. I want them to feel welcome. So when they don't know where the bathroom is, or they flip the wrong lightswitch, I don't make fun of them, I just pretend it didn't happen and offer to help if they ask.