I guess it wouldn't annoy me so much if my job and social circles (many of them involving foreigners who don't speak my native language) didn't require me to consume so much of it. There's so much subtlety, nuance and "color" that's lost in a conversation where both participants aren't speaking in their native language. Even if both are relatively good at it.
Then there's the issue of native English speakers who seem to be only very dimly aware that other languages exist and think that everyone speaking their language fluently is a given. (This shapes a lot of current politics, such as language policing or the whole 'pronouns' issue - I'm mentioning it because antirez was the center of a controversy for not changing terminology that's loaded for Americans only). And many native speakers think the less of people who struggle with English, even unconsciously.