But whispering is a deliberate attempt to conceal what you're saying. Speaking to somebody in another language is usually not an attempt at concealment, it's just about people communicating in the way they are most comfortable.

I guess I just am sensitive to the unfortunately common American assumption that everybody should speak English. My first time at SFO a hispanic woman was at the baggage carousel with her son. The son asked, in English, if he could lift the bag off the carousel when it arrived and his mother explained, in English, that he was too small to lift it. The kid didn't drop it and the mother lost her temper and scolded him in Spanish. At that point, a lady beside her turned to her and said "speak in English, please"! That blew my mind...