Obviously talking loudly on a cell phone is an undesirable trait for a driver. No question. But I do question the assertion that they're talking about you. I speak a few common languages and have heard taxi drivers around the world speak on their phones and assuming that I only speak English. They very rarely talk about me and if they do, it's usually in a relatively professional context, i.e. "I just picked a guy up at the airport", not "this idiot doesn't know what I'm saying so I'm going to say insulting things about him".
I don't know quite why people who hear others speaking a foreign language tend to assume they are the topic of conversation. People usually have more important things to talk about. Of course, sometimes you do catch somebody talking about you and that tends to be hilariously awkward when you respond in his/her language!