This is either a problem with you, or a problem with the specific place you live in.
I talk to people every day in my 2nd language. They are generally delighted to do so.
Yes, some people are suspicious of me, depending on the context. But even then I can generally disarm it with body language etc, because I'm a skilled and empathetic human.