It’s not grammatical accuracy on their part, it’s the opposite, it’s a calque, a word by word translation from their native tongue.
If it was actually perfect grammatical accuracy on their part they’d be saying “Who were you talking to” or “Who were you speaking to”.
Correct is what the people say, not the opinion of literal singular English language teachers.