If it was actually perfect grammatical accuracy on their part they’d be saying “Who were you talking to” or “Who were you speaking to”.
Correct is what the people say, not the opinion of literal singular English language teachers.
There is a sizable contingent of English speakers out there that continue to insist that it’s incorrect to end a sentence in a preposition. (I am not part of this group and it is fortunately shrinking over time.)
Texan: Where are you from?
Harvard Student: I'm from a place where we don't end sentences with prepositions.
Texan: OK. Where are you from, jackass?That is the sort of nonsense up with which one should not put.
I couldn't tell you why, maybe it's a feeling of lack of authenticity?
I find it impressive when someone has acquired a good accent personally since it's quite difficult to do.
When learning Japanese I was encouraged to pick a person with a dialect I prefer and listen to them a lot, as I'll likely pick up their accent and mannerisms.