I think once we're to the level of criticizing
talking, a behavior that has been concurrent with driving the whole time driving has existed, we're close to demonizing devices [0] rather than poor behavior.
I don't see a problem with talking to close friends, holding the phone to my ear with one arm, on a highway in low traffic conditions (eg long drives at night). But any time I come up on a situation (passing a meandering car, heavy traffic passing me, construction zone, car stopped in breakdown lane [1]), I abruptly say "hold on", and take the phone away from my head. I probably laid the foundations for this behavior driving stick (where you want your second arm back to be able to shift etc). Most situations I also take the vehicle out of cruise control so that keeping speed has to be done with intent.
I can see the temptation to make the opposite choice - assume that everything will be fine, keeping some attention on the conversation. Meaning that if something unexpected occurs, I wouldn't be fully primed to deal and the problems multiply.
But I admit that this is all modulo the same bias of personal perception that makes many people think it's fine to stare at their phone half the time while driving. So perhaps data would disagree with me. But it takes more nuanced data than a simplistic division between "talking on phone" and "not talking on phone".
[0] There is a small possibility here that the DSP involved in making mobile calls sound reasonable only changes perception and still requires much more brainpower to process the audio.
[1] And because it doesn't seem universal: if someone - anyone - is stopped in the breakdown lane, move to the left even if you aren't in the right lane, to facilitate people in the right lane also moving over. The less "serious" a situation on the side of the road looks, the more likely it is that said people are amateurs unaccustomed to being pedestrians near the highway.