Playing the devil's advocate here, but what would be the difference between talking on a phone via loudspeaker vs. (loudly) talking with another person sitting next to you?
Tinny cellphone speakers play noisy, low-fidelity speech audio captured with a mic that's likely clipping and compressing the caller's voice (audio-wise, not data-wise, although that too is happening and likely doesn't help). Actual people speaking is broad-band, has full fidelity and dynamics, and is directional and context sensitive
Loudspeakers broadcast whereas a person physically next to you can adjust their loudness based on what's socially acceptable
Wouldn't it be equivalent to adjust / turn down the speaker volume on the phone?