It measurably impacts my own ability to concentrate, I have noticed. We should dispense with such absolutist claims; there is no doubt that any action not directly related to driving a car has potential to impact safe operation of that vehicle.
The function of the law is to discover and enforce tolerances, balancing freedom with safety. We don't have to ignore even minimal safety implications in order to secure freedom, an honest conversation is better for both sides.
It is definitely inhumane to expect drivers to not move their lips or talk while working in solitude for 8-10 hours a day. Instead, we need to design our various systems such that they are capable of withstanding such a tolerance without severely impacting safety.