> reducing them only affects the slowest quartile or so of drivers

And having to navigate traffic that is not of a homogeneous speed creates cognitive load for all drivers. So then they don't notice things like cyclists and pedestrians.