You ever look at other drivers when you are stuck in traffic? You will see people looking at their phones. You will see people eating. You will see people shaving. You will see people doing their makeup. You will see people reading books. You will see people with dogs on their laps. And that is just the people who might accidentally kill you due to carelessness. Vehicular suicide is more common than most people expect and there is no telling if you might be collateral damage in one of those collisions.
Self driving cars don't have to be perfect to be dramatically safer than human drivers.