Because driving risk is far from evenly distributed. If a self-driving car drives like an average human driver, for responsible drivers, this would mean incurring more risk by getting into one.
A huge portion of the risk of human drivers is due to confounding human factors like negligence, distraction, medical issues, fatigue, etc. Responsible drivers can take most of these factors into their own hands -- not driving while sleepy, not driving drunk, not using their cell phone. These people don't want a vehicle
A better comparison would be comparing the capability of a self-driving car to the skill of a human driver driving under ideal circumstances of health/responsibility/etc.