People understandably prefer to keep control of their own safety, even if on average they are worse at it. Because individuals that do care about safety can actually do many things to lower the risks well below the statistics, and likely remain much better than self-driving.
There’s also the question of responsibility and liability. In society we have such pressures so that there’s an incentive to continually reduce risks. It is dangerous to shift the responsibility to fast-growth tech companies that will care much less about individual accidents than the drivers involved, and are more capable of absorbing liability without substantive remediation.
Safety is outsourced in many contexts (aviation, police, military…), but it has to be substantially better than what the individual can achieve on their own. For instance, due to the expertise required, or because you can achieve more through coordinated collective action. And there has to be very high trust in the companies involved, the regulatory systems around them, and the technologies employed.