There are two things I do not trust: I do not trust that our technology can successfully and intelligently respond in the event of an traffic or roadway anomaly, the kind that usually causes car crashes i.e. damaged roads, debris falling off of cars, erratic drivers, etc. Although I do trust safety features, which are pretty nice lately, I would rather have not have a wreck.
Finally, I do not trust other drivers and I do not know how they will react/interact with a self driving car. It only takes one time for an idiot to flip out on the highway and cause a wreck.
When I feel that the self driving can adequately handle anomalies, wrecks, and suicidally bad drivers, then I will have no qualms with the self driving car. Till then, I'm not willing to bet my life that it works well enough.