Experience. Indirectly linked to age. When you have been driving for a number of years and have experienced how quickly and unexpectedly things can happen while driving, it's hard to completely put your life in the hands of an automated system.
If I were to buy a new car today, I'd for sure look for one that has some of the modern safety features like lane keeping and automatic emergency braking. Dynamic cruise control would be great for long journeys. But I wouldn't feel safe riding in a car with what Tesla calls 'autopilot' capabilities, or more complete self-driving like Waymo - although to be fair in a city due to lower speeds the risk of injury (to myself) is going to less than on a highway, so I'd actually say I'd feel safer in Waymo than a Tesla.