As a full-time pedestrian without a car, while I'm modestly afraid of autonomous cars, I'm really afraid of distracted drivers on their phones.
At an intersection you can usually see where the driver is looking to help inform your frogger-like quest to not die. There's nothing to look at in an autonomous vehicle. You are operating on hope that the OS was designed to see you and stop accordingly.