The history of people being scared and unprepared for technological change is pretty thorough:
http://www.techradar.com/us/news/world-of-tech/12-technologi...
My grandfather grew up in a rural part of Ireland that had no electricity until the late 1920s. One of his earliest memories was when a man came to town in a car with a battery powered lantern. It was the first most people had ever seen if either, and everyone around went to the pub to see the car and light.
The car was an obvious improvement over a horse, in terms of speed, cost and versatility. Very few people voyages more than 10-15 miles from their place of birth pre automobile.
Autonomous cars are not that. They will clearly be expensive, and being a service with complex liability problems will most likely be an Über-style, pay by the drink model. As amazing as Uber is, riding them is a lot more expensive than leasing a Honda Accord.
They also make occasional distance trips prohibitively expensive. I do a 1,000 mile road trip every year -- which costs about $500. What would a robot Uber cost?
IMO, they are a solution looking for a problem. Negative effects will include undermining public transit, and making suburban sprawl worse.
Or I drive for 20 hours and get another week in a 5 star hotel.