1) They underestimate problems with their ideas, and then brush them off with solutions which, when summed together, make the visionary idea much less palatable. For example, the idea that we can abandon traffic lights and stop signs is complicated by the fact that not everybody can, will, or even should use a car to get around. But in the world without traffic lights and stop signs, pedestrians and cyclists are trapped by fear. Sure, they might technically be able to walk out into traffic and the wondercars will stop for them, but in reality, nobody will do that, because the cost of being wrong is death. In every conversation I have had with these technophiles, the solution always ends up being some massively costly network of pedestrian bridges at every intersection of the city. Yeah, not so palatable anymore.
2) They fail to understand that, like evolution, intermediate states have to be viable improvements on their own merits. Wheeled animals do not exist, not because wheeled animals wouldn't have superior mobility, but because no possible intermediate evolutionary state has incremental benefit over the previous legged state. How do you transition a 2 lane road into three lanes when only 50% of the cars are self-driving? What about when 99.9% are self-driving? There are thousands of imaginary future states that do not have credible intermediate states, and will likely never see the future state as a result.
With computer in charge it will be even more safer.
There are a few reasons for why it can work. One is the idea that drivers on a well delimited road become complacent. If it's a road designed for cars, then drivers go faster, believe they have priority, and pay less attention to non-drivers.
You can't just get rid of signs and expect things to work, because signs are the only signifier. For example, the curb is a physical indicator of the boundary between the place for cars and the place for non-cars. Also, if there are more non-drivers often near the streets (cyclists, walkers, people sitting at a cafe table) then drivers will be more attuned to events which are currently rather low.
It also depends on a relatively slow speed. That video for Saigon shows that the traffic is about that of a jog. At that speed, it's easy for drivers to slow down, and easily within the normal reaction time for humans.
If you go this route then you can't have broad, open, clear avenues since that encourages drivers to go a lot faster, which makes things more dangerous.
It broke off from driverless cars a while back, I think. I personally can't wrap my head around the idea that a personal chauffeur would get rid of city congestion. Who wants to wait more than 5 minutes for their car to come? That sets a maximum distance to wherever the car depot might be.
Even today, cities introduce pedestrian areas or limit the number of cars to certain areas, or they have special bus and taxi lanes. It seems likely that they would introduce special "only robot cars" areas eventually, as a first step.
The transition from traditional cars to driverless cars will be smooth, insofar as the cars are intermingling. But the supposed additional benefits that idealists like to portray as inevitable are nearly impossible to intermingle. The only way that they can appear is through a forceful transition...eg banning non-driverless cars.
It also seems to me that the transition to driverless cars will probably sneak up on us: "normal" cars will become more and more intelligent until suddenly they can drive without human input.
Think of a electrical utility truck. They need to do way more than just follow the rules of the road...and in some cases they need to break the rules of the road.
Technically they are apparently not wheels because the whole animal/plant rotates, but I'd argue that nature worked out the wheel principle just fine. In fact it seems an incredible cool solution to the problem that there aren't roads everywhere: the whole animal can transform from wheel form to walking form and back. It's almost like a biological Transformer (TM).
Many people will still want their own car for comfort and status.
Most people will still travel to and from work during the traditional rush hours.