There's significant trouble with routing, delays, and reliability: A self driving car does nothing naturally different that a cheaper Taxi, and you can see the lack of popularity of the carpooling offerings Lyft and Uber had.

Ultimately the argument against lower congestion is even easier. Self driving cars for individuals make traveling less annoying than driving. So what we get is more tolerance for long commutes, and ultimately more miles traveled, with ups congestion. Now Timmy can go to his activities by himself, so mom goes elsewhere. Again, when we make something cheaper, we increase demand.