* Take surface space that can be better used for walkable densification, or are already taken by roads.
* Produce significant atmospheric and noise pollution, even from EVs.
* Conflict at cross-traffic, especially so for dense road networks like grids.
* Have to navigate around building geography, so for example can't support high speed turns.
* Conflict with pedestrians so are limited in speed and have to support frequent stopping points.
* Have no central routing and control for journeys so suffer unbounded traffic, even where additional vehicles lower capacity.
* Are hard to add to existing developments that need additional capacity.
* Don't have out-of-line stations for on- and off-boarding.
Add to this the differences you've already mentioned, like uniform taxies rather than mixed private cars, a comparative ease of automation, and being additive to existing network capacity rather than conflicting with it.