The thing is, I see no way to have full self driving without AGI. And I don't think humanity is anywhere close to developing AGI. Without AGI you can have level 4++ self driving, maybe, but not level 5.
Why? AGI seems like a significantly harder problem than self driving cars (itself a hard problem admittedly).
What I personally think will happen is we'll meet somewhere in the middle: we can redesign roads/cars to make solving the problem of self driving easier.
But I agree with you that if we meet in the middle, that could actually work well.