It’s gonna be decades before...
We can very accurately identify children, pedestrians, cyclists, and street vendors with machine vision. And with the LIDAR that Waymo is using, very accurately tell where they are and more importantly where they're going. Stopping for those things is pretty low on the "problems that need more work for Level 5" agenda.
Imagine driving slowly through a large gathering of people, slowly having them move around your car, etc. Such situations will necessarily become major issues.
Especially if you imagine a snowy dense downtown street, with pedestrians crossing everywhere and snow all around you making the LIDAR useless.
You're very right that OD works great, that doesn't mean you can train a neural network to make the right decision though.
Find the path to where you want to go, avoid the parts with negative weights.
Again, the hard part is not "here is a human, avoid it". That part is done (in Waymo's case).