Like the uncanny valley, the edge cases for self-driving cars will take more work than the rest of the effort combined... by at least an order of magnitude.
There is far more money in automating long-distance trucking. My bet is the first real use of self-driving "cars" will be for trucks. The human driver can sleep, eat a meal, etc but will remain on-board for safety's sake and dealing with unanticipated edge cases (e.g. the AI wakes up the driver when it runs into a traffic jam due to an accident. The AI can probably recognize cop cars, flares, etc but won't be able to navigate the scene of the accident while responding to instructions from police officers)
The next small step will be "follower" trucks. The lead truck has a human driver same as before, but one or two AI-driven trucks follow. They do their best to track the human driver's response to unexpected situations. Worst-case, they pull over and flag the human driver who pulls over and uses a VR rig to pilot them past the problem.
Meanwhile the application of "self-driving" technology to cars over the next 10 years will be to avoid accidents, not to actually be self-driving.
Will we have fully autonomous vehicles within 50 years? Almost certainly. Will we have them in the next 5? Almost certainly not.