Even on planes, we've had problems with people being too reliant on autopilot.
http://99percentinvisible.org/episode/children-of-the-magent...
With self driving cars where we're relying on the car for moment-to-moment instantaneous reactions, I think it's very unlikely that human drivers will be able to pick up the slack in a system failure. If you're expecting the car to stop it self and it doesn't, you have what, a quarter second to jump in manually?
The recent video of a self driving uber going through a red light is probably a similar scenario. Even though uber insists it's a "driver assistance technology" and not a self driving car, the driver expected the car to stop itself, so they didn't bother applying the brakes. And that one was at low speeds, maybe 10 mph.
The saving grace here is that they're competing against human drivers. The technology doesn't have to be perfect, just better than us. But relying on human oversight to maintain 100% perfect attention for a six hour journey of doing literally nothing but watching a car drive itself in case you need to step in? I think that's a non-starter. For these to work, the car has to be able to manage on its own.