The thing you're not addressing is that there is a wide range of conditions and activities which all fall under the title of "driving". Would I trust a driverless car with current technology to navigate a busy parking lot? Not a chance. Would I trust it with the 40 mile commute I had, 38 miles of which were 6 lane highway? Probably. Would I trust it to make that same commute at times of low traffic when I'm likely tired myself? Absolutely. Not all driving needs a human's full attention, and the situations where a human doesn't need to pay full attention can be handled better by a computer which is immune to many of the distractions and weaknesses of the human mind. It's all situational and difficult to figure out, but the proper response is to keep plugging away at the parts of the problem you can gain some traction on and take what you can get.
There are cases like the busy parking lot where human control will likely be necessary well into the future, there are cases which can be addressed today, and there are cases which were addressed successfully over 20 years ago. Look up Ernst Dickmanns - he was an early pioneer in this field, and had an autonomous vehicle driving on European highways in traffic in the 80's. It's really easy to say that this is an impossible problem to tackle, but the person who says something can't be done should make damn sure he's not getting in the way of the person doing it.
I did my M.S. on this topic, so although I would not claim to be an expert in the field at Google's level, I do believe strongly in the technology and have given it more than cursory thought. The larger problems are more societal than technological, and as such I really hope that Google makes this happen because they have the societal clout to get people on board with the idea that a computer can, for most of the miles you drive, do a better job of getting you from A to B than you can.