I feel very much the same way. A significant portion of drivers also deal with winter/icy conditions for at least part of the year, and I have a really hard time believing that self-driving cars will be able to handle those conditions well, at least not for a long time. You have snow falling, snow covering many/most of the lines on the road, varying degrees of traction - added to the uncertainties of pedestrians, construction, other crazy drivers, etc. Heck, the sensors alone are going to have an inch of snow and ice on them after being outdoors for a couple of hours. Winter driving is hard enough for highly-focused human drivers, making decisions that are often based on factors that don't easily translate into algorithms and computer models.
Google speaks of a need to leapfrog the driver assistance systems and instead focus on fully-automated driving systems because the driver assistance technologies won't incrementally improve to the level we need for full driver-less cars[1]. I think that's a fair point, but I'd go one step further: self-driving cars thrown into a mix of human-operated vehicles and roads that aren't specifically designed for self-driving cars is (in my opinion) a similar mistake. I believe we'll reach a point very soon where great technical advances in the vehicle alone will have diminishing returns in terms of safety on the road.
If we are going to solve this problem once and for all, I'd be much more enthusiastic about a self-driving system that operates only on distinct roads/lanes/tracks that are designed only for self-driving vehicles. Perhaps that's an unreasonable expectation, but trying to design systems that anticipate and make decisions based on the (often irrational) behaviors of other human drivers seems no different than designing a full human-level artificial intelligence - and even then, it'll still be prone to mistakes because other humans are involved.
Either we get to the point where all vehicles on public roads are networked and cooperating through software to self-drive, or we're just making guesses about other people's behaviors. As Chris Urmson puts it, "just making the cars [and I'll add, roads] incrementally smarter, we're probably not going to see the wins we really need."
[1] https://www.ted.com/talks/chris_urmson_how_a_driverless_car_...