Guess we'll need to agree to disagree and see how the situation shakes out, as we run the "in vivo" experiment.
In the area where I live (N. VA), there are already an astonishing (to me, anyway) number of people who drive an hour or more each way to work. 2+ hours a day of just piloting a car down the highway, frequently in heavy traffic. This isn't especially uncommon or anything -- it's not the majority (thankfully, due to all the externalities involved), but it's not rare either. People generally do it because it allows them to buy a significantly bigger/nicer/whatever-er house out in the exurbs than in the inner suburbs, for a given amount of money. I think it's an insane tradeoff, but people are already doing it.
If you didn't have to drive the car, and could basically reconfigure the car's interior to some sort of fully-reclining "pod" like the very high-end cabins of transcontinential airliners... it'd seem a lot less insane. I could imagine dragging myself up at oh-dark-thirty much more easily, if all I was doing was staggering from my bed to a vehicle, where I could pass back out until I was five minutes out from the office.
The afternoon commute would still be rough, because I don't really do naps, but a self-driving car would let you legitimately work from the road; it wouldn't be that hard to stack up a few hours of email, coding, or other non-face-to-face work to do on the way home.
All of a sudden, living in West Virginia and commuting to DC doesn't seem entirely impractical -- and that's probably a nasty problem for transportation infrastructure if a bunch of people all decide to try it in the near future.