I've wondered about this for a while, particularly with cars designed outside of the US. Much of (western) European[1] road infrastructure is in much better shape than in the US. Since this development started to surface I've thought Michigan would be the perfect testing ground for the technology. Drivers not only have to contend with poorly marked roads, but roadways that for 9 months out of the year significant portions are under resurfacing/construction, requiring one to carefully hug the roadway due to lack of shoulder or any number of one-off situations that would be difficult to account for in software.
Then there's the pot holes. On M-59, an 8-lane non-highway state road, you have to constantly adjust your position in the lane to avoid them. In many cases they are bad enough that if you hit one at a reasonable velocity, you'll suffer a full-on blow-out and run the risk of completely losing control of the vehicle. Parts of it are like playing a video-game (but with real life, costly, consequences).