"Ever" is a long time. Though I agree insofar as it's probably further out than a lot of people are assuming.
But, yes, it will be a long time before maps are sufficiently good across wide areas of the rural US (much less the world as a whole) to enable self-driving. The situation when you start talking about unpaved roads becomes even more problematic.