It could definitely be one of those experience it to understand it things, since I've never driven very far in a car with more advanced self-driving (trying it in a colleague's Tesla on a short drive to lunch is about the extent of my experience).
I'm a private pilot though and what you're describing sounds a lot like the autopilots on small planes. It handles keeping course and altitude for you, so you can focus on the rest of the flight and it is indeed way less taxing than hand flying a longer distance. And like Tesla, a plane autopilot will occasionally do something weird so you can't entirely ignore it. I would think the difference though is that in a plane, nothing the autopilot does will be fatal or dangerous very quickly, so you have plenty of time to notice and correct it as long as you aren't sleeping or something. When you're driving a car though, a second of steering in the wrong direction can quickly turn into an accident, so you as the driver have to be constantly ready to immediately take over. Even in the chance is low, as long as it's often enough to be a factor, you still need that instant readiness.
I wonder if the value is more on longer drives, as you said, vs city driving. The fatigue factor driving between cities is a bigger issue, and the driving tasks are simpler with less opportunity for the car to do something crazy than in busy city driving.