This is imo the biggest problems that forces users from not buying the tech. I used the beta, and not only did I have to basically pay attention, but at times, when I was already paying attention, Tesla would ask me to apply pressure on the steering to the point where self driving disengaged and I had to take quick actions. On top of that, just driving around my neighborhood, it failed miserably. It couldn't park itself when the entire curb was empty and I stopped in a non ideal spot. It would often incorrectly identify parked cars on curved streets and brake. A couple of times it almost ran over pedestrians at a stop sign, mostly because it accelerates too much on turns at a stop sign. It absolutely refuses to drive faster than the speed limits, going 25 on two lane main streets where everyone expects you to go 40.
At some point, its more stressful for the driver on self driving than driving yourself.