So humans that can barely drive a single vehicle will now be responsible for monitoring 20 vehicles in highly different environments that change at a moments notice?
As soon as the car sees it cannot proceed properly, it parks somewhere or go in a safe situation and requests a remote human operator to intervene. Then the whole point is to have a call center sized so that you do not have to wait too much.
We're not there yet, and giving back the control to the passengers is also a possibility. However, in a world where passengers almost never drive, it might be safer to give the control to trained remote operators.