Though, your comment on latency did give me an idea. What if we were to co-locate the person monitoring the system within the car itself? That would free up the driver to do other tasks, and would avoid the latency issue entirely.
Granted, there would be costs associated with having a co-located monitor of the driving, but that's something we can solve by scaling the problem up. Similar to running multiple virtual containers on the same physical host, we can have several virtual cars, each with their own driver, instantiated on the same physical hardware. This setup would only require a single co-located monitor per physical hardware instance, rather than needing a monitor for each virtual system.
Extending this analogy to no-server solutions, there's probably a market for no-car solutions. If I have a sudden need for transit, but don't have a car available, I could rent a virtual car on top of existing physical hardware, without needing to bring my own car.