I think it's crazy for reasons other people state, especially latency and connection reliability issues. But I think a related idea can work pretty well.
Self-driving cars are classified into 6 SAE levels, 0-5. 5 is full autonomy, 4 is autonomy in limited conditions, 3 is self driving but there has to be a driver ready to take over.
Getting to 5 is really hard. It could take decades, because there are a zillion edge cases. But what if you could turn cars loose at the "can handle 95% of driving" level and then let it call a human to give guidance when the car is nervous or confused?
Obviously, the phone-a-friend approach can't work for things requiring quick action. But if you bias the car to just stop safely and wait for orders, I think it's doable. E.g., the car comes upon an accident where the lanes in one direction are blocked. Humans know to go around the accident by slowly and carefully sharing the remainder of the road. So perhaps the car stops before the obstruction and asks for help. A human looks at it and draws in the path it should follow to get back to normal road.
So I think remote "robot coaches" could plug the gap between levels 4 and 5. And maybe some of the level 3 gap, too. And from what I hear of the Waymo operation in Arizona, I suspect Google's doing something akin to this.