Latency prohibits "just" remote driving cars, the car needs local smarts to drive itself.
There's not a hard line. There's a degradation in response time, and so old people drive slower (at least, they drive slower if they're wise). And then there's a point where even that isn't safe, and they need to stop.
But yes, not everyone can drive. Some are too old. Some are too young. Some can't afford a car for the amount of time they need to use one. Some are dealing with a temporary thing, like a broken leg or a bad sprain, and they can't drive just for a few days, but they still shouldn't drive right now. Some are coming back from a bar, or from a medical procedure involving anesthesia. (I forget the procedure, but I was told "don't sign any legal documents in the next 24 hours" along with "don't drive".) Lots of people can't drive for lots of reasons.
But, you know, we used to have things called "cabs" that helped with all of that. "Not everyone can drive" is orthogonal to "should a car be self-driving, or should a human drive it".
Self driving cars offer the chance for inexpensive point to point transportation. Just because humans were once needed to drive cabs doesn’t mean they always will, any more than we need elevator operators anymore.
I mean, it's absurdly expensive except for all of the other types of car usage right? Companies have been pushing car subscriptions for years because they sound nice: "Don't worry about maintenance, just pick a new car every so often!". Except that it turns out to be even more expensive than just owning a car.
Leasing (unless you have some tax advantage) is typically more expensive than owning.
Ride hailing apps are great, but I don't know anyone who has run the numbers and decided it makes more sense for a daily commute than owning a car (maybe someone with no or extremely expensive parking options?).
I'm skeptical that using a self-driving car service from a profit-seeking company would end up being less expensive than owning your own self-driving car, but maybe we'll see.
NB: I did not used to think about this much until I recently went to price an extended warranty for my low-mileage car: all of the plans were denominated in years _or_ miles, which meant manufacturers price maintenance in terms of years owned, regardless of usage. The same is true with insurance, which is a huge cost for most drivers: some insurance plans offer a modest discount for extremely low mileage drivers, but to a first approximation the fixed costs of simply owning a parked car are huge. Basically the only saving for owning a parked car is fuel (which can often by offset by the cost of parking, whether that's born by the owner or subsidized by the city.)
Tesla claims their autopilot is much better than humans - but unbiased sources that should have data are not talking.