"we would never build a system that might “learn” that people whose destination is a hospital tend to be willing to pay higher fares"
I'm sorry, but wouldn't that exactly be the point of "surge pricing"? To allow people who REALLY need a ride to be able to pay a higher price and get where they need to be?
"Surge pricing" would kick in because of scarcity, as in, not enough rides available for everybody.
The person who needs to go to the hospital needs it more urgently than the person who wants to meet her friends at the pub, so she should be able to pay a higher price to get where she needs to be (the hospital).
The alternative is that she doesn't have the option of paying a higher price out of "ethical considerations", and has a high probability of not actually finding a ride to the hospital. How is that the better variant.
If you want to get all ethical about it, maybe you can set up a system where people get their money back if they can prove they needed to go to the hospital (or health insurance could cover it). But it is unlikely to cover all possible cases of urgency ("needed to flee from a terrorist attack", "needed to catch the love of my life before she marries somebody else" and so on)
. Much easier and more effective to regulate by market prices.
It seems to me yet another case of misguided socialist ideology causing harm in the real world.
An even more cynical interpretation would be that companies prioritize "looking good" over saving lives, so they rather implement the virtue signaling "no surges for hospital rides" than actually letting people catch their ride to the hospital.
If you are worried people might not be able to afford the ride to the hospital, you could give them credit.