At some point everyone realized they could charge a massive premium per bit and as long as everyone did it, customers would have to pay. So here we are.
Even if a cloud provider had competitive transfer costs they likely wouldn't attract any new customers and would have less margin left over to subsidize the main cost customers look at, $ per instance hour.
The less attention is paid to transfer costs the better for AWS/GCP/Azure. Why hasn't a spot-market for transfer been introduced? Same reason why I can't sell my unused home internet bandwidth to my neighbors, the money is in controlling the means of transportation/communication and the providers want to keep as tight a control on that as possible.
(Source Open Guide to AWS - https://github.com/open-guides/og-aws)
Then again, I worked for AWS for years, so maybe I'm just used to thinking this way so I'm not really surprised by it.
You still see paying for bandwidth with residential connections, though some operators (like Comcast) are trying to do away with it.
This is just the static picture though. What's harder to predict are the consequences of some innocuous looking code change.
What I'm saying is that for a hosting architecture to make it difficult to predict the cost of any code change is a downside compared to an architecture that makes such predictions easy and intuitive.
Of course you will try to mitigate any downsides and learn what you can from any mistakes. But unpredictability makes learning far more difficult than it should, which inevitably means a waste of development resources.
The issue with per-bit pricing is that a fair agreement for network use would probably look like paying a fee that makes up for the amortization of the network equipment. Anything else is an artificially restricted market created in an attempt to extract more value out of consumers by having them bid against each other.
At some point, yes, we will run out of places to put the switches and routers and then the cost of connectivity will be closer to the cost of land use and will mimic rent, but we are a ways away from that.
Enron tried to create a market for this.
[0] https://www.wired.com/2001/11/enron-a-bandwidth-bloodbath/