Is the current pricing structure simply because consumers have become so accostumed to playing a flat rate, or is there something else, that I am missing?
Because "speed" sells, and the customers Comcast wants don't use much bandwidth.
The #1 source of revenue for Comcast is the customer who has cable, has high speed Internet, doesn't really use the Internet for much, and just pays their cable bill every month without really thinking about it.
The family with 3 teenagers all watching Fortnite live streams or whatever the kids are doing these days is Comcast's second worst customer, only superseded by the guy streaming those videos.They're taking up a significant chunk of their area's local bandwidth, and that costs real money to upgrade. Comcast has no incentive to allow for this use case unless they can find a price tier that makes you a good customer again (notice that for $50/month, Comcast will remove the bandwidth caps).
It's effectively the same reason that Comcast doesn't let you sign up for individual cable channels.
... But that being said, I certainly can relate to what you mean. Charging by the megabyte would require a very, very low price per megabyte for this to be acceptable.
But wireline carriers have seen how poorly tiered data was received in the wireless industry by consumers, and I think it'd take an act of Congress to divorce them of being able to use the word "unlimited" in marketing materials, no matter how fraudulent the claim might be.
(They already do, read your terms of service carefuly)
Even with overselling, their network is still bound by that capacity. So they charge for both.
Because they can and do charge you for both and laugh all the way to the bank.