> without being able to efficiently price the scarce resource via bandwidth caps and segmented pricing models on the consumer sideFirst of all, the situation in the US is different than in the UK, as I understand it. I don't know the details of how ISPs are able to price in the UK, but in the US, the ISPs can price however they want to, and many of them have what is essentially monopoly access to their markets, so the pricing scheme that is in use in the US, with mostly flat rates and very little segmentation, is, I am sure, to the ISPs' advantage, whatever their publicity might say. I suspect that the flat rates are set so that, while people who use full video streaming bandwidth regularly might be subsidized to some extent, people who use much less bandwidth (like me), are paying far more than we would in a true free market, so that on net, the ISP is making much more profit than they would if they truly had to price by bandwidth actually used.
>the ISP is reduced to negotiating with the upstream providers of packets that are consuming the scarce resource on behalf of their customers.
No, they are "reduced" (at least in the US--again, I don't know what the situation is in the UK) to trying to get upstream providers to pay for bandwidth that their downstream customers, on net, have already paid for (as above). The logic of giving ISPs exclusive access to particular geographical areas (in the US) in the first place was that they would be able to count on a steady revenue stream to fund buildout of the necessary bandwidth. They have now had that benefit (in the US) for at least a couple of decades. If they are complaining that they need upstream providers to pay to fund more bandwidth, that tells me that at least one of two things is true:
(1) The ISPs have not used their monopoly revenues for the purpose intended: instead of keeping pace with bandwidth needs, they have just been pocketing the cash; or
(2) The ISPs actually have the bandwidth, but they are using their monopoly position to control access to it in order to get paid twice for the same thing.
(My personal opinion is that it's a combination of both.)