http://www.nytimes.com/2003/03/17/business/betting-speed-pur...
Comcast made a big bet in 2003 that many other companies did not make. Demand for broadband has been strong, but it was far from a given at the time Comcast made the investment that it would result in strong returns well into 2017.
It's easy to look back with full knowledge of how things played out and determine that Comcast was unsophisticated in its financial and business planning.
Comcast has grown because it has provided a "neutral enough" internet for its customers. There is absolutely no reason to believe it would begin to act differently now. If anything, Comcast's strong profit margins increase the chances that other companies will enter into competition for the business.
I'd rather have Google invest money and brainpower into creating things like Webm than to pay lawyers and PR specialists to promote net neutrality.
The sorts of packet inspection techniques you describe can be thwarted by encryption, and are no different than what is being done within Google to monetize user behavior/preferences to the nth degree.
You argue that Comcast wants to charge content providers for preferred access. I'm sure they'd like to, but this is the case for financial, not technical reasons:
The following is the most important point: Once a service (like Youtube) becomes popular enough, the traffic pattern of packets going to that service begin to resemble the traffic pattern of the internet itself, making arbitrage impossible. So the economics that once worked for ISPs to arbitrage based on upstream vs downstream bandwidth no longer work because upstream circuits are largely priced based on broad data access patterns which are heavily influenced by major content providers.
So the profit margin on residential bandwidth starts to approach $0 as 100% of the content consumption matches the exact characteristics of the upstream provider's pricing model, since upstream providers also do arbitrage, but with much less aggressive hedges and less leverage.
Consider it this way, you can start a competitor to the post office that is cheaper for some small list of addresses. Maybe you could profitably deliver all first class mail for 25 cents within a single zip code. But if you try to offer a similar service to all zip codes, you will likely face nearly identical constraints to the post office and you will not be able to undercut prices.
For ISPs, the more predictable customer demand gets, the more likely the arbitrage has already been done upstream, and the closer the residential circuit becomes to a dedicated circuit.
These days, suppose 90% of internet users follow a sine wave data usage pattern day after day that is divided between Netflix, Youtube, and a handful of other hosts. Video is massively more bandwidth intensive than other stuff so video content providers dominate. This means a lot less profit from bandwidth price arbitrage, which is why ISPs are toying with the idea of charging content providers for access (it's also a bit silly because the moment youtube goes down people would call in saying their cable modem was broken).
I don't think it's a bad thing for Google to help pay for the ISP's upstream bandwidth costs, since whatever contracts the ISP buys are largely serving the purpose of piping content from Google to consumers. It may or may not be extortion for Comcast to try to do this, but that should be up to regulators to decide after the fact. If this happened, by the way, it would likely accelerate Google's slow-as-molasses last mile effort and result in much more competition for residential ISPs.
ISPs provide a very helpful non-google intermediary for the small bit of our internet behavior that is not being captured and stored by Google. I think it is better to have a variety of firms, each with its own core specialty, fighting over this market than for regulators to simply follow Google's advice and hamstring ISPs into lower profit margins.
In an ideal world all internet connections would be unconstrained by QoS considerations. But if you think back to dated last mile tech like T1 lines, the financial arrangement is leasing the piece of copper wire and paying for a 24/7 circuit that can saturate that copper's bandwidth in both directions. For this reason, a T1 cost 10x the price of a residential connection supporting the same average bandwidth. If you are running a data center, this contract is important, you can't afford to deal with congestion and reduced bandwidth that is beyond your control. But for residential customers it is a desirable and worthwhile tradeoff that does not benefit from regulation whatsoever.