But the proof is in the pudding, the speed tests speak for themselves, no?
That's exactly it -- multiple, independent yet individually unreliable measures can actually produce very reliable aggregate evidence through convergence. This is particularly true if one uses abductive reasoning, but IIRC, there's statistical proof behind this idea as well...
That is to say that the speed tests say something and the chat says something, but together, they begin to paint an troublesome picture. Combine that with the timing and it starts to look bad. More data points will add to that convergent picture, of course...
We need this to be verified by someone with intricate knowledge of whatever is going on inside Verizon's network. The number of people with the exact knowledge of what is going on will be necessarily limited.
1) Verizon decided to enter some router configurations that affect the packet forwarding to Netflix (well, most likely FROM netflix, as the internet is not actually symmetrical). What we should see as a result of this is a sudden change in bandwidth.
2) The load on the interfaces in between is changing, either because of more netflix subscribers, more AWS traffic, or more netflix traffic. Since these are things happening slowly over time, you'd see a gradual worsening of traffic conditions.
I think it's pretty fucking likely that what's happening here is 2). Now granted, Verizon could solve this problem by cooperating with Netflix. But for them that's an expense with no upside, so likely they're demanding Netflix make it worth their while to upgrade the interconnects.
This is one of the basic problems with the internet today. It used to be the case that bandwidth between providers was much more plentiful than last-mile bandwidth. So we upgraded last-mile bandwidth. Problem : doing that resulted in an exponential increase in load on the core network and the interconnects between different networks. Needless to say, nobody can keep up with exponentially rising demands (especially without subscriber growth). The basic problem is that a linear increase in bandwidth for the customer results in an exponential increase in costs for the ISP (so -surpise- small ISPs don't feel the effects nearly as badly).
So ISPs are desperate. Their costs are spiralling out of control, with everyone demanding they follow this exponential curve, but of course their customers not willing to pay for it (and all the money in the world would only buy time if thrown at this problem). Demanding the ISPs pay for it is simply not going to work. Right now, yes, they could theoretically pay for it, but that won't last long.
Plus Netflix is being a rather bad netizen themselves. The polite thing to do is to carry the traffic to very close to the user on your own network, and only use other's links, especially transit, as a last resort ("cold-potato" routing, for content providers). Netflix has caches, but no own network.
There's other problems that result from this cost problem. Already transcontinental and other long-haul links are strained to the limit. This has resulted in a massive degradation of non-local internet traffic, and it's getting worse fast.
It is really a very simple problem. You have consumers, and you have producers. And you get more and more of each. To a limited approximation you can assume everyone on the internet is both. To simplify things, let's say everyone uses a tiny amount of everyone else's services. Now given N participants in the network, what is the load on the core network ?
N * (N - 1) =~ N^2 (assuming full duplex links, which would be generally correct for the internet)
So adding more participants in the internet increases resources required to give every existing participant in the network his "old" speed. These need to be paid for. But unless everyone want to start paying an amount for your internet connections that rises with the square of the size of the internet, we're going to see progressively worse and worse throttling. Right now the strategy is fast becoming limiting interconnect capacity to prevent the core network overloading, which would have far worse consequences.
While, yes, historically this has mostly been monopolies cheating the market, that's less and less true. This is going to get worse, fast. And since those resources are obviously not going to be made available, there's only one thing to say :
Get used to it.
> N * (N - 1) =~ N^2 (assuming full duplex links, which would be generally correct for the internet)
1) N^2 is not exponential, it's polynomial.
2) Expanding capacity isn't even N^2, it's just linear.
If you double each residential customer's bandwidth, you "only" have to double the capacity of your uplinks to other networks. The number of endpoints or total nodes in the network is entirely irrelevant, because you have the same 50Mbps whether you're drawing it from one AWS server or a thousand BitTorrent peers. The only way you would get the result you're assuming is if every endpoint had a fully meshed connection to every other endpoint, i.e. each customer gets 50Mbps to each Amazon server, a separate 50Mbps to each Google server and a separate 50Mbps to every single other user on the internet, for a total of several hundred billion Mbps for every customer. That's not how it works.
And in practice it's even less expensive. It's sub-linear. Because just giving everybody a connection which is twice as fast doesn't mean everybody is going to immediately double their usage. A large fraction of users, given a faster connection, will do with it only what they currently do with the slower connection. Their pages will load faster but they won't load more pages, so the load they put on uplinks to other networks will remain the same.
There are no exponentially increasing costs. There are only the same costs as there ever were: Cisco et al come out with a new router that puts twice as many bits through the same piece of cable and if you want those bits you pay them a fixed cost and swap out your old model for the new one, and in another few years you do it again.
My understanding is that peak load is the issue, not total data transferred in a month. If Netflix downloads/buffers video content as fast as it can, and people mostly use Netflix at the same time of day, then giving them more bandwidth is a potentially big problem
The argument they might make is that their services keep the data closer to the users, but that's such a cop out. They could do the same thing for Netflix with transparent proxies if they wanted to (but the result would be to make Netflix more usable instead of less, which is adverse to their interests as competitors). And in any event the uplink to a large peer is a totally inconsequential part of the cost of operating a network. The thing which is expensive is upgrading the last mile to handle a large amount of video traffic, but that cost is the same whether the traffic is Netflix or FiOS TV.
In the UK, I believe the main practice revolves around broadband, whereas in US it's all around phone (VoIP or otherwise). If you could enable competition within the same last mile service, you'd a)give a definite incentive to make sure the entire network delivers the best experience and b)have alternate sources of revenue to maintain the last mile network.
The current duopoly that exists SoCal in most areas (1 cable provider and 1 DSL provider) is a really terrible consumer experience. For those who have access to FiOS, it's a tad better, but not by much.
If you had access to 10 providers at once with comparable services, there would be real incentive for them to offer the best possible experience and lowest possible prices to consumers... but of course that would hurt corporate profits and so on and no politician getting those donations wants that...
The problem with just local loop unbundling is that the last mile provider has the incentive to favor their own services. They can operate their own ISP division at a loss or with zero margins (more than made up for by profits in the last mile division) which disadvantages competitors who have to lease the last mile and thereby maintains the status quo of no competition.
The answer is that there exists a box that mounts on a telephone pole. The purpose of the box is to terminate hundreds/thousands of DSL lines and connect them all to the central office with a single fiber optic cable. It allows you to get 30Mbps out of your 30Mbps DSL instead of getting .8Mbps, because the box is close enough to your house to get full speed unlike the central office. It saves the phone company from having to string a new strand of fiber to the premises of every individual customer who lives too far away from the central office for high speeds over twisted pair copper.
The first generation would have been actually installing the boxes, which they apparently never even did. Then you periodically upgrade the boxes to provide faster speeds as the technology improves and/or the cost comes down, e.g. from 1.5Mbps DSL to 3Mbps to 6Mbps to 12Mbps etc.
However, theoretical networking problems like this have been solved many years ago. The Clos Network can provide full duplex links at significantly better big-O
http://en.wikipedia.org/wiki/Clos_network
I don't fully remember the details, but I believe you grow at approximately N*log(n). The problem is that the original Clos paper was published before big-O notation was invented... and that this was just "one other homework problem" that a professor gave to me about 5 years ago when I was in college.
So my memory is fuzzy, and the math is undocumented on the internet :-(
The internet is built on top of unreliable datagrams, which means the connectivity problem is even simpler. It satisfies the conditions of a Rearrangeably nonblocking Clos network. So the Big-O is even smaller than the above networking problem.
Clos wrote his paper specifically for phone connections. You cannot disconnect people for no reason while they're in the middle of a conversation. However, unreliable IP packets can be disconnected and rearranged, allowing you to use a cheaper form of the Clos network.
Either way, (like my professor from half a decade ago...), I'm going to leave the asymptotic complexity of "Strict Clos networks" and "Rearrangeably nonblocking CLOS networks" as an exercise up to the reader.
Mostly because I don't remember the solution... As a hint, replace the crossbar switches (in the wikipedia page) with a recursive Clos network and solve for the Recurrence relation. Use a 2x2 crossbar as the base case.
In other words, core network capacity (meaning total bandwidth in all links) if you have a clos scales even worse than n^2, with n the bandwidth you deliver to users. Even worse : you need as many long-haul links as you have subscribers (because your subscribers are connected to different locations).
Is there question in your mind as to why ISPs don't do that ?
The main driver behind increased internet traffic is the migration from television to internet for video content. I think that trend is going to continue over the next decade or so until everyone uses the internet for video content. But after that I think you'll see internet traffic level off to an extent.
ISPs need to build out their systems to handle that. I can't see anything else causing a dramatic rise in traffic so once they build out their system to handle video content it should be good for awhile.
Unfortunately, most ISPs are also cable companies, so naturally they are resistant to spending money just to shoot themselves in the foot. But this trend is going to continue whether they like it or not. It's probably going to be a really painful process for the ISPs and their customers. Netflix is going to be trapped in the middle. The ISPs will want to choke netflix out and take over their business. With net neutrality gone they just might.It's going to take some good PR and strategy for netflix to come out of this fight alive. If I was a betting man I'd say they'll put up a good fight but eventually be bought out by one of the big ISPs.
Isn't the cost to deliver a gigabyte of traffic something less than five cents, and decreasing?
The real cost is building the infrastructure to handle higher and higher peak bandwidth. Suppose you're Comcast with 20 million subscribers, half of whom are trying to watch 5Mbit Super-HD Netflix during prime time. That's 5 terabit, and even if you could agree with Netflix's providers to divide it up geographically, it's still an outrageous amount of bits to figure out how to ship.
1. Increase the price for the subscription
2. Remove unlimited bandwidth for the subscription and go back to volume based models
It's pretty simple, really. You offer a product where cost is higher than revenue? Change the product, change your process, or change the price.
Like everything else, there's fine print: 20Mbps or whatever is described as a maximum speed, and many people actually see burst speeds that high. But as it would be unreasonable to be upset you can't get 20Mbps to Nigeria, it's also somewhat unreasonable to be completely upset at Comcast that you can't get 20Mbps to Netflix if Netflix is paying for cheap ISPs that won't fairly peer. I'm not sufficiently informed to take sides in that dispute, but I can easily imagine that Cogent (for example) is dumping way more traffic into Comcast than Comcast is dumping into it, and trying to get away without paying for that disparity.
No, their advertising just needs to be accurate! They need to say "250Kbp bandwidth, max peak 20Mb". The real number has to be the lead, not the BS peak number. There's a parallel here to audio amplifiers. Advertising the peak wattage and not the RMS is stupid and deceptive.
The bigger issue, however, is whether those limits are based on the underlying limits of the network or artificial caps: this is currently completely opaque. It'd be much better if they were forced to publish any traffic shaping performed so the consumer can actually make an informed decision. This might lead to other questions such as whether they should be refusing to deploy Netflix OpenConnect nodes which would be healthy – and no doubt a key factor in why they won't talk about it unless forced.
Not if you judge by the telcos' profitability.
The cost of transferring each bit is exponentially decreasing as network equipment tracks Moore's law. Their costs are going down, not up.
What this is really about is damaging competitors to their cable TV operations. Wail about Netflix using a lot of bandwidth, never mind that the cost of providing that bandwidth is falling as fast as the demand for it is rising, and you can put on a good show for the regulators as to why you need to destroy Netflix and push everybody back onto U-verse and cable TV.
Today this has been twisted into a false narrative: "the teclos got paid 200 billion to build fiber and never did".
Saying they are 'desperate' makes them sound fearful. Verizon made $70B in gross profit last year. I can't see how they would be afraid of upgrading their network - it's part of the business.
See page 10 here: http://ptt.br/pttforum/6/doc/Netflix-Open-Connect-Network-BR...
"So, how does this analysis account for the fact that, in many reported cases like this, other services work just fine at high-speed? Your point 2 is that more consumers and producers are flooding the net as a whole, but that would cause widespread issues, not just premier services issues, right?"
https://signup.netflix.com/openconnect/faq
This is a boon for indie ISPs who are too small to host a cache as the bandwidth is cheap and it makes Super HD streams available. I wouldn't call NetFlix a bad netizen.
It is also worth noting, that FIOS was #6. I do wonder if that has really changed or not.
But you're asserting the truth of this position, not its probability, and I don't think anyone participating in this discussion thus far has sufficient information to do that.