FCC gets Comcast, Verizon to reveal Netflix’s paid peering deals
arstechnica.com
arstechnica.com
Would the consumer be responsible for paying for: -DDOS (not only can you knock them offline, you can cost them big money) -Spam (Both received and botnet sent) -Retransmissions for traffic the ISP dropped due to congestion -Ads (adblockers could reduce bandwidth costs by a third)
I personally would love volume pricing. But I also would not want to provide tech support next time my parents get hit by malware and get hit with a $1000 isp bill.
It's the incentive that's lacking at the moment. My machine may send out bursts of spam at 3am, or a website ad may weigh 10MB, but as long as I can still check Facebook what do I care? But I would care if I were billed for those transfers. I'd start shopping for devices that helped me self-throttle. I'd start caring whether websites were bloated. That would pass the incentive on to those providers, etc.
I get that once the traffic hits the broader 'net that all bets are off. That's true and nothing can be done about it.
But in this case the ISPs aren't even making a "best effort" on their own network. They're deliberately not upgrading congested links where their peering counterparties are willing to upgrade. That is not a "best effort" class of service.
There is a simple solution to unbalanced Peering that Netflix could implement – just setup their clients to upload dummy garbage data at 70% max uplink speed whenever their client software is running. All of a sudden traffic will go both ways and they will be able to peer as an equal party.
This alone proves Peering agreements based on traffic amounts are a joke.
Streaming service that generates one way traffic is simply 'too nice' to comcast. Flood those asshats with /dev/random.
Even if the US government did somehow manage to regulate the US-based telcos, the FCC would have no legal way of regulating net neutrality outside of the US. An effective net neutrality policy could even harm our economy, if its a rule that nobody else needs to play by.
If we want access to a free internet, we need to figure out how to democratize the infrastructure. As long as we depend on fiber and copper controlled by big companies, net neutrality will continue to slip through our fingers like water.
To some extent. This was discussed extensively in the recent Verizon-FCC lawsuit.
On US soil with US citizens and US companies, it should be simple. But it is a complicated world in which we live, so it sounds like the FCC could only really protect equality in the best of circumstances.
For the sake of conversation, lets say the FCC isn't powerful enough. Lets say the internet is just too powerful, and there is no hope of the FCC or any other organization to regulate it. Would net neutrality be lost forever? Is it lost already?
> FCC
> Comcast
> Netflix
It depends on how the FCC classifies the ISPs. If the FCC classifies the ISPs as a common carrier, then yes that will basically force them to honor net neutrality. Due to <politics>, the FCC has been reticent to classify the ISPs as common carriers (instead classifying them as infomation services), but it has tried to regulate them as a common carrier in some ways (but not in all ways), ie net neutrality.
Recently verizon took the fcc to court over this and the gist of the court's ruling was 'The FCC has the power to enforce net neutrality on the ISPs iff it designates the ISPs as common carriers.'
> An effective net neutrality policy could even harm our economy, if its a rule that nobody else needs to play by.
An effective net neutrality rule could just as easily bolster our economy if no one else decides to play by that rule. What makes you think that net neutrality isn't a competitive advantage?
> If we want access to a free internet, we need to figure out how to democratize the infrastructure.
It's hard to democratize the infrastructure because it is so expensive to keep and maintain, short of nationalizing it like the interstate system, though based on the previous paragraph I don't think that's what you meant.
Why do you think that?
Can you expand on this? I don't see it. What does it matter to America, if Brazil allows telcos to meter speed to subscribers based on bribes paid by third parties?
For instance, let's say Brazil's telcos started throttling their citizens' access to Netflix, but not to Braziliflix, a hypothetical Brazilian version of Netflix. That could prevent Netflix from gaining traction in Brazil. Meanwhile, Braziliflix would be free to compete on an even playing field for customers in the US.
Meanwhile, consumers in Brazil enjoy shitty services and exorbitant prices...
Would a FOIA request on the FCC for them since they are technically Comcast, Verizon copyright?
Still, I'd be curious to see if a FOIA request would work.
But there's a specific exemption for: "Trade secrets and commercial or financial information obtained from a person and privileged or confidential, 5 U.S.C § 552(b)(4)"
Years ago I lost a fight with the FCC for data on cell phone carrier reliability. The FCC has great insight into who has the fastest or most reliable cell network, but as part of mandating outage reporting, Congress promised the carriers everything would stay secret.
EDIT: I'll even give you some of my credit I have with them to use for your request.
Somehow it doesn't make me feel much better about it.
I mean if Netflix is paying Comcast $2k/month to fix the issue that's stupid but not very bad. If it's $2m/month that's a huge problem.
The issue is that the last-mile ISPs have hardware that gets overloaded by the responses to the requests their clients are making, e.g. Bob sends out 10 packets to Cogent, and Cogent sends 10 million packets in response.
The last-mile ISP isn't prepared to deal with those 10 million packets all at once.
http://www.comcast.com/internet-service.html
If they advertise they can provide the service then they should be held accountable.
If they say they can't provide the service, then they should not be allowed to advertise that they can.
This is quite literally a "have a cake and eat it too" type of situation.
How is NetFlix any different? They make the choice to lower the cost to deliver their service by choosing a CDN with inadequate existing peering relationships with ISPs. They also severely limit client side functionality because it makes their service cheaper/easier to support on a wide range of devices. NetFlix collects this data about ISP performance so why don't they say 'oh hey sorry ISP X user. We can't let you sign up because you'll have poor performance!' -- instead they happily take the money.
Netflix has Internet Connection Speed Recommendations (https://help.netflix.com/en/node/306). ISPs offer service plans that cost more for higher maximum speeds. Is it Netflix's responsibility to ensure particular subscribers receive particular qualities of content, without regard for the contents' source qualities, the subscriber's bandwidth, the policies of their ISP, or any other range of factors in the last mile? Netflix is offering content, not quality. It doesn't sell differently priced subscriptions based on "maximum video quality", it offers a range of bitrates at a flat rate and sends the one that allows the best experience for your bandwidth.
Although, that's sort of the issue, isn't it? It's not "your" bandwidth that determines that now. It's not the plan you're paying your ISP for that's the limiting factor, it's the ISPs particular traffic shaping policies. Netflix isn't slowing its traffic down or decreasing video quality prejudicially based on your ISP or internet plan. It's the opposite, but they're being asked to pay, rightfully or not, so that some of their subscribers won't have their bandwidth throttled by their own ISPs.
I think ISP should sell "25 Mbps" connection and "Up to 25 Mbps" connection, priced differently. Then customers will (I predict) choose "Up to 25 Mbps" connection.
Because the interconnection between your transit provider and Comcast is frequently saturated and Comcast is not going to fix it since they would prefer to be paid.
Net Neutrality would be a good step. Common carrier status would be better.
I don't want to spite Netflix at all, I just don't want my network flooded.
(I probably also shouldn't have promised folks I could deliver all of these packets, either, but that's another issue)
You have different incentives than people trying to protect their transactional VOD content businesses.
The "flood" is actually your paying customers who thought that you were telling the truth when you promised them X/Y internet service. They are trying to utilize the service they have paid for and you are preventing them from doing so.
To not provide them with the service that was promised could be construed as fraud informally if not legally. I'm sure there is plenty of language in an ISPs retail contract that says that customers can fuck right off for any and all reasons and no matter what the ISP is not at fault. But while it might be legal it's definitely dishonest.
The issue is, what triggers the costs? There's no good definition of what threshold requires a content provider to connect to the ISP network. Because ISPs are consolidating so rapidly, there's a huge opportunity for abuse.
I would argue that point. I pay for Internet access, not "All of the Internet MINUS Netflix" access. As a result my ISP SHOULD (doesn't mean they will) ensure that all it's peering connections are of sufficient capacity to provide me with whatever bandwidth I choose to use up until I run into the rate limits for the plan I'm on.
My ISP sells me 25/5 I think so the idea that I can't get 3Mbps worth of Netflix is suspect.
Of course you have to consider that when you're not saturating your last-mile connection it could be a bottleneck anywhere. If it's 15 hops from me to a server if any of those 15 hops is saturated, that's the limiting factor no matter how much capacity there is before or after. That's why closing a single lane on a freeway for as little as 10 feet can really screw up traffic. You have to merge down and deal with reduced capacity even though it's not miles and miles of lane closure.
What we've been told via one of Netflix's backbone providers (Level 3) is that Netflix isn't the slowdown, the backbone provider isn't the slowdown, but the ISPs are the slowdown. How can they say this with certainty? Because they're peered with various ISPs, the connections are saturated, and the ISPs refuse to upgrade their side of the peering connection unless they're paid.
http://blog.level3.com/global-connectivity/observations-inte...
That might sound reasonable at first brush, the ISP will incur costs associated with upgrading the peering connection. BUT so will Level 3 incur costs. So on that front they should cancel out, Level 3 pays for their side, ISP pays for their side, upgrade gets done and peering connection gets un-saturated.
Now wait, you might say, the backbone provider is pushing all this traffic on the ISP and it's not the ISP's fault that this deluge of data is headed it's way, but rather the backbone provider's fault. And there's some meat on those bones under one condition: the ISP doesn't have customers who are paying it for internet access. So that argument is not valid. The ISP's customers are the ones requesting the data that the ISP has promised to deliver and then the ISP wants to get paid twice. First by the customer for internet access, and again by the backbone provider for access to customers.
If it's so painful to provide the product to your customers that you have promised to them, perhaps you need to sell the business and get into another one. That is what literally every other company in the world does:
1. Offer to sell a thing to a customer
2. Customer agrees to buy a thing
3. Provide the thing to the customer
4. Customer pays for a thing
5. Profit!
These ISPs are so incompetent that they can't manage step 3 without getting paid extra, again, not just by a customer but also by their supplier. It boggles the mind!
But, I just completed an Office 365 implementation for a huge enterprise. Email for businesses of our size is similar to netflix in many ways.
Guess what we found out early on? 365 is different than web surfing and providing services to consumers and business partners. The model we used to deliver Internet wasn't effective for a variety of reasons, so guess what we did? We paid a bunch of money to establish a peering connection to the Microsoft network at our expense.
Netflix's solution is was to peer and to place a bunch of storage gear at ISP points of presence for free. That works, but may not be a solution that makes sense for the ISP for a variety of reasons. The issues that we're witnessing now is the lame ultimatum driven behavior that happens in contract negotiations.
IMO, we need rules that prohibit ISPs from discriminating against certain traffic. But there needs to be a line where exceptional bandwidth destinations have to take measures to be good citizens, which may require payment for connection or hosting.
Here's the thing. The ISP is over-subscribed, i.e. they've sold probably 25 more "contracts" for service than they can actually deliver. They advertise 25/5 Mbps service to customers but allocate perhaps only 1/0.2 Mbps (only 4% of what they're advertising for sale) across their back-haul and peering stations for their customers. This is under the premise that the network is bursty and people download webpages in a second or two and then read them for several minutes, etc. That assumption might have been valid 10 years ago. It's much less valid today.
So the situation isn't that Netflix is DDoSing the poor little ISP which is doing it's best but just can't keep up. It's that the ISP has KNOWINGLY oversold itself and it's now getting caught in the act and it's wildly pointing fingers at anyone and everyone it can. And it's attempting to re-frame the argument in a fashion that makes them look like the victims when in fact they're the villains.
In reality what Netflix is doing is bending over backwards to try and make their customers (who also by definition are an ISPs customers) get the level of service that the customer has already paid for and has a very legitimate claim against their ISP for in terms of transport from their house all the way to the ISPs peering stations. The ISP isn't upgrading their peering links unless they get paid to do so. They are holding their customers hostage and demanding ransom from people who their customers have quite literally already paid to be able to communicate with.
I can understand how this can be confusing because the ISPs are doing an AMAZING job of framing the argument in terms that make them look like the victims of horrible abuse. But that's simply not the case. They signed up for this "abuse" when they advertised certain speeds and started billing customers for it.
They're basically upset that customers are asking them to make good on their promises because they thought they'd never have to. I am sympathetic to their plight in the sense that I know how much it sucks to make a utilization assumption and to price it accordingly and to get it wrong. But having gotten it wrong in the past doesn't make it OK to continue to get it wrong always and forever.
EDIT: I just reconciled my 100x, 4% math which didn't make any sense.
At the same time NetFlix doesn't want to pay to expand their CDN capacity/diversity. Their motivations are exactly the same as ISPs -- cut costs & maximize profits.
Bursty traffic and steady traffic cost different, so I think, in order to be honest, ISP should sell two differently priced 25/5 Mbps connection, one priced same as now, and one much more expensive. I predict people will not move to the more expensive pricing. So while there is some deception going on, I doubt people actually want to pay for steady traffic.
Let's imagine for a moment a very simple case. A network with nodes A, B, C, D and E
A <-> B <-> C <-> D <-> E
Now let's suppose that (A,B) has a 1Gbps link. And (B,C) has a 10 Gbps link. And (C,D) has a 10Mbps link. Finally (D, E) also has a 1Gbps link. What is your ability at A to receive data from E?
It's the minimum bandwidth of all the links that limits your rate. So clearly we can see that the available bandwidth from (A,E) is 10Mbps because that's the limiting factor.
Let's suppose that my ISP consists of just the link (A,B) and they advertise speeds up to 1Gbps. And then let's assume that I can't (at node A) download from node E as fast as I'd like. The problem is that the ISP doesn't have any meaningful control over the link (C,D) because their network only covers the link (A,B). That's the reason that ISPs advertise "up to" a particular speed, because they have no control over what happens once you leave their network.
Now let's suppose that the ISP actually covers A, B and C. In other words they have both a consumer facing link, (A,B) which is the cable modem to POP link. And they have a back-haul link, which is (B,C) to their peering station, which is at node C. Node D is on the backbone providers network and is the other end of the peering arrangement from the ISP to the backbone provider. E is just about anyone else on the backbone network, perhaps Netflix.
In this case the ISP does have some sort of control over the bottleneck link. They can't unilaterally upgrade it like they could (A,B) or (B,C) but with the cooperation of D the 10Mbps link could get upgraded thus providing better (A,E) service to me. Ostensibly when I purchase 25/5 service I am paying to get 25/5 all the way up until the ISP doesn't have control over the network anymore. But here the ISP has DELIBERATELY bottlenecked me ON THEIR OWN NETWORK by refusing to upgrade their half of the (C,D) link.
I am 100% sympathetic to the idea that ISPs don't have any control once traffic leaves their network and thus the 25/5 isn't a guarantee. But the idea that they won't make a good-faith effort to provide 25/5 transit across their own network to a backbone provider but still advertise "up to" 25/5 to me as a customer is completely reprehensible.
If you don't have to make a good-faith effort to provide the 25/5 then why not advertise "up to" 1Gbps or 100Gbps? Customers are never going to see those kinds of speeds but by the logic that "up to" language absolves them of any and all responsibility to actually provide it, why not advertise it?
The reason is that they would get sued for fraud. They have no reasonable ability to claim the 1Gbps number over a link that can never ever get close to 1Gbps, either cable or DSL. What's the difference if they move the bottleneck a single link or two away but still inside the ISPs own network? As far as I can tell it just makes it harder for customers to figure out and thus less likely that they will get caught. The fact that it's confusing and many people don't know how networks work doesn't make it not fraudulent. It just means that it's highly unlikely that enough people will figure it out and start a class-action suit.
Related; I haven't seen anyone point out yet that Comcast (and most other ISPs IIRC) already do have a bandwidth cap, and it looks like they're now on an even more complex model where the pricing / overage varies by market: http://customer.comcast.com/help-and-support/internet/common...