ISP Interconnection and its Impact on Consumer Internet Performance
measurementlab.net
measurementlab.net
If I could make it longer, I might have said "Due to neglect of relatively cheap interconnects, US ISPs are and were measured to be providing sub-broadband connections to many portions of the Internet".
The title "ISP Interconnection and its Impact on Consumer Internet Performance" just says that there was a study and says nothing about its results, but it's the results that are interesting. It was kind of a surprise to see the title changed out from underneath me. I did not know HN titles were edited in that way.
ISPs are rightly a little pissed that Cogent, for example, takes Netflix's money for transit service then hot potatoes the data right onto the ISP for the ISP to transit to the end customer.
Remember, Cogent is being paid to deliver the content, but they are contracting to carry more data that the interconnect can hold. Now you could say the ISP should just let them expand the interconnect, but that is no excuse for Cogent overselling their own capacity.
Interconnects have always been a wild west.
They can do this because they have a monopoly on their customers. Not just a "Microsoft monopoly" where there are plenty of alternatives but network effects that keep people using Windows, but honest-to-god monopolies not just granted but ENFORCED by the government.
What we have here is a failure of the government to break the monopolies, revoke the monopolies, or police the monopolies. A little legislation that said "you can't oversell your bandwidth by more than X amount" would go a long way towards giving consumers an effective stick to beat their ISP with for over-promising* and under-delivering.
Note: The * is there to denote that they've written the contracts such that basically no matter what happens they're not in violation of them. We're only entering those contracts willingly in the sense that it's an abusive contracts from one of two or three vendors, or no contract at all. It's kind of like asking someone if they'd rather be beaten or stabbed. Given the choice; neither. If I have to pick one, how big is the knife?
Now I think there is a decent argument that ISPs should have to have free peering available at their local loop level. If congent wants to peer at the CO office of the local telecom, it should be able to do it.
The issue is that Verizon is not just a local ISP, it's a nationwide T1 provider. It sells transit service. It should be allowed to charge for transit to it's own local loop networks.
2) There are no enforced monopolies anymore, and haven't been for a long time. Damn near all profitable areas have a telecom and a cable provider. Congress banned monopoly franchise agreements for cable and fiber in 1992. Cable Television Consumer Protection and Competition Act of 1992, Pub. L. No. 102-385, 106 Stat. 1460 (1992).
3) I believe ISPs should have to meet some benchmarks for speed compared what they advertise, but you can't force them to maintain speed on specific interconnects. If Cogent has a shitty network that isn't on Verizon.
That's what the whole dispute is about. "Verizon is a Tier 1 provider" is some kind of meaningless talking point with no relevance to anything. Cogent and Level 3 are already transit providers with national networks, they don't need to use Verizon's transit network because they have their own. The problem is that they have traffic they want to pass to Verizon in LA for Verizon's LA customers and Verizon won't give them access without paying a toll. The fact that Verizon also has fiber going from LA to Houston and San Francisco has no effect on any part of that.
The traditional arrangement is that you have a last mile provider in each region and a collection of Tier 1 providers who they pay to provide transit between them. If a last mile provider is also a Tier 1 network then the relationship is different. Rather than a monopoly last mile network shopping for a transit provider and the transit providers all competing with and peering with each other, you have a large monopoly last mile provider who doesn't need a transit provider and is in competition with the other transit providers.
The situation changes from one where Standard Oil is buying trucks from Ford and Chevy to one where Standard Oil starts making their own trucks and can charge prohibitively high prices for fuel to Ford and Chevy customers to destroy their business and force customers to buy the monopolist's trucks.
Verizon isn't a fake T1, it's a real honest to goodness T1.
Do you have some kind of cite for this? If it didn't already matter which Verizon peering point they were using then it wouldn't matter if any particular one was congested because Cogent could just dump the traffic on a different one in a different city.
Is your point that Level 3 isn't a Tier 2 network but Verizon is? Obviously Level 3 is going to peer with Verizon somewhere in Greater Los Angeles and Verizon would carry the traffic from there to Long Beach or whatever. But then you would seem to be suggesting that the Tier 1 network should be paying the Tier 2 network.
> Verizon isn't a fake T1, it's a real honest to goodness T1.
The Tier 1 network they own is not fictional, it is only not relevant. The traffic for Verizon's customers in Los Angeles is not delivered to Verizon in New York.
Haven't they been the party to buy transit in the past?
Aren't Comcast and Verizon just taking advantage of their size and monopoly position to play hardball with the Tier 1 providers?
2) Small ISPs buy transit if they need it. Comcast and Verizon don't because their larger networks acts as a transit for them.
3) Yes they are taking advantage of their size, but they are performing the same service as a transit provider. If Verizon spun off their local ISP network, Cogent would have to pay VerizonT1 to transit to the local ISP.
Why does Verizon owning the transit and local somehow make it unreasonable to ask for payment?
Verizon and Comcast aren't asking for money to accept data, they are asking for money to transit the data for Cogent and L3.
Keep in mind the ISPs provide highly asymmetrical connections to their customers, prohibit those customers from running servers, then cry foul over an imbalance.
Their consumer networks are designed to be imbalanced; there is not even a chance of them being balanced.
> ...but you can't force them to maintain speed on specific interconnects. If Cogent has a shitty network that isn't on Verizon.
No one is proposing holding Verizon accountable for Cogent's performance across Cogent nodes. It is possible to mandate packet delay / rejection limits specifically at the interconnect.
Actually, it's worse than that. They then cry foul over the people who are doing their best to provide that asymmetric data without complaining. Why aren't Level3 and Cogent bitching about the imbalance?
From a very high level the internet can (and should) be thought of as pull-only. If you start sending me data that I have't requested, most people would consider that a DOS. If a bunch of people start sending me data I haven't requested, that's a DDoS. If both parties consent to the data being transferred, that's GENERALLY done through a pull-style arrangement.
Yes I do realize that uploading files to a website might be considered and colloquially referred to as "push" but if the machine that's receiving the upload stops sending ACKs then the machine sending the upload stops sending more data.
The ISPs are selling as asymmetric service but then getting upset if you use it asymmetrically because they assumed that the average utilization of the service might only be 0.1%. In reality it's turning out that the average utilization is more like 1% or 5% and this means they have to upgrade their infrastructure continually rather than one-and-done. I'm not sure that I feel badly for them. It's obvious that people use a lot of bandwidth and they're only going to want to use more.
If you're a small ISP or datacenter (T2/T3) Cogent/L3/etc won't peer with you even if traffic levels are balanced. They'd rather you have to purchase more transit instead because then there's a chance you'll purchase transit from them. OFC now they gripe about MSO/ILECs not peering with them. As a small T2 (peering + transit) provider, cry me a river. They are getting what they deserve.
Part of the problem is Cogent (and their ilk) primarily ride cheap intercity fiber between NFL/MBA cities and ignore the other places. Fiber to COs is the province of the ILEC, MSO, or boutique transport provider (INDATEL etc).
If it doesn't make sense for Cogent/L3 to pay the ILEC/MSO for ports to offload traffic at a datacenter it certainly doesn't make sense for Cogent to buy fiber to the CO from ILEC.
Yes, it's true that this is how peering agreements have worked in the past, but if that's not sustainable going forward maybe another agreement needs to be reached.
2) What you say may be true, but it doesn't change the fact that nearly every market in the US has but one single ISP provider. That sure feels a lot like a monopoly to me.
3) I largely agree with you here, except that I would clarify that you can force them to maintain speed between the interconnect. Both the interconnect and the ISP have to have capacity to spare, you can't under-provision your link to the interconnect and then point the finger back at them.
They could shift to a sort of bittorrent distributed system if they wanted to, but that is a huge shift in models and not really worth talking about.
2) IIRC 79% have two providers. Their local telecom and cable company.
3) You can but it's a crappy policy that would destroy the ISPs negotiating position. Potentially even letting the transit providers charge the ISP, like they already charge small ISPs.
A) Remember, Cogent and L3 are still selling Netflix and all their other customers, space on interconnect links that it doesn't have.
In what other business can you sell your client service and then fail to deliver and just blame your supplier for not giving you a cheap deal.
B)If you are going to force free peering on ISPs they should at least be able to charge the peer the actual cost to set up the peer point.
Isn't this exactly what the problem ISPs are demanding, which is what we're up in arms about? They're trying to double dip by getting their customers to pay and trying to extort Netflix and Level3 as well.
At least we can both agree that this is wrong.
What Comcast is (and other ISPs are) doing is intentionally not spending the $10k-$50k that it would take to upgrade the peering with Cogent or Level3 so that they can deliver to me the bandwidth that they ostensibly have promised but which they are contractually not obliged to provide. And people think that's shitty, because it sure seems like a monopolist abusing their position of power. Perhaps because it is.
Cogent and L3 are ISPs as well, so I think saying "ISPs are failing to provide broadband speeds" is fair. However, to add context, L3 had a blog post a while back explaining that they are standing ready to add more capacity for cheap and that the congestion is, in at least one instance (in LA), Verizon's fault. http://blog.level3.com/open-internet/verizons-accidental-mea...
Generally speaking though, the original title is the one to us, and if you use something else, it will generally be changed to match the original one.
I've given up on trying to come up with what I think is the most useful title to HN though, or on trying to guess what the mods will do. I just use the original title and let the mods change it if they want. If I think the original title isn't good at explaining the main point of interest from a HN view, I feel sad, but oh well, that's clearly how HN wants it.
It might be convenient if HN had a tool like reddit to automatically fetch the title from HTML given a URL. In fact, if they had such a tool, I'd expect HN to maybe actually mandatorily make that the title and not allow you to edit it, since it seems to be HN editorial preferences not to let submitters write titles.
The HN guidelines (edit: which are at https://news.ycombinator.com/newsguidelines.html and linked at the bottom of every page) say not to use the original title when it is misleading or linkbait. How could that be clearer?
Do we judge every case correctly? Good lord no. That's why we change things when people point out our mistakes. The community contribution here is considerable, so the best way to complain about a title is to suggest a better one.
The guideline also explains why titles can't be automated in the way you suggest: the software would have to identify what is misleading or linkbait. If anyone can show us such software, believe me, no one would use it more eagerly.
My titles have been changed so many times (in both directions), that I just decided to always use the original title, and let you do your mod job and change it according to your judgement.
Do the HN guidelines also say to always use the original title unless it's misleading or linkbait too? I hadn't actually been aware of that -- where do I find these guidelines? I'm not actually sure. If I had known that, it would have spared my energy in coming with titles I thought were more informative only to have them, as a rule, changed back.
Thanks for your work!
I just found the Hacker News Guidelines you mention by googling -- previously I hadn't been aware they even existed, so didn't know to look for them. I see they do say to ordinarily prefer the original title. I am not sure how people find out about these guidelines, are they linked to from somewhere on HN that I'm missing? It might make a lot of sense to include a link to them on the 'submit' form.
That's a good idea. It's easy for us to forget that not everyone knows about this, and I'm sorry for assuming that you did.
> let you do your mod job and change it according to your judgement
I hope you didn't miss my point above about how we get it wrong too. The most I'll claim is that we try to correct the mistakes people tell us about.
I'm glad HN is like this. It's one of the reasons I enjoy the website.
It might be convenient if HN had a tool like reddit to automatically fetch the title from HTML given a URL. In fact, if they had such a tool, I'd expect HN to maybe actually mandatorily make that the title and not allow you to edit it, since it seems to be HN editorial preferences not to let submitters write titles.
This was tried, I think. It was proven to be a bad idea in practice. If you think the complaints about titles were bad now, you should've seen them when the software did automated reversions. At least, I think they were automated reversions.
Sometimes the subtitle is a better title. Sometimes the original title is clickbaity, as you said. Regardless, it takes a person to moderate the titles.