Hacker News Reader

Dropping packets makes load problems worse, because failing users start to send even more traffic.

The opposite, dropped packets are bad because it slows the Internet down. When you need to retransmit a packet because one is dropped, the network assumes congestion and radically throttles speed to prevent the network blowing up.

This is not true. TCP will retransmit, yes, but TCP congestion control will also reduce the transmit rate in response to lost packets. Overall, proper TCP implementations will back off to approximately their fair share of a contended bottleneck. The application layer sees reduced throughput, not message loss. New TCP connections start with very conservative transmit rates, so an application which is overly aggressive in abandoning slow connections and retrying will not add substantially to congestion.

UDP doesn't necessarily have these properties, but anyone who implements a high-bandwidth UDP protocol which isn't TCP congestion control friendly deserves public shaming.

Except between buffer bloat[1] and users spamming the reload button, TCP's congestion control is not going to solve the problem.

1. http://en.wikipedia.org/wiki/Bufferbloat

That is true. TCP congestion control throttles back the speed in the presence of dropped packets.

Thus, dropped packets adversely affect throughput at a magnitude greater than the simple overhead of packet retransmission.

AFAICT, the reason why packets are getting dropped is because the ISP's equipment doesn't have the capacity to handle the traffic.

i.e. internet is a series of pipes example. Level3 has a big fat pipe with 10Gbit/s of traffic going through to the peering point, where $ISP has a pipe that goes to their network rated at 1Gbit/s. Since the traffic can't fit into $ISP's pipe, the extra packets get dropped.

Level3 is dropping packets because $BIG_CABLE_ISP doesn't want to upgrade their cross-connects at the exchange points. Level3 sounds happy to increase this capacity, they are saying that the cable ISPs aren't negotiating with them to provide more bandwidth at these choke points. It's not Level3's job to pay for $BIG_CABLE_ISP's infrastructure, when it's their own customers who are requesting the traffic over these links.

But isn't that the point? Level3 is capable of handling more traffic but the ISPs are not. So what?

It sounds like Level3 wants to enter into bigger-money contracts with the ISPs but the ISPs don't want to, so Level3 is appealing to the masses to help them rake in extra cash (to justify the expenses they've incurred upgrading their equipment).

Why does Level3 get the benefit of being David here? IT just seems like they're trying to leverage the net neutrality debate to get themselves more favorable peering contracts with ISPs, which in a twist of irony, is exactly the kind of thing which goes against net neutrality in the first place (why are Level3's packets so special anyway?).

What? Level3 isn't appealing to the masses to help them rake in extra cash. They don't need to.

Do you think Level3 just throws data at Comcast nonstop, and Comcast is helpless to stop it? Customers of Comcast ask for data that Level3 is trying to provide; Comcast can't handle the bandwidth. But the only reason the data is flowing is because the customers of Comcast want it; it's not like Level3 is forcing Comcast to accept it.

Why is Level3 obligated to pay for upgrades to Comcast's infrastructure to satisfy demands placed on Comcast by Comcast's customers?

You've got to ask yourself why Level3 is writing blog posts about this in the first place. It'd be like a bread making company calling out the sandwich shop it sells bread to for not having enough stores.

BUT WE'VE MADE ALL THIS BREAD WHY WON'T YOU BUY MORE?! JUST OPEN UP 100 NEW STORES I KNOW YOU HAVE THE MONEY TO DO IT!

Level3 invested money in their infrastructure, but Comcast, et. al. have not (yet). Level3 is losing money by having all this additional capacity, are they not? So clearly it's in their best interest to light a fire under the ISP's collective ass to try to get them to upgrade quickly.

Oh hey look, a political issue has appeared that does precisely that. Why wouldn't Level3 latch onto that movement and attempt to use it for financial gain in the form of more favorable peering agreements?

>It'd be like a bread making company calling out the sandwich shop it sells bread to for not having enough stores.

No. It isn't. It's like a bread making company calling out the grocery stores for not having enough shelf space to stock all the bread that customers of the grocery store are asking to buy, saying "People keep on asking us for bread and you can't stock enough of it".

edit to add: You keep on not understanding that the only reason the demand exists is because the customers of the ISPs want it. This isn't a scenario where L3 is just trying to push data at Comcast; it just isn't. You keep on acting like it is, and it isn't, and I can't think of another way to phrase it so that you'll understand. Level3 is not just pushing data at comcast that nobody wants, and whining that comcast can't handle it. Customers of Comcast are asking for the data, Level3 is trying to provide it, and Comcast is blocking that transaction.

edit again: It's also worth noting that in this scenario, people are paying a monthly subscription fee to the grocery store that says they can come in and get certain amounts of bread (or UP TO certain amounts of bread whatever) and yet because there is not enough shelf space customers are going hungry. And yet despite the fact that people are going hungry, the grocery stores are insisting that the bakery provide, in addition to the bread, the shelves on which to store the bread. And you're arguing that since the bakery has already spent money upgrading their ovens, their proofing rooms, and their mixers, they should also upgrade the grocery store's shelves.

The demand is not for L3, the demand is for Netflix. Level3 is pushing itself on "Comcast" (which is really a stand-in for the 5 ISPs who aren't playing ball with L3, and may not actually be Comcast at all).

Why must these ISPs play ball with L3, and not try to find competitors to do what L3 does? Isn't competition good?

Edit: People aren't going hungry (I know it's an analogy, but come on), people are watching Netflix at 480p instead of 720p/1080p. Let's try to keep things in perspective here.

L3 is not pushing. I cannot stress that enough. L3 is not pushing. L3 does not push. Pushing is not something L3 does. In the Game of Pushing, L3 is not a player. If you navigate to wikipedia.org/things_which_push, L3 is not on that list. If all the entities of the universe were put into two groups, one labeled "pushers" and one labeled "not pushers", L3 would be in the second group.

The ISPs must play ball with L3 because their customers are asking for content that L3 is trying to provide. If the customers ask for different content from somebody else, then somebody else would be providing the data.

And excuse me? Don't blame me for the crappy analogy; you're the one who brought it up. I thought it was a stupid analogy too, but I thought that speaking in your own terms might help you understand the situation.

Customers are not asking for L3. Customers are asking for content, of which L3 is one of the providers. L3 is pushing L3, even if you say they're not.

Without customers asking for content, L3 wouldn't provide anything to Comcast. It's at the specific request of the customers that L3 relays the replies. The customers initiate the request, and L3 provides the response.

Without the service L3 provides, Comcast wouldn't be able to (as efficiently) provide the service to their customers. In a sense, Comcast should pay L3 for their access to the content.

But they're not asking for L3, they're asking for Netflix. So maybe Netflix should hire a CDN that can play ball with Comcast, instead of L3, who is apparently being shut out.

> So maybe Netflix should hire a CDN that can play ball with Comcast, instead of L3

Um, didn't someone upthread point out that Level3 doesn't actually carry any Netflix content?

So what you are saying is that Netflix should pay extortion money to Comcast - either directly, or via a CDN - because Comcast have a monopoly situation.

You can try to package it any other way you like, but that's what you are suggesting: The only reason Comcast is in position to try this gambit is because there are alternatives (albeit very few) to backbone providers, and major content providers can be bullied directly, but most Comcast customers do not have alternatives to Comcast.

If anything, that Comcast even dares ask for this should be evidence that they are abusing a monopoly situation, and ought to be a trigger for massive reform - in a free market, they'd get told to take a hike.

L3 is not the provider, Netflix is the provider. Netflix has chosen to contract with L3 so that their customers can receive the content. L3 is also an ISP and so has peering agreements with other ISPs. Netflix has customers who aren't using L3 as an ISP so L3 and these ISPs make use of their peering setup to transmit Netflix's content to the ISPs' customers.

L3 is not pushing, the customers are pulling. The ISPs serving the customers are not able to keep up.

You want pushing? That'd be L3 using (as an example) Comcast to reach customers of TWC. Then L3 would be pushing through Comcast. That's not the (apparent) situation here. The end user is on the other side of this peering divide, not multiple peers away. Customers are pulling data from Netflix which happens to be on L3. Blame the customers and up their rates if that's what's needed to cover the costs. But at $15+ billion a year in profit Comcast can probably afford to use the customers' money to provide them with the promised experience.

L3 is pushing to maintain its contract with Netflix. L3 is pushing, because L3 isn't the only CDN responsible for Netflix traffic. "Comcast" (it may not actually be Comcast and in fact I doubt it is given Netflix's recent agreement with Comcast) can just use Netflix's other CDNs, and traffic will continue to flow normally.

L3 is not necessary here, and everyone who's actually involved knows it, including L3.

If Comcast can't handle the amount of data, how would using a different CDN help? Does the other CDN somehow send less data given exactly the same requests?

Or are you arguing that an ISP should be allowed to restrict which services people are allowed to access depending on how much those services pay the ISP?

> It sounds like Level3 wants to enter into bigger-money contracts with the ISPs

As the article points out, peering rarely involves money (and shouldn't ever; as the word peer implies, its a largely symmetric relationship).

The ISPs' customers are paying the ISPs for bandwidth and then requesting traffic. Level3 is providing said traffic and then the ISP drops it because they don't have enough bandwidth. How can a reasonable person possibly interpret this as being Level3's fault?

If an ISP isn't willing to supply its customers with the bandwidth that is being paid for, they need to either fix that or stop promising things they aren't willing to deliver on.

> stop promising things they aren't willing to deliver on.

Very few ISPs offer residential Internet access with SLAs that guarantee performance or even availability. They mostly promise only a 'best effort' class of service.

That's a good point. I meant promise in the sense that that's how customers often interpret a bandwidth claim - legally, they're making no such promise (otherwise they'd be in a lot of trouble).

"Best effort" doesn't mean "Here's your dinner, take it or leave it."

ISPs have been successfully managing aggregate network bandwidth growth for over twenty years.

I work at a place that has transformed it's utilization of the internet via cloud services -- utilization going from 80-120MB/s to multi-gigabit. Adapting to that change was relatively easy and cheap. With the economies of scale that a large Telco has, it's a drop in the bucket.

Right, and the problem is that their version of best effort doesn't involve trying very hard at all, since they have minimal competition in most local markets.

I've been speed testing my shitty (Time Warner, yay) internet lately and am getting fully 50% of my 15m/s that I pay a whopping $80+/mo (hard to disambiguate between the 'packaged' cost with TV). I once got a Comcast rep to admit that they basically were unconcerned with providing anything which is at least %60 of the level that they advertise. Unless the entire company is both mentally and physically challenged I seriously doubt that this represents a "best effort" class of service. This is go-fuck-yourself service that a dive bar would be embarrassed by.

If Level3 can't give Netflix what it promised Netflix it would deliver, maybe Netflix should use a different CDN.

Other CDNs aren't having this problem, because they're willing to work with the ISPs.

> Other CDNs aren't having this problem, because they're willing to work with the ISPs.

Do we know this?

You asked this question in like 3 places, but yes, if you read the blog post written by Level3, you'll see them mention that other backbone providers are making deals with the ISPs, but that they're refusing to.

> if you read the blog post written by Level3, you'll see them mention that other backbone providers are making deals with the ISPs, but that they're refusing to

Where in Level3's blog post does it say this? I'm not seeing it. Certainly at least one other transit provider, Cogent, doesn't seem to be making ISP deals; if it did, the Netflix-Comcast deal wouldn't have happened

(Also, in the post I was responding to in this subthread, you said CDN's, not transit providers. They're not the same.)

Yes, let's encourage people to pay ransoms. That's going to end in a rational system.

Netflix switching would, in the short term, give them more bandwidth. In the long term, encouraging ransoms will cause them to go up and Netflix will be forced to keep raising their prices. This is exactly what Netflix does not want to happen, and they are wisely refusing to play ball.

I'm not very familiar with networks at this level, but it seems logical: Level3 wants to push more traffic through the ISP's pipe than the ISP's pipe can handle (because the ISP's customers want those packets).

The only solution for Level3 is dropping or queueing packets, and I'm assuming queueing is not feasible.

The ISP is pulling the packets from Level 3, Level 3 is not pushing it. Also Level 3 claims they can get the packets all the way to the cross connect between their network and the ISP's and they have extra capacity at the cross connect, but the ISP's refuse to upgrade their side of the cross connect (IE the part that turns them from a network provider to an Internet provider.)

>Also, dropped packets isn't the end of the world. In TCP at least, the packets just get resent.

Depends on what the application is, and how many are being dropped.

True, standard HTTP can in theory even still work with 99% drop rate (if you're willing to wait long enough) but Netflix, multiplayer games, and any other application you can think of that demands either high-bandwidth or low-latency aren't going to work. As far as those applications are concerned, it is the end of the world.

Streaming services utilize a buffer, so no, it isn't the end of the world if packets are dropped. The buffer gets used while the packets get retransmitted. As for multiplayer games, most use UDP, and the ones that use TCP tend to use some tricks that allow for a certain amount of latency.

This is why the Internet isn't currently down, even though this is currently a problem.

No widely used Internet-based technology can't handle dropped packets. Zero. None.

Besides, Netflix is a unique case, as it constitutes 30% or so of all Internet traffic. You can't list Netflix as one of many, because it stands alone.

Twitch.tv, justin.tv, youtube. That's off the top of my head and I'm confident there are more than I'm missing. There are plenty of video streaming services; not all of them are as large as netflix, but to say "Netflix is a unique case" is pretty weird.

And dropping packets isn't the end of the world, but it's certainly not good. Should we just ignore a problem just because the world isn't ending?

Netflix is a unique case. No one even comes close to network usage. Not even close. It's a very unique case, given its volume.

And I don't know who here said to ignore this problem, but they're wrong too. It's just that this isn't the "It’s insulting." problem the submission tries to paint it as. His solution is to "throw the bums out". Are you kidding me? Why are we wasting our time reading and talking about this tripe?

Can you clarify for me why I shouldn't feel insulted that I'm paying for a certain level of service from my ISP, and yet not receiving that service while they insist that it's not their fault?

Because you actually are receiving the level of service you're paying for, you just didn't know what you were paying for when you signed up.

In what sense? Are you referring to the "well actually the contract just says you pay for UP TO certain speeds" bullshit?

I'm referring more to the fact that ISPs probably reserve the right to negotiate with peers on their terms in their contracts, not on your terms.

You having a month of 720p instead of 1080p Netflix is worth it to the ISPs instead of raising your service prices by $5 bucks every single month.

Oh; instead of referring to some bullshit clause in my contract, you're referring to something that's actually irrelevant. That's weird, but okay. I can roll with it.

ISPs probably do reserve the right to negotiate with peers on their terms, but I don't care about that because I'm not paying them to have a certain agreement with peers, I'm paying them for a certain quality of service. If they are not providing that service (an example being giving me 720p Netflix instead of 1080p like I have paid for) then that seems like a pretty straightforward contractual violation. I'm paying for service and not receiving it. Netflix is providing the service to the ISP. I am providing my money to the ISP. Why is the ISP excused from holding up their end of the bargain?

This is turning into a flame war. If you can't see how negotiating peering agreements is relevant to a post whose first sentence begins, "There’s a peering crisis apparently happening right now among American Internet Service Providers (ISPs) and backbone providers..." then I don't know how we can continue...

The crisis, if you've read the article, is that Level3 is getting shunned by 5 of its client ISPs. The ISPs are NOT shunning other backbone providers, just Level3. So yes, this is entirely a contract negotiation between Level3 and 5 ISPs.

This isn't an existential crisis of the Internet, like Level3 is trying to portray it, it's Level3 being shut out of additional business by 5 ISPs because they won't play ball like the other backbone providers.

Why should L3 play ball? Just because everybody else is doing it doesn't make it a good reason.

They spent money on upgrading their network, and now that money is going to waste if the ISPs don't make use of the additional bandwidth (for additional money).

ISP customers are not suffering, because L3 is just one of many providers. What content does L3 provide that no one else can? And if that's the case, how is L3 not a monopoly in that respect? Aren't we all calling for more competition in this space anyway?

I guess I'm confused; in one post you're pointing out that I'm getting a month of 720p video instead of 1080p, but now you're saying that I'm not suffering. If my quality of service is being degraded, is that not suffering in the context of this issue? I'm paying for service and yet not getting it. So clearly there IS an issue, because connections are suffering. Are you telling me that ports between L3 and various ISPs are totally saturated and dropping packets while there are other nicely upgraded connections with plenty of bandwidth to spare? Because I'd like to see a source for that claim if so.

L3 doesn't provide unique content; they provide content that ISPs don't have. People ask for that content, L3 tries to provide it, and the ISPs can't accept it. Are you telling me that there other providers and L3 is just shouldering them out of the way? Again, I'd like to see a source for that.

First of all, L3 doesn't actually provide Netflix content, I don't think, so your video stream is unaffected entirely by L3's agreements with any ISP.

Second of all, you have no SLA with your ISP, so your quality of service is, "best effort". If you want an SLA, you have to pay, otherwise you're going to get the best that the ISP can give you, which means some degraded performance. That's what you've paid for. If you want better, pay more money.

Third of all, you really should read the submitted article. It's very clear you haven't. When you do, you'll find that L3 is the only backbone provider not playing ball, and the other backbone providers are paying, which means that there actually are nicely upgraded connections (I don't know about "plenty of bandwidth to spare"), paid for by the network backbone providers.

When you read the article, you'll be much better informed on this topic. I do recommend also reading L3's blog post. There's some good info there, and it provides all the "sourcing" you've asked for.

I've read both the blog post and the linked article. Can you provide specific quotes for the claims? Because the only thing that came close which I saw in either the post or the link was a graph of a different connection which isn't fully saturated, but has very little head room, certainly not enough to handle the spillage that L3 is dealing with. There's a mention of capacity upgrades in the works for that connection, but no specific numbers; it's not clear that the upgraded connection will be large enough to take on the lost packets from L3. And I didn't see anything in there about L3 pushing other providers out of the way.

So like I said; can you provide sources for those claims? Because you also thought the bakery analogy was a good idea, so I'd like to read your primary sources instead of trying to explain myself to you further.

This isn't the forum. Email me (provided in my profile) if you'd like to continue this discussion.

Your profile, at time of viewing. Congratulations; I'm done here.

user: diminoten created: 1831 days ago karma: 1799 avg: 1.51 about: QA/Wannabe Dev submissions comments

Oh sorry, I thought the email field was available!

diminoten@gmail.com

> When you do, you'll find that L3 is the only backbone provider not playing ball

Where does the article say that? It says Level3 is having peering problems, but I don't see anywhere that it says Level3 is the only transit provider having peering problems.

> The ISPs are NOT shunning other backbone providers, just Level3.

Is this true? Or is Level3 just the only one we know about? As I remember it, the transit provider that was linking Netflix to Comcast before the Comcast-Netflix deal was Cogent, not Level3.

> If they are not providing that service (an example being giving me 720p Netflix instead of 1080p like I have paid for) then that seems like a pretty straightforward contractual violation

But your contract with your ISP most probably does not guarantee you 1080p Netflix instead of 720p. It basically says something equivalent to "we'll give you enough bandwidth for 1080p Netflix whenever we can, but we don't guarantee it, and if you don't get it, there's nothing you can do about it".

Please note that I am not saying the ISPs are the good guys here. I'm just saying that trying to nail them on contractual violations doesn't seem like a fruitful approach to me.

It's a shared network. It's not bullshit that you don't have SLA guarantees. Business customers get those, and they pay a lot more for them.

There's an old saying in the car world – people buy horsepower and drive torque. Car companies always talk up the HP (even when they're kind of sad for the weight of the car, but I digress)…

ISPs sell peak speeds and customers live the reality of the fine print that talks all about it being a shared line and that real speeds will often be lower than the advertised peak…

Yes, it's their fault, but no, it's not – expecting average consumers to fully investigate every spec and to understand things like SLA guarantees for every product and service they buy in a world that is growing ever more complex has to end somewhere right? At some point, that total burden becomes unreasonable from a user-centered perspective, right?

Even the power company -- which, for all its complexity, is a vastly simpler system to model and predict -- turns off my AC from time to time to keep up with load.

Availability of a shared network cannot be guaranteed. It's the irresistible force. Even if consumers don't understand, it still cannot be guaranteed.

Or, they could each have explicitly allocated pipes. They will pay a lot more for the privilege, though, just like they would pay a lot more for a private road instead of one they have to share with their neighbors.

I have problems using myfreecams.com for live adult video chat because of the high rate of packets being dropped.

I don't really think your comment makes sense, if you and I are juggling balls and suddenly you throw up your hands and shout "I can't take any more balls" do you think I should pick up more balls and throw them at your face or should I leave them on the ground (which in this analogy would be level 3 dropping the packets)

What do you mean when you say, "Throwing their hands up"? Are you saying the ISPs are refusing to route any traffic after a certain threshold? Does Level3 get cut off entirely if it exceeds its bandwidth limit?

They effectively are refusing to route after a certain threshold. Their connections are saturated and they aren't wanting to play ball the way other peers have (at least per Level 3's side of the story). So once their X Gigabit link is saturated, quality goes down. They don't want to use the same arrangement that Level 3 has made with others (with respect to cost distribution) to upgrade the connection so the situation remains.

You should try browsing the Web with a 10% packet loss sometime.

> Shouldn't the ISPs replace Level3 if they keep dropping packets?

How would that help when the reason Level3 is dropping packets is because the ISPs don't have enough capacity to receive them?

Level3 can't magically make a 10Gbps port take 20Gbps of traffic, nor could any hypothetical replacement the ISPs might go to (there aren't many realistic alternatives).