Hacker News Reader

Why does it matter who sends who how many bits? It seems like that would only matter if one peer is sending traffic over another peer's network that's has yet a third peer as the real destination. But if e.g. Level3 is sending bits to Comcast because they're going to Comcast customers, then that seems entirely ok. These aren't bits that Level3 is forcing on Comcast. Comcast asked for these bits. Or rather, Comcast's customers did, but that's the same thing as far as I'm concerned.

If Comcast is asking Level3 for massive amounts of traffic, it makes no sense at all for Comcast to then turn around and accuse Level3 of sending them too much traffic. I suspect the real problem is that Comcast simply didn't expects its customers to actually use all the bandwidth Comcast promised them. They can't very well accuse their own customers of using too much traffic (for legitimate purposes, I'm ignoring BitTorrent here).

Comcast just needs to admit their equipment simply isn't up to the task of servicing their own customers, and they need to upgrade. Trying to charge Netflix for all the traffic is just pure greed.

>Why does it matter who sends who how many bits?

Because that's what peering is.

No one gets to connect to a network, any network, for free. But if you have your own network, the company that owns another network might decide to hook up to yours for free, and you to his for free.

Supposing, of course, that your networks are roughly equal. Why bill each other, if your bill to him will be about as much as his bill to you? Doing all that accounting costs money, so it's easier to just connect to each other for free.

But this only works if both your networks are roughly equal.

If some jackass walks up to you and says "I'm selling crap to your customers, let me hook up to your network for free!" you aren't going to agree to that. It'd be dumb.

I really, really have trouble making sense of why so many people are ignorant of peering. It's going to be a really big problem too, because ignorant people are screaming for politicians to fix it, and neither the people nor the politicians have any clue at all.

> If Comcast is asking Level3 for massive amounts of traffic

Comcast is a network. It doesn't ask for traffic.

It lets people who are connected to their network communicate with each other. If only one party connects to their network, it is impossible in a practical sense to facilitate communication between the two.

Comcast doesn't guarantee connectivity to any specific host. It is a best effort service.

> Comcast just needs to admit their equipment simply isn't up to the task of servicing their own customers,

A kernel of truth in a turd of mistruths.

Yes, I suspect this too. But it's not their equipment... that stuff gets switched out every few years. It's their network itself. HFC isn't sufficient for a modern network.

Also, there's no point in them admitting this. That helps them not at all. And, since they're familiar with all the numbers, I also suspect that they know they can't afford to build a modern network.

> If some jackass walks up to you and says "I'm selling crap to your customers, let me hook up to your network for free!" you aren't going to agree to that. It'd be dumb.

But as the ISP, aren't you billing your customers for that usage? Does it matter where the bits are coming from?

I'm not sure you've addressed eridius'a comments. I understand what you're saying about peering, but I think his/her point is that ISP customers pay ISPs not to care about peering.

I think that's generally the point that neutrality activists make, that in large par the internet works because users have equal access to any network, regardless of the asymmetry inherent to that access, and that users are willing to pay for the cost of that asymmetry.

If Comcast needs more money to properly connect it's users to Netflix, then that's fine, charge me more money. I'll gladly pay directly for real infrastructure upgrades. But forcing that revenue to go through Netflix first, you're fundamentally changing a key feature of the Internet.

The fact that these issues go away if there is no asymmetry (peering) doesn't seem so relevant.

> I understand what you're saying about peering, but I think his/her point is that ISP customers pay ISPs not to care about peering.

Nothing in their contract with Comcast suggests that they've somehow bought "the right not to care about or understand peering".

So if they think they've bought that, then the problem really is one of "stupid customers".

Do you know that I absolutely loathe Comcast? I hate coming off like I'm defending them. I want the company abolished, its assets sold off at auction, its shareholders prohibited from ever investing in shares again, its executives sent to prison to rot for the rest of their lives.

There are many bad things about Comcast. We don't need to make up lies about them.

> If Comcast needs more money to properly connect it's users to Netflix, then that's fine, charge me more money.

Then they're hit with accusations of violating network neutrality (if they only charge Netflix users), or of price-gouging if they charge everyone.

This was already taken off the table by the same people who are rabble-rousing everyone.

> But forcing that revenue to go through Netflix first, you're fundamentally changing a key feature of the Internet.

By not forcing it, you're changing a key feature of the internet: settlement-free peering.

This is why my original, top comment explains how things have changed and peering may not work anymore. It's impossible to come to a resolution where all parties feel as if they've been treated fairly.

And that's a much bigger problem than anything anyone else is talking about. It's a problem that won't go away if we ignore it.

Then they're hit with accusations of violating network neutrality (if they only charge Netflix users)

I'm not sure why you think this is a 'bad thing'.

Comcast sells users bandwidth (X download, Y upload). It doesn't sell "Access to static and dynamic web sites, but not video streaming services." If the customer is using less than X and less than Y, why does the type of traffic matter? If Comcast doesn't want users using the amount bandwidth they paid for, then they need to be more upfront about that in the plan pricing (lower the bandwidth, or increase the price).

That's what people are asking for. Just like I shouldn't expect my cell phone bill to increase because I called across state lines instead of local, I shouldn't expect my internet bill to increase just because I used the bandwidth for watching Netflix as opposed to something like seeding Linux distributions.

If Comcast doesn't peer with Level 3, then Comcast customers can't reach Level 3 customers[1]. The internet is only the internet if everyone can actually talk to each other. Comcast can't sell internet service to its customers if it doesn't actually provide access to the entire internet. Which is to say, Comcast's customers are already paying for the privilege of peering with other networks.

So no, this is not like some jackass (Netflix) walking up to Comcast's customers and demanding the right to sell crap. Netflix is already paying Level 3 for the right to internet access, and Comcast needs to provide access to Level 3's customers in order to actually provide what it promised to its own customers.

A more apt analogy would be considering ISPs as cities, connected by roads. The roads are the peering agreements. Comcastica is a city where everyone lives. Level3ton is a city with a lot of warehouses and other commercial infrastructure. The residents of Comcastica love ordering products from Netflix, who has warehouses in Level3ton. As such, the roads between Comcast and Level3ton are constantly crowded with big trucks hauling all the ordered goods from Level3ton to Comcastica.

Comcastica is now saying to Level3ton that they're sending far too many trucks for Comcastica's roads, and wants Level3ton to pay for Comcastica to expand their roads.

Level3ton is, of course, only sending the trucks because the residents of Comcastica are demanding them. If Comcastica's roads aren't large enough, it's because they underbuilt their road infrastructure given the number of residents they have.

The only reasonable course of action here is for Comcastica to take all the taxes they collect from their residents and use some of it to build better roads, in order to service all the vehicle traffic that Comcastica's residents are creating. It doesn't matter that the vehicles are coming from Level3ton, it only matters that the vehicles are servicing Comcastica's residents needs.

[1] Presumably Comcast could route traffic through a common peer to both, but they'd have to pay for that privilege, so I'm going to consider that irrelevant at the moment.

> Netflix is already paying Level 3 for the right to internet access,

No, they're paying for the right to Level 3 access.

> Comcastica is now saying to Level3ton that they're sending far too many trucks for Comcastica's roads,

False.

Comcast has never done this. They say "send all the trucks you can fit on that road".

But all roads have limits.

Netflix says "you need to build another road, I have alot of trucks".

This is literally what happened. The link was 90% saturated. Level 3 themselves admitted this in that blog post the week before last.

I don't mind the metaphors and analogies, if it's to help you understand something that is foreign to you... but that's not a license to twist the issue.

Netflix is saying "your roads do not meet the needs of your residents. They're asking for a lot of trucks but your roads aren't large enough". Although this metaphor is getting rather strained because cities don't charge their residents for varying levels of road access.

So without the metaphor, Netflix is telling Comcast that their links do not have the capacity to serve the needs of their (Comcast's) customers. This isn't something that Netflix should even need to tell Comcast; Comcast already knows it. In fact, not being able to meet their obligations to their customers was an intentional decision on Comcast's part. They assumed that their customers wouldn't actually use all the bandwidth they are paying for. But with the rise of video streaming in general, and Netflix in particular, Comcast's customers are using more and more of their bought and paid for bandwidth.

That's the issue right there. Comcast intentionally oversold their capacity (or intentionally under-built their infrastructure, whichever way you want to look at it), and it's now coming to bite Comcast in the ass. But instead of admitting that they did this and working to build out their infrastructure, which would cut into their obscene profits, they're trying to blame Netflix instead and demand that Netflix pay. In short, they're trying to double-dip.

It's a simple fact that if Comcast's customers can't reach Netflix (but customers of other ISPs can), then Comcast has sold a false bill of goods to their customers. They haven't sold them "access to all websites that are willing to pay for the privilege of talking to Comcast". They've sold them access to the internet. An internet that includes Netflix.

> Why is Comcast obligated to do this for free?

Because of a hundreds of thousands of people paying them already over-priced networking fees? Comcast wouldn't be doing it for free... I'm already capped at 300Gb per month on my Comcast internet... If I choose to spend that 300Gb in a week on streaming netflix, fucking let me; That's why I pay for it. At this point not only are ISPs trying to limit how much of their service I can use, but when, where, and how.

This isn't how free-flowing information works.

> Because of a hundreds of thousands of people paying them already over-priced networking fees?

1. Those people are free to stop paying. 2. Why are they obligated to give something to party A for free, because they overcharged party B?

You're not really offering any sort of logical point here. You're not thinking, you're just reacting.

> If I choose to spend that 300Gb in a week

If you chose to use that 300gb downloading from my website, would Comcast be obligated to let me put a server in their data center for free?

No.

You're just being absurd.

> That's why I pay for it.

Nothing in your contract with Comcast says anything remotely resembling the claims that you're making. If you are mistaken and think you're paying for this, then perhaps you should end your relationship with Comcast. Or, if that is too uncomfortable to consider, then perhaps you should realign your expectations with those laid out in the contract.

No judge would ever listen to some juvenile whining for very long, and god help us all if you get the politicians to do so. We'll be in real trouble then.

>If you chose to use that 300gb downloading from my website, would Comcast be obligated to let me put a server in their data center for free?

No, but they're obligated to get those GB to the consumer in some frickin way, because they have a signed contract with the consumer obligating them to get them their damn bytes. Who cares if it's netflix or 1,000 myriad other sites. The only reason netflix is an issue is because they're a big enough target to extort.

> No, but they're obligated to get those GB to the consumer in some frickin way, b

No, they're not obligated.

Not legally.

Not contractually.

Not morally.

Consumer internet is a "best effort service". If Netflix refuses to acquire sufficient connectivity, then it's not Comcast's fault.

This is really simple. You should get it after I repeat it a few dozen times.

> because they have a signed contract with the consumer obligating them to get them their damn bytes.

It's an implied contract, but the problem's still the same: you never read it.

> Who cares if it's netflix or 1,000 myriad other sites. The only reason netflix is an issue is because they're a big enough target to extort.

The only reason it's an issue because only Netflix is so big that the existing connection is insufficient. When you subtract the Netflix traffic, the link's not saturated.

So your argument is that internet providers are perfectly within their rights to cash your check and then not provide internet service, and that's all perfectly ok and the way things should be? I'm not talking about the lack of an SLA when temporary outages happen, or temporary service degradation as they roll out new infrastructure to cope with traffic growth, I'm talking about consistently and systemically not living up to their end of the bargain and no intention to fix it. Complete lack of "best effort".

Subtract netflix traffic and more traffic will come from someplace else. The internet's not going away, usage is growing across the board, and netflix is just (one of) the first to hit our current traffic peaks, which will themselves look small in a few years.

If ISPs can't deliver that bandwidth at current prices, they should A) look at whatever korea, japan and northern europe are doing right, and B) failing that, increase prices until they can deliver the service they're charging for. This is really simple. You should get it.. well, you won't, probably. But the rest of us won't stand for it.

> If you chose to use that 300gb downloading from my website, would Comcast be obligated to let me put a server in their data center for free?

That's not what peering is, though, right? Nobody's asking for anything like this. They're asking for building a couple links between a small number of large networks, or have I got this wrong?

Nobody's saying build out huge infrastructure that belongs to a particular website for free.

Comcast would very quickly start paying for upgrades if they were losing customers over it. I'd love to switch to someone else.

Peering is only broken because consumer ISPs are monopolistic, and the big ones are some of the most hated companies in the US, and don't seem to care.

> Comcast would very quickly start paying for upgrades if they were losing customers over it.

This seems irrelevant. They're not losing customers.

Is Comcast obligated to "lose" those customers themselves, as some sort of self-punishment?

> I'd love to switch to someone else.

About 3 weeks ago, I had an idea. It might have not been a very good idea. But I turned to my coworker, and I said "hey, those signs Google gives out in Kansas City for people to put in their yards... do you think we could make the local news if we could get 100 people to put them in their yards?".

He said he thought that might actually be possible. And if we could get on the evening news, maybe we could drum up support for our city council to actually do something about it.

The signs cost about $600 (give or take), and we launched an indiegogo campaign that will end soon.

No one is helping though. We've got hundreds and maybe even thousands of Facebook likes, but people won't even donate $5. Hell, they'd end up getting the yard sign that the $5 buys. But they're not just cheapskates, they're also dumb as rocks. One person was complaining about the idea that someone else might get a yard sign if they donated... as if it matters whose yard the sign ends up in, supposing this works. Another said it was a waste of money, that we should just go speak at city council... absent the display of public support of the idea, apparently.

Despite all the griping and complaining, I'm having alot of trouble believing anyone who says they want something different. Maybe you would want something different, if it just happens to you. But none of you really deserve anymore than you're getting.

> Peering is only broken because consumer ISPs are monopolistic,

That's dumb. I've already described how and why peering might be broken. If you had Comcast and Evil Universe Comcast both available where you live right now... we'd still be going through the same crap. Why?

Because Netflix would be flooding both of them with asymmetric traffic. And both would still refuse to pay for the upgrades that only benefit Netflix.

It's not a monopoly problem. You'd get that if you bothered to think about it.

> some of the most hated companies in the US,

Some of that hatred is deserved. But that doesn't change the fact that it may not be their fault here.

So blaming them, even punishing them... that's not going to fix it.

> And both would still refuse to pay for the upgrades that only benefit Netflix.

Those upgrades benefit the consumer, not netflix. The consumers that are paying the ISP for alleged internet service. Then they're paying netflix for the right to access netflix's content using their existing internet provider. The means of moving the bytes is a cost on netflix's end of the network, just like on the ISP's end.

I think you're missing the overall point. This isn't a nerd problem, its a MBA probelm.

From a networking engineering point of view, these problems are very solvable the way they always have been -- more stuff.. You increase bandwidth to the peering points, or find other ways to deliver the traffic (ie. have Netflix use dark fiber to deliver traffic to different regions, etc)

The cost of doing this is negligible, and on the business ISP side, the ISPs are doing an excellent job of solving these problems as enterprises start sending gigabits/sec of data to O365, etc.

The reason they are doing this is that the increased traffic is attacking television distribution. When you are Comcast, a wholly owned subsidiary of NBC, you tend to care about that.

> The reason they are doing this is that the increased traffic is attacking television distribution.

I disagree. I admit that this is subtle, and I don't fault you for seeing it immediately. If it were only true that Comcast was afraid of Netflix as a media competitor, then Comcast could easily beat Netflix at that game, and likely reap a few cost savings. So it's not this. I suspect that at least some of the executives know that Comcast as a tv distributor is already a lost cause, and that this doesn't bother them too much.

What does bother them is, having run the numbers, they realize that Netflix traffic will only continue growing, on a very steep curve. While some chump change upgrades could fix the immediate problem (probably cheap enough that they'd do it just for PR), it won't fix the bigger problem: what happens in 4 years when the HFC network itself is inadequate?

They can't afford to go to an all-fiber network. That's billions of outlays that they'd have to recoup over decades.

They're afraid of losing their internet business, not their tv business.

These aren't local-loop issues, it's core network.

Will they have issues with the local coax networks getting saturated in the coming years?

Sure -- they do anyway, which is why they are investing in DOCCIS 3 in all markets, and experimenting with coax/fiber hybrids in a few places. None of their equipment works forever, they refresh the networks in 5-10 year cycles.

The issue is that they are like an old-school bank or telephone company. They own infrastructure, and extract a fee for running it that doesn't fluctuate much. They want to be higher growth companies, but the only way to do that is to generate more revenue.

If you surrender and let them say "Netflix's bandwidth needs are soooo complicated, we must be compensated!", then tomorrow they'll say "VoIP and Gaming latency needs are soooo complicated, we need more $$$ for that too!"

> In 2014, this may not be the case. Level3 wants to send much more traffic to Comcast than Comcast would ever send to Level3. This is because Netflix (a Level3 customer) is asymmetric in nature.

Oh, you're so right! That awful, no-good, pesky L3 just keeps sending these packets that nobody wants and nobody ever asked for... oh wait. Oh yes. I just remembered. L3 sends those packets because Comcast's customers requested them. People pay Comcast to transfer packets between themselves and the internet services they wish to use. In a free market, Comcast's inability to do that would result in customers leaving in droves. It really shouldn't matter (within the bounds of legal reason) what's in the packets.

> That awful, no-good, pesky L3 just keeps sending these packets that nobody wants

You don't seem to understand peering at all. That's ok. Be quiet, listen to the rest of us that do.

No one gets to connect to Comcast's network unless they pay. You and me, we pay a monthly bill.

Peers pay the way by accepting Comcast's traffic in (roughly) equal amounts.

Level 3 isn't paying. You're absolutely right, Comcast doesn't want those. If you want them despite this, then the onus is on you to connect to Level 3 directly.

> People pay Comcast to transfer packets between themselves and the internet services

No they do not. If they believe this, they are mistaken. If that is a problem for them, they're free to discontinue doing business with Comcast.

Comcast cannot even charge them and offer a special "Netflix connectivity" service, because your herders have already taught you how to chant "network neutrality".

If all of this bothers you very much, then I would hope that you take the time to think carefully and profoundly and realize (as I have) that perhaps peering no longer works.

> In a free market, Comcast's inability to do that would result in customers leaving in droves.

They can leave now.

> It really shouldn't matter (within the bounds of legal reason) what's in the packets.

It doesn't matter. It matters where they're from, and that they're in roughly equal amounts to what Comcast sends.

"You don't seem to understand peering at all. That's ok. Be quiet, listen to the rest of us that do."

You don't seem to understand reality at all. If you can't say something nice, STFU.

As to reality:

Yes, peering often works the way you describe, but Netflix tweaking their player to ship back a bunch of garbage traffic to something on L3 (Netflix themselves, or otherwise) would balance those ratios and change "who should pay" in terms of your analysis, while being a total non-solution.

The simplistic analysis here is inappropriate. This is about 1) the fact that Comcast has a virtual (or actual) monopoly in many markets, and 2) Netflix competes with Comcast's other business. Abuse of market power to restrict unrelated trade is awful and often appropriately illegal.

> If you can't say something nice,

No one else is saying anything nice at all. They're spouting dumb shit, they're directing it at politicians and bureaucrats who will only make it worse. They refuse to stop and think, even though that would only take a moment.

This is much worse than any words I've ever said on the issue. It's worse than any name-calling I've ever done.

> Yes, peering often works the way you describe,

Not often, but always.

It is a voluntary arrangement. And if we try to force it to work differently, even when it's to one side's disadvantage, the logical thing is for them to decide not to peer at all.

But you've not thought it through very far.

> but Netflix tweaking their player to ship back a bunch of garbage traffic to something on L3 (Netflix themselves, or otherwise)

So your argument is that Netflix could cheat and make the world a worse place, and that Comcast should just give in to the implied extortion?

Something's wrong with your head if you can say these things and not give them a second thought.

> Netflix competes with Comcast's other business.

That's even less reason for Comcast to cooperate then. Comcast isn't obligated to give away free service to competitors, either legally or morally.

> Abuse of market power

What abuse? The voluntary agreement was for settlement-free peering for (roughly) equal levels of traffic.

Netflix is the abuser here.

"Something's wrong with your head if you can say these things and not give them a second thought."

Again, there's no reason to be abusive. I'm not going to respond to any further comments in this thread that don't take a more respectful tone.

As it happens, I seem to have given it deeper thought than you have:

"So your argument is that Netflix could cheat and make the world a worse place, and that Comcast should just give in to the implied extortion?"

No, my argument was that "same amount of traffic" is obviously a meaningless metric. In my example with Netflix passing back garbage data, you can call it cheating... but what if that data was useful?

Let's imagine the following:

Netflix cuts a deal with some content providers, such that they will provide (anonymized) information on facial expressions and eye tracking of (opted-in) viewers while content plays. Studios are pay for this. Netflix passes some of that money along to customers who opt in. Netflix decides to do the processing server side, because of the resources required for whatever analysis they're doing.

Now the ratios are significantly more balanced, to the benefit of Netflix (who is making more money), the Netflix customers (who are paying less money), and the determent of Comcast (who is passing more traffic). How is it right that Comcast is owed less money in that case? They probably need to build out more infrastructure than in the present case.

Settlement-free peering fits when there is roughly equal levels of value derived from the traffic. That only sometimes corresponds to roughly equal levels of traffic.

You haven't earned my respect. At every opportunity, you lose what little there might have ever been.

If you won't listen to reason unless reason strokes your ego, then you were never interested in it anyway. There are plenty of people willing to treat you with the false respect you think you deserve, you don't need me.

And everything you get, you'll deserve it.

> No, my argument was that "same amount of traffic" is obviously a meaningless metric.

It's not meaningless. It's what was agreed upon.

How would you like it if someone made a deal with you, and then a few months later wants to change it because "hey that stuff we agreed to is meaningless"?

Seriously, what the fuck would that even mean? It would be nothing more than someone trying to bully you.

No judge would rule in that guy's favor. If (when) it made it to court, the judge would say "this is not meaningless, it is what was agreed to in the contract".

When you say "meaningless", what you're saying is "I don't give a shit about the agreement, I just want things my way".

The real world doesn't work like this.

> Settlement-free peering fits when there is roughly equal levels of value derived from the traffic.

Yes. And when that is not the case, one party either pays the other... or they stop peering.

You get that right? Comcast has more than one peer. They don't need Level 3. As others have said in replies to my own comments, no one is going to dump Comcast over that, so Comcast wouldn't have much to lose except some bad press.

I'm not asking you to stroke my ego (or anything else). I'm asking you not to be a dick, because this forum is of most use to everyone when people aren't needlessly dicks. I'm done here, per my earlier statements.

> Level 3 isn't paying. You're absolutely right, Comcast doesn't want those. If you want them despite this, then the onus is on you to connect to Level 3 directly.

When Comcast stops advertising an Internet connection and starts advertising a "connection to the Comcast network", then you'll have a leg to stand on.

Until then, here's your sign.

> When Comcast stops advertising an Internet connection and starts advertising a "connection to the Comcast network"

So you don't know what "internet" is either. That figures. It's just a magic computer thingy that let's you watch german scat porn and cat pics on Facebook.

I don't think "Netflix feels cheated" is necessarily accurate. It's entirely plausible that Netflix's PR stance is not aligned with the internal understanding of appropriate pricing.

That would be even worse if true.

That means the public is composed of blithering imbeciles who rally to inane causes without even understanding what's going on. Please, I beg all the gods that you are wrong.

> That means the public is composed of blithering imbeciles who rally to inane causes without even understanding what's going on.

This has been my overall experience when dealing with people on the internet or otherwise.

I agree with you, for what its worth - Settlement only works if traffic is near symmetrical otherwise some money needs to change hands. It's just a matter of who is doing the paying for the extra bandwidth, it all has an incremental cost, and someone at least has to pay for the bearer.