Why is Netflix Buffering? Dispelling the Congestion Myth | Hacker News Reader

Why is Netflix Buffering? Dispelling the Congestion Myth

publicpolicy.verizon.com

106 pointsdanielsamuels214 comments

From the article: "Netflix sends out an unprecedented amount of traffic."

YES! Yes they do! But Netflix doesn't send that traffic for fun and giggles, just to swamp your network and clog up the port. They do it because YOUR CUSTOMERS HAVE ASKED THEM TO and you, Verizon, you duplicitous shitbag, have SOLD THAT CAPABILITY to your customers. They have LITERALLY PAID you to successfully receive up to 75Mbps worth of traffic from ANYONE, INCLUDING NETFLIX.

I can't fucking stand this "they send a lot of traffic our way, but we don't send any back!" bullshit. Netflix doesn't PUSH traffic to your customers, your customers PULL traffic FROM Netflix.

Maybe we're getting to the point where Internet connections are more like public roads.

Maybe no private entity should control these roads.

Maybe it's time for a sweeping change.

The "sweeping change" is an old legal standard: "common carrier". They're contracted to move bytes from source A to customer B at "up to" X Mbps; so long as (albeit limited) bandwidth is available at any given moment, fill the pipe and shove that data thru regardless of source. Traffic shaping may be fair if the pipe is full, but not if unused bandwidth remains sparsely used based solely on source or content.

Raise your hand if you've contacted your elected representative regarding ISPs becoming Common Carriers.

(Full disclosure, I'm not an American and thus have little to no say regrading American ISPs, but I advocate for this in my home country.)

/me largely expects crickets in response.

Unfortunately I think we just aren't pissed off enough about this as a group yet, evidenced by the lack of action from us and from legislators.

Raise your hand if your elected representative has any idea what having ISPs become Common Carriers means or entails. Keep your hand up if 10% of your district also knows. Keep your hand up if you (and your district collectively) can out lobby Comcast, TWC, ATT, Verizon, etc.

I don't know that it's that people aren't pissed off, its probably more that the people who are pissed off are uninformed, will not become informed easily, and will not organize as easily as already well organized and very well funded corporations, lobbyists, and PACs.

I think this might be a clear description of what would happen if roads became privatized.

Imagine UPS as Netflix and a turnpike operator as Verizon.

Or just call it a toll road. Of course those tolls will go up every year for things like maintenance and improvements and other esoteric vagaries....

They have LITERALLY PAID you to successfully receive up to 75Mbps worth of traffic from ANYONE

Aye, there's the rub: "up to". Nothing says (and perhaps deep in the EULA there's the opposite) you'll get that, or even a substantial fraction of unused shared bandwidth, commensurate to your needs.

Customers should push the meme that "up to X Mbps" really means "you won't get more than X Mbps, and likely much less".

Customers should be writing Verizon et al complaining "you CAN send that data faster but you aren't, and we have proof you're choosing to practically renege on our contract."

Yeah, but we're talking about a 1-5MBps stream here. Even though the headline 75Mbps number is, as you say, a "we promise never to exceed", I don't think it's unreasonable to expect that the ISP provide 7% of the listed capacity.

+1 for "duplicitous shitbag". That really hits the nail on the head.

Peering is the part I wonder about. Netflix pays for the pipes to upload. Customers pay for the pipes to download. But who pays for the peering pipes in the middle? Should it be Netflix's ISP? Customer's ISP? Both? Neither?

Exactly this. I try to explain this to people in layman terms like this: Comcast ist trying to charge the water fountain, saying it "wants" to pump to much water into people's homes.

Also, don't they tend to sell service with much higher download capacity than upload? If so it's arguably their fault that traffic is unbalanced.

I almost upvoted this.

Verizon is blaming Netflix for sending "an unprecedented amount of traffic" and "not taking steps to ensure that there is adequate capacity for their traffic to enter" Verizon's network.

The thing is, Netflix is NOT pushing any of that traffic to Verizon. It is Verizon's customers who are pulling that traffic from NetFlix -- and they are paying Verizon to deliver it!!!

More here: https://news.ycombinator.com/item?id=7701494

You're paying Verizon for a connection to their network, but that doesn't make it Verizon's responsibility to pay to upgrade every intermediate network on the way to every destination a Verizon customer might request data from. That is not how peering on the internet has ever worked.

See this circa 1999 article about peering and settlement on the internet: http://www.cisco.com/web/about/ac123/ac147/archived_issues/i... (Part I); http://www.cisco.com/web/about/ac123/ac147/ac174/ac200/about... (Part 2). Note that this was published by Cisco 15 years ago, long before any of these disputes arose. We're talking about a very settled understanding of how the internet works that Netflix is trying to overturn to help its own bottom line.

I think many may be like me and literally thought we were buying access to the Internet at large. It wasn't until recently that I've had to learn how the thing actually works (for small values of learn and actual - just enough for some intranet hacking). I suspect a lot of the outrage is simply that we feel the service has been deeply misrepresented, from what qualifies as "high speed broadband" to what "Internet" access really is.

I dunno who is going to cave first, but my money's usually on the angry ignorant masses.

Wrong -- I'm paying Verizon for access to the internet -- more specifically, for access to Netflix. If I can't get Netflix, I don't pay Verizon. I either get a new ISP, or don't have internet at all.

It's rather simple. Verizon's customers are consuming traffic, and it's Verizon's job, as a consumer network, to facilitate consumption of said traffic.

There is no such thing as the "internet." What appears to be the "internet" is just a mesh of interconnected but private networks all running the same protocol. The Cisco article I linked to above shows you what you're actually buying. What you're talking about is an illusion that only exists in software.

That's funny, why does verizon call their service "high speed internet service", when the is no thing as the internet. And if you go to the landing page for high speed internet access on the verizon homepage, it becomes immediately obvious that they are selling it as a internet service.

Should we now start talking about there is no such thing as a video, just 1's and zero's, or, there is no such thing as 1's and 0's, just timed voltage oscillations. Or perhaps in verizon's case, there is no such thing as customer service, just opportunistic moneymaking leveraging monopoly and an army of lobbyists.

I mean to say that there is no such thing as what software folks think of as the "internet": the sort of idealized network that you think of when you're dealing with sockets instead of edge routers. Obviously "internet" exists as a marketing term, but it refers to what's described in the Cisco article, not to the idealization.

Do you get upset when you sign up for a United flight and one of the legs is operated by a codeshare partner? It's the same thing: there's an idealized abstraction of a "flight from New York to Burkina Faso" and the actual reality that the underlying physical service is a distributed one.

Abstraction or not, you are missing the bigger picture.

1) Level3 has offered to split the cost with the Big5 ISP's of upgrading their infrastructure.[1]

* This would increase throughput and capacity for both Level3 and the consumer networks.

2) Netflix has offered free Open Connect Appliances to all major ISP's.[2]

* This would allow consumer networks to only download a video once, then serve it up locally within their network to all their customers. In-network traffic is almost free for the ISP's.

3) Netflix has offered to change their entire distribution model into a P2P model.[3]

* This would allow consumer networks to only download a video once, then serve it up locally within their network to all customers. In-network traffic is almost free for ISP's.

All of these options have been flat-out refused. The Big5 ISP's are purely after the money. There is no other compromise for them.

[1] http://blog.level3.com/global-connectivity/chicken-game-play...

[2] https://www.netflix.com/openconnect/hardware

[3] http://arstechnica.com/information-technology/2014/04/netfli...

In a world where everyone watches Netflix, the traditional peering arrangements dramatically favor companies like Verizon. Unsurprisingly, they're fine with the existing peering arrangements. Companies like Level 3 are offering to "share the cost" of upgrading the networks, because under the traditional arrangements, they would have to bear the full cost of the upgrades. Everyone in this discussion is just looking out for their own bottom lines.

Deleted comment

We heard this excuse from Blockbuster and Hollywood Video when RedBox and Netflix ate their lunch

This comment is a complete non sequitur in an otherwise good exchange. Blockbuster and Netflix weren't in dispute over who should pay for sharing an infrastructure node shared between them. Instead it's a generic "popular Internet company good, other company bad" argument.

There is this tendency to paint any dispute between established companies and new internet companies as "old-world archaic business models" holding up progress, but it's inane. There's nothing obsolete about selling bandwidth for money. Indeed, in a "modern data-driven society" building and owning the pipes is more relevant and important than ever!

> I mean to say that there is no such thing as what software folks think of as the "internet": the sort of idealized network that you think of when you're dealing with sockets instead of edge routers. Obviously "internet" exists as a marketing term

So what you're saying is, Verizon's selling a product ("internet access") that doesn't exist—and that it's not their fault that they can't provide it, since it doesn't actually exist? Maybe they shouldn't be selling it, then.

The product ("internet access") is the same thing that everyone sells under that label: access to a private network that's connected to all the other private networks that are sold under the same label. That's what the "internet" has always meant. What doesn't exist is what software folks and net neutrality proponents think of as the "internet": this idealized homogenous network where every path between any two TCP/IP sockets is equally good. But you don't get to take a well-understood label, apply your own idealized definition to it, then attack companies that are using that label consistently with its traditional definition.

> But you don't get to take a well-understood label, apply your own idealized definition to it, then attack companies that are using that label consistently with its traditional definition.

If someone were to talk to Verizon's sales people, and watch their commercial, and read their sales copy—and then be presented with those two definitions, which do you expect they would believe Verizon is using?

You are arguing over the semantics of the word "internet" and completely missing the bigger picture.

I only pay Verizon so that I can have access to Netflix content. No access to Netflix, then I'm not a Verizon customer any longer. Verizon is a Consumer Network -- so by definition, almost all of their traffic will be ingress.

Level3 has a great blog post detailing the issues at hand. Specifically, they state the consumer networks are refusing to upgrade their networks, and this is shifting the burden to Level3. Level3 knows they are refusing, because they have discussed the matter with the "Big 5" ISP's here in the states.

Level3 has even offered to split the cost with the Big5, which as they pointed out, is a win-win for everyone involved. Yet, the Big5 still refused.

http://blog.level3.com/global-connectivity/chicken-game-play...

> Wrong -- I'm paying Verizon for access to the internet -- more specifically, for access to Netflix. If I can't get Netflix, I don't pay Verizon. I either get a new ISP, or don't have internet at all

You might not like the logical conclusion: the ISPs charge people for what they actually use.

There is no logical inconsistency with the notion that wired ISPs offer flat-rate unlimited service and periodically upgrade their networks to be able to provide that service. The cost of the upgrades are completely sustainable notwithstanding Netflix. Verizon has no trouble allowing its customers to watch an unlimited amount of FiOS TV.

The cause of this dispute is that the ISPs each have a monopoly over access to a large distinct block of customers and want to leverage that into a tax on content distributors, creating an unlevel playing field in favor of the ISP's own over the top services in competition with services like Netflix.

Not necessarily. If Level3 is be believed (and they are credible imho), the Big5 ISP's here in the states are just flat-out refusing to make any network upgrades. Level3 has even offered to split the costs with them, since it benefits Level3 as well.

What happened to all that "National Broadband Act" funding the Big5 ISP's got? What about annual infrastructure upgrades? These guys are not coping with the data-driven society the wold is now, and are trying to stretch their buck the furthest (I don't fault for that, it's business).

However, when they wrongly shift the blame to others, it really fires me up.

What's hilarious here, is that Level3 played the exact same game Verizon is playing now. Back in 2005 they dropped their peering entirely for several days with Cogent due to "Unbalanced Traffic Ratios" as cogent was sending L3 more traffic than L3 was sending them.

It's always historically been the case that settlement free peering required balanced traffic ratios. The difference is L3 used to have far more consumer networks buying transit from them. Before, level3 was needed to connect a bunch of different regional ISPs together who each purchased IP transit. Now, every consumer ISP of any size can orderup a nationwide backbone network (from level3 ironically, yes despite being in a peering war for ip transit, comcast's nationwide fiber network is leased from level3.)

Many of the old Ma Bells that were broken up used to be clients of level3. Through the mergers, many of them now own Tier 1 backbone networks. Verizon, ATT, and CenturyLink all are now also competing backbone providers with Level3 as well as being consumer ISPs. This leaves Level3 in the position Cogent was in when L3 depeered them.

Level3 will now gladly split the costs for unbalanced traffic ratios now that it benefits them. 10 years ago they would have tried to extract transit/paid peering from the same situation from someone else.

Perhaps the whole balanced traffic ratios requirement doesn't work in a time when every consumer ISP has their own nationwide fiber network and many also own Tier 1 IP networks. Or perhaps expect Level3 to start looking to get bought/merge by Time Warner cable or Cox to even the playing field.

Why is Level3 particularly credible to you?

Because the cost of Internet access in the United States is higher, and quality of service lower than what it is in other markets serviced by Level3?

1) Because they aren't a last-mile carrier, but instead a backbone provider and thus don't have chips on the table.

2) As trevelyan said, they operate around the world, and state that this situation, and mindset of last-mile carriers, is unique to the US.

3) One would think a backbone provider would share the same views as the last-mile carriers, since it would mean more money for them too. However, they state that big content providers such as Netflix are very much so not a problem, but instead it's the last-mile carrier's refusal to make necessary network infrastructure upgrades to augment capacity.

This information is available on their blog. This post in particular worth a read: http://blog.level3.com/global-connectivity/chicken-game-play...

I would have no problem with that, and I think the only reason we haven't moved to that model already is that we started out with the "unlimited" model. That's how my other utilities work.

Great comment -- it deserves an answer.

Imagine for a moment that we're talking about water service instead, and that we live in an alternate reality in which, due to accidents of history, the water network evolved much like our Internet, with peering and settlement agreements between privately owned water networks.

So, to borrow your words: you're paying your water service provider for a connection to their water network, but that doesn't make it their responsibility to pay to upgrade every intermediate water network on the way to every home what might need water -- even if water pressure drops every morning when everyone in town wants to take a shower.

Regardless of the technicalities, that doesn't seem right, does it?

Just as water service providers in that alternate reality would be responsible for making sure customers can pull enough water to shower every morning, ISPs are responsible for making sure their paying customers can pull the traffic for which they have paid.

But the problem here isn't that you can't fill your internet connection, period, but that you can't fill it with traffic from one particular endpoint. There is a bilateral aspect to the problem here that's totally ignored in your hypothetical.

Different example. Air Canada has an unlimited North America flight pass: http://www.aircanada.com/en/offers/air/wallet_na_faq/wallet_.... You can characterize this pass as buying you "unlimited travel to any destination in North America." Okay, so is it Air Canada's fault if you really want to go to Buffalo regularly, but you often have trouble booking a flight because of limited capacity at BUF? Does Air Canada have to pay for the airport upgrades? After all, it's their customers that want to go to Buffalo, and they're paying to travel to any destination in North America.

The bilateral arrangement here is between the ISP and an internet backbone - in this case, likely Level3. Level3 has stated that they are willing to split the cost of upgrading the link. Verizon is unwilling to do so. It's also useful to note that the cost of upgrading a link is tiny in the terms of these companies - on the order of $10-20k[1]. Netflix already pays to have sufficient capacity at their end of the connection; it is the ISP's responsibility to do the same, since their customers are paying to access whatever internet content they want. It is unreasonable to expect Netflix to pay for a separate, unnecessary connection. That's more like if the Buffalo airport decided they were getting too many flights from LA, so if anyone else from LA wanted to fly to Buffalo, they would have to pay for the addition of their own private terminal. It's not the way things are normally done, and it doesn't make sense.

[1] http://blog.level3.com/global-connectivity/observations-inte...

No, the problem is that the ISP's network does not have sufficient capacity at that particular endpoint to receive the traffic pulled (and paid for) by its customers. The bottleneck at that endpoint is the ISP.

PS. Travel is far too different to use in an analogy -- e.g., it doesn't involve delivery of continuous services to customer premises.

Your analogy paints the internet as a distribution system, with some fungible commodity (water, electricity) flowing from sources to sinks. That may be how Netflix uses the internet, but that's not how the internet is designed. It's designed to be a mesh, with nodes being peers to each other. The distributed structure of ownership isn't an accident of history. The internet was designed around that structure. Its much more like the hub and spoke airport system then a water distribution system. And as in the airport system, its not the sole responsiblity of the most popular destinations to pay to upgrade congested hub nodes.

Yes, my analogy paints the internet as a distribution system for a fungible commodity, because bits ARE a fungible commodity, just like water and electricity. A 1 is indistinguishable from every other 1, and a 0 is indistinguishable from every other 0.

In case it's not obvious, I agree that ISPs are NOT reponsible for upgrading infrastructure outside their network. What I'm saying is that ISPs are responsible for upgrading their network so it has sufficient capacity to receive the traffic pulled (and paid for) by ISP customers.

> See this circa 1999 article about peering and settlement on the internet

The purpose of the arrangement you're pointing to is so that transit providers (e.g. Level 3 or Cogent) could distinguish other transit providers from transit customers whose traffic was characteristically asymmetric. The reason transit providers care more about upload bandwidth than download bandwidth is that customer ISPs can easily offer web hosting services to balance their traffic load, so it would be fruitless to charge the ISPs more for not sending the transit provider more traffic they have to carry. Better to give the ISPs free transit than that.

But Verizon isn't the transit provider here, they're the ISP. By sending Verizon a lot more traffic than they receive, Level 3 is effectively offering Verizon a huge amount of free transit in the other direction. That's like free money. Verizon could trivially use it to out-compete AWS on bandwidth pricing and make a boatload of easy cash. But that wouldn't unbalance the playing field for competitors of Verizon's video services.

Please remember that Verizon is also a Tier 1 transit provider! They own (via Verizon Business) the old MCI/Worldcom/UUnet backbone that was one of the largest transit providers. Level3 and Verizon are both transit providers.

Verizon is not a transit provider in this context. Level 3 is not asking Verizon to take traffic in New York and deliver it to customers in California or Germany. They're giving Verizon traffic destined for local Verizon customers.

Importantly, there is no other way to reach those customers. If you need to get traffic from New York to San Francisco you can use Level 3 or Cogent or Verizon or any of a dozen others. If you need to get traffic to a particular household which is a Verizon subscriber, there is no path other than Verizon.

Hey rayiner. I just wanted to say I appreciate your posting on HN about net neutrality and topics related to this and how diligent you are about it even though it's often contrarian (but right) and going to get you downvotes. I upvote every comment you make whenever I see a story like this unfolding in exchange for being unwilling to fight the good fight myself. I don't know how you do it, but thanks.

Not to be devils advocate, but if I have 75mbps pipe at home and some internet server is on 1mbps, I'll still be buffering because of the server connection not mine. Right?

You shouldn't be downvoted for this. It's a reasonable question, it's just not what's happening. The traffic is reaching Verizon (or Comcast etc's) network at reasonable speed, and then congesting.

But it's also not right to say it's reaching Verizon. It's not. It's just hitting the edge of their network.

Now you can say that Verizon is responsible for making sure the edge connection works properly.

But you could also say that Netflix shouldn't just dump it's entire load onto shitty congested peer points.

Think of Verizon as an island city with 10 bridges. Netflix is trying to send 1/3 of all the traffic through one bridge. This causes congestion on that bridge when the other 10 bridges are moving fine.

Verizon is saying, use a different bridge they are all open.

Netflix is saying, add more lanes to the one I use because I'm delivering loads to people who ordered it.

Both are valid arguments. However, one thing to consider is that Netflix could switch ISPS tomorrow and start dumping their load on a different bridge.

Realistically it's absurd that Netflix isn't use a whole variety of ISPs to manage the load themselves. Other big data providers do it. Why can't they?

But also, Verizon is probably playing hardball because they are a competitor.

The real question is whether Verizon is saying "Use the other bridges" or "If you pay us we can open another bridge for you". What I've read mostly points to the latter and Netflix is either taking a stand against this practice or Verizon wants to much money.

> Realistically it's absurd that Netflix isn't use a whole variety of ISPs to manage the load themselves. Other big data providers do it. Why can't they?

IIRC, Netflix offered to provide hardware to all the major ISPs that would cache Netflix content at the ISP data centers. Not a single one took them up on the offer.

That was the Open Connect Hardware. Free big storage boxes for all ISP's who wanted them.

They also offered to change their entire distribution model to P2P as-to keep the traffic burden inside the consumer isp's network (which effectively costs them nothing, it's ingress traffic that is the major costs).

Level3 has even offered to split the costs of upgrading the Big5 ISP's peering connections.

All options were refused.

> They also offered to change their entire distribution model to P2P as-to keep the traffic burden inside the consumer isp's network (which effectively costs them nothing, it's ingress traffic that is the major costs).

Why wouldn't they do this anyway? How does the ISP have any say in whether or not Netflix distributes their content via P2P? Or am I misunderstanding?

P2P would make the service quality depend on how many people were online, etc... I think they'd rather sort it out another way.

But regardless, even with that offer, Comcast scoffed at it, citing Netflix would still be a "large burden" to their network. (which would be even more-so false than it currently is).

But they weren't going to pay for rack space and power. Also, IMO offering free rackspace violates net neutrality since a start up couldn't get a similar deal.

Netflix could do something differently, the question is really whose responsibility is it? The fact that the data gets to the edge of Verizon's network uncontested means the congestion is on Verizon's network. The fact that the traffic was requested by their members, and then delivered uncongested to them tells me it's on Verizon to fix the issue.

In an ideal world Netflix would pay a single ISP for access to "the internet" and their traffic would be handled properly across peered networks. Unfortunately it's not an ideal world, and so Netflix has made a bunch of philosophical compromises to increase service for its customers (including setting up other Fast Lanes). But that doesn't mean we should expect Netflix to continue to do this.

And Verizon says that they have plenty of open peering points and it's Netflix's responsibility to deliver it in a responsible manner.

>In an ideal world Netflix would pay a single ISP for access to "the internet" and their traffic would be handled properly across peered networks.

Netflix can do that now, but its not what they are doing. It's paying an ISP to send it over only their peering points by choosing the lowest bidder. It saves Netflix on bandwidth costs. A normal ISP will send the data by any means to get it there. Netflix's ISP is just letting their saturated points sit there. That's the ISP's fault.

>The fact that the traffic was requested by their members, and then delivered uncongested to them tells me it's on Verizon to fix the issue.

You could easily make the argument that Netflix is selling their service and is the one sending the data. It should be on their ISP.

> Verizon is saying, use a different bridge they are all open.

You're assuming Verizon isn't over-subscribing all of their other peering connections too.

> Both are valid arguments. However, one thing to consider is that Netflix could switch ISPS tomorrow and start dumping their load on a different bridge.

Your analogy is broken because peering connections aren't like bridges. Expanding a bridge costs millions of dollars and can't reasonably be moved from where you build it. 10Gbps switch ports do not cost millions of dollars and can trivially be moved to a different peer if loads change.

According to Level3 - a 10Gbps link bears only a one-time cost of $10K-$15K USD.[1](see comments)

Chump change for major ISP's.

[1] http://blog.level3.com/global-connectivity/chicken-game-play...

>You're assuming Verizon isn't over-subscribing all of their other peering connections too.

The PR release linked as the topic of this thread claims it's only 44% on the other connections.

>Your analogy is broken because peering connections aren't like bridges. Expanding a bridge costs millions of dollars and can't reasonably be moved from where you build it. 10Gbps switch ports do not cost millions of dollars and can trivially be moved to a different peer if loads change.

There isn't just one peering ports there are many many peering points.

It's also cheap for Netflix to use another ISP who won't cram it down saturated peering points.

Finally, if it's so cheap why won't LEVEL3 or Cogent pay 100% of the cost?

> The PR release linked as the topic of this thread claims it's only 44% on the other connections.

Percentages don't tell you anything. The fact that some 10Mbps link is at 23% utilization is of no utility. Neither is the capacity of links not connected to the relevant customers. The real question is, if Netflix used some other transit provider and their traffic went over some other Verizon links, wouldn't those links still get saturated?

> There isn't just one peering ports there are many many peering points.

...and?

> It's also cheap for Netflix to use another ISP who won't cram it down saturated peering points.

If that were true then there would be no dispute.

If they're pushing more than 1mbps to all connections, yes.

Why are Bridges Dangerous? Dispelling the Killing Myth

signed: The Trolls

Is there a special lane for billy goats?

Well, Netflix is pushing traffic to Verizon. Saying they're not is factually incorrect.

They're just doing so in response to requests coming out of Verizon networks.

Still though, Verizon customers are running Netflix apps, which are making the requests, so it's not like customers are doing the requesting "in the raw", or as "informed" consumers. They're just clicking "play" on Netflix video devices.

Also, Verizon customers aren't paying Verizon to deliver content, they're paying Verizon to try to deliver content. This goes all the way back to the "unlimited" deception ISPs pull on consumers, and Verizon is double-guilty of this deception, as they're a cell phone carrier as well.

Edit: I really hate to be that guy (and I know I'm breaking rules by saying this), but have we really devolved to the point where we downvote everything that isn't 100% in agreement with the hive mind? I thought HN was better than this. I'm just getting so sick of being downvoted because I'm offering a different angle - I've been polite, deferent, and explanatory, and my words get sent straight into the "grey" anyway. You're given downvote privileges because it's been determined that you're capable of thought, please don't disprove that.

That's silly word twisting. Following that logic, my bartender is pushing drinks on me, just by my request.

Bartenders will get fined and lose their license if they serve drunk people.

Also, calling what I said "word twisting" isn't really addressing their argument, it's more or less just name-calling, which companies are really good at ignoring. To beat them, you've got to do better than that.

> Bartenders will get fined and lose their license if they serve drunk people.

only because they don't have a large lobby organization to protect them from the laws ;)

Heh yeah, I mean the "let people get wasted in public" lobby isn't quite as big as the ISP lobby.

The analogy isn't great, I'll agree. I prefer the "water" analogy, myself.

I think we downvote people who just post nit-picky irrelevant "corrections".

The reply about a bartender "pushing a drink on you" is spot-on for why your point is pointless and not in need of visibility.

What's irrelevant about "some of the major points you said are factually wrong"?

It just sounds like you're okay with misrepresentation of the facts, because it agrees with your overall point.

Also, like I said in the above comment, bartenders can lose their license and get fined heavily for continuing to respond to requests for drinks once a user is "saturated", so I don't think it's the best analogy possible for your argument.

A better analogy might be water usage, which lends itself well to an "Internet as a utility" segue.

You wrote:

Well, Netflix is pushing traffic to Verizon. Saying they're not is factually incorrect.

Then you wrote:

They're just doing so in response to requests coming out of Verizon networks.

The point at hand, of course, was whether it's fair to frame things in terms of "Netflix pushing traffic to Verizon" when the reason for the traffic is requests initiated by customers on Verizon's network. Nit-picking about whether someone is "pushing" data when responding to a request does not in any way further the discussion. Hence, downvote into invisibility.

Incidentally, the way to continue the script here is for you to ignore everything I just wrote and continue focusing with laser-beam intensity on bartenders serving drunk people in order to cover the fact that you did not have any relevant point to make in your original comment.

Netflix wrote all of the apps people are using to make requests to Netflix. It's important to note that, because that means Netflix is in complete control over the number of requests that come "from" Verizon. To say that customers are directly making the requests is factually incorrect - Netflix is making the requests, to itself, over the connections of Verizon customers.

Important distinction, because Netflix is in control the whole time, and can do things, if it wants to, to mitigate the flood of requests. It's not, and in fact is intentionally ramping up the volume of requests by offering content such as 4k resolution videos, most likely to further push this issue.

So no, again, this isn't a nit-pick, it's a central point to this conversation.

I don't really understand why you're being such a jackass about this, though.

Wow, so your claim is that Netflix is maliciously, and intentionally, offering high(er) definition content not to please its customers, but to spite ISPs?

And you're saying that with a straight face...

And wondering why people aren't taking you seriously?

Just like your argument that Netflix is 'pushing content to Verizon consumers', ignoring that 'Verizon consumers are requesting said content'.

Maybe one of the things Netflix could do to deal with the volume of data is let Verizon customers know that that their service is being degraded because of that volume...

Wait... I'm getting deja vu...

Maliciously? I haven't said. Intentionally? Yes.

I guess it's more of the, "added bonus" of further stressing ISP networks to further push the issue.

Like I've been saying, I'm offering no "ought" statements, only "is" statements[0].

[0] http://en.wikipedia.org/wiki/Is%E2%80%93ought_problem

Coldcode notes elsewhere in the thread that to the consumer, the Internet is a requests based system.

Your comment, while well written, sounds suspiciously like the Current Push theory of modeling electrical flow: put a huge motor on a small drive, and you'll blow the drive from all the current the motor forces through it. In reality, that's backwards, and the motor simply stalls. (Full disclosure: I'm not an EE, but man a master electrician I knew thought it was funny.)

Back to the point. Consumers request data, and then expect it to arrive. Netflix offers a service that replies with enormous amounts of data for very little request data. Now, most sites are like this, but video is a caricature of the imbalance typically seen. And Netflix serves a _lot_ of video. So a quick Pareto diagram points right to Netflix for congestion. But none of that is preemptive - all of that data was requested from the Verizon side, and so the word pushing gets you downvoted.

It's just that Netflix's service is less like Newton's equal and opposite force and more like Flubber.

But like all things, there's more to it than just Netflix replying to requests with an asymptotic amount of data. Check out the Level3 blog posts linked elsewhere in this thread, as I think they're a reasonable counterpoint. At least, I thought it was neat.

I agree, it ultimately ends up being a cost issue.

Analogies just start to break down a bit when you consider the fact that these consumers aren't making these requests of Netflix's CDNs on their own. Netflix is giving them an app that does this, so Netflix the company is actually in charge of the volume and direction of these requests from start to finish.

From what I gather from Level3's blog post, they've built out a lot of extra capacity to specifically handle the kinds of traffic that a company like Netflix would be pushing, and that the ISPs have not done so, intentionally, and are trying to get Level3 and others to pay the costs for the upgrades.

Why? Well, again from what I've gathered from the submission and Comcast's article on the same topic, it's because they don't want ALL of their customers paying for upgrades that will only benefit the folks who actually use that increased capacity - specifically, Level3's content. If I don't use YouTube and Netflix, the argument goes, why should I have to pay for the upgrades that'll only benefit Netflix and YouTube? The ISPs are therefore saying that the folks providing the content would be the best ones to pay for these upgrades, because that way the costs are only on the consumers paying for the services (in the form of money or ad views), instead of all of the ISP customers.

Now I think the debate can really take place once we start talking about "who benefits" from these upgrades. The ISPs appear to be saying Netflix, et. al. are the only beneficiaries, and Netflix, et. al. are claiming that since the upgrades are taking place on the ISP side, everyone who is a customer of the ISPs would benefit from the upgrades.

I tend to side on L3's notion that upgrading the interconnects is something with such a large pay-off it's miserly to even consider worrying about it. My understanding of it is that as the pipes get larger demand follows to consume more, and with higher demand comes a greater market to serve. And you can bet many companies salivate at the chance to get a piece of the Netflix pie (including Verizon et al.)!

It always struck me as duplicitous of the ISPs to talk about who holds the cost for the Netflix volume, since most are also attempting to sell in the same market as Netflix (namely streaming video and TV shows). It just smacks of conflict of interest. It's not like Comcast, Verizon, and Time Warner don't all stream _enormous_ amounts of video either, and it's not like I as a consumer care about all the technical nonsense. (I also was terribly underwhelmed by the switch to HD broadcast TV, and generally feel like the signal glitches are so grievously bad and corrupted that surely someone pulled a fast one there.)

Really, I suspect this whole mess may merely be the result of the companies providing the pipes also competing on content, and their roles get muddied. We expect to be able to abstract away all that peering nonsense as a consumer, and since the Internet is so stupefyingly big, creating special cases seems like an unreasonable burden to push on the typical consumer who neither understands not cares how the magic info-pipe works (my personal stance).

Of course the techie part of me has a load of popcorn to see how it turns out. As I noted elsewhere, it kinda doesn't matter who is technically right. My money is on the angry ignorant masses who just want some shows before sleep, and against whomever they blame. So far, my money's on Netflix, as cable companies are somewhat reviled by comparison.

Following your logic, restaurants should be charging their wholesale suppliers for all the ingredients they're 'pushing' into the restaurant's kitchen.

> I really hate to be that guy (and I know I'm breaking rules by saying this), but have we really devolved to the point where we downvote everything that isn't 100% in agreement with the hive mind?

I think people are seeing this as a misinformed viewpoint and reacting to that.

I never made a value judgement - there is no "should" statement in my comment. Like I said in another comment, I've given up trying to express a view on this topic.

This isn't "my" logic, it's fact. It's a fact that customers aren't making informed requests, and it's a fact that Netflix is in control of the requests, from start to finish, and it's a fact that Netflix is increasing the request volume over time by offering more and more higher-quality video (they even offer 4k video streams for some of their content and have said they intend to do more of this over time).

'[I]t's not like customers are doing the requesting "in the raw", or as "informed" consumers. They're just clicking "play" on Netflix video devices.'

I think most people clicking "play" on Netflix devices understand that it entails Netflix sending the content to them. I think most of them probably understand that this is more data than your average webpage, too. Maybe I'm significantly overestimating the intelligence of the average Netflix user, but I don't see how a deliberate request to play a show differs from "a request for the traffic".

My point is that Netflix controls the requests, the whole time. So to say that "customers" are making these requests isn't actually true. Netflix is asking its own servers for the streaming video, which means they can control the volumes at which the data gets sent.

Netflix could ask for less data, in the form of lower quality streams, but have intentionally decided to ask for, at all times, the highest possible video quality possible. In fact, they're so intent on sending as much data as possible, that they've begun rolling out 4k quality videos of more and more of their content.

It's just important to recognize that it's not like Netflix has no control over these request volumes, because it's their software that's making the requests. They're in charge of the entire end-to-end.

It's their software proxying customer requests. It's not like Netflix devices up and decide to watch a movie. It's true that Netflix software is involved at every point, but it's still fundamentally the customer demanding the data. Note that users can opt to stream lower quality than their link supports:

    Adjusting the data usage settings for your account is
    the easiest way to reduce the amount of bandwidth used
    while watching Netflix. There are 4 data usage settings 
    to choose from, each estimate below is per stream:

        Low (0.3 GB per hour)
        Medium (SD: 0.7 GB per hour)
        High (HD: 3 GB per hour, 3D: 4.7 GB per hour, Ultra HD 4K: 7 GB per hour)
        Auto (adjusts automatically to deliver the highest possible quality, based on your current internet connection speed)

From https://help.netflix.com/en/node/87

Now this is where I don't think consumers are as privy to the technology as they could be. I doubt most Netflix customers adjust their data usage settings with their ISPs in mind.

Frankly, I'm not sure they should, but I doubt they're even thinking about that, either way. I'm not going to watch a 480p video because I'm trying to conserve my ISP's capacity. That's silly.

Besides, I think Netflix has a vested interest in continuing to push this issue, and thus the "Auto" adjustment might favor as much data as possible, irrespective of the overall ISP's capacity.

Why should they be adjusting their settings "with their ISPs in mind"? They're paying Verizon for "up to X mbps", Netflix is paying Level 3 for something enormous.

When the amount the user is requesting exceeds X mbps, then certainly Verizon has zero responsibility to meet that. When it's an appreciable fraction, it's understandable that they don't always meet it - it's "best effort", after all. When it's a small fraction, "too much is being requested collectively by all the users over that link" seems like it should quite legitimately be Verizon's problem.

If I had to have an actual opinion on this, I'd say they shouldn't, and the ISPs should just eat the cost of the upgrades, like they do for all their other network upgrades.

But this just goes all the way back to the "up to" and "unlimited" bullshit verbiage ISPs have been using for years to sell their product.

I just don't think this is as much about net neutrality as people are making it out to be. It's more or less, "Who should pay for this?" and the choices are, "Customers of ISPs" or "Customers of CDNs".

I gave you an upvote correction. I know it's frustrating, but sometimes comments get an unfair downvote, and then I think two issues kick in. One, less people see the comment, so you're less likely to get the correction upvote. And two, the priming effect kicks in, and people see your negatively voted comment as inherently negative, and don't give it a correction upvote, or pile on.

I've seen this happen for a while. My only advice is to just roll with the punches.

They can claim whatever they want, but when you tunnel Netflix traffic through a VPN so all the ISP see's is encrypted traffic and bandwidth issues disappear, it removes all mystery.

I've tried several VPN's to test out my theory, and with each one, as soon as the ISP couldn't inspect traffic, all throttling seem to magically vanish.

considering what we're paying for broadband, we should be getting what we pay for without resorting to obfuscating our data.

That's because your VPN providers are no doubt using a different transit provider than Netflix (Level 3) so your traffic is bypassing the congested peering points. The peering disputes are affecting not just Netflix traffic, but traffic between Verizon and any site using Level 3 for transit.

(Not to take the blame away from Verizon. I'd love to see them try to use their logic to explain why other, non-Netflix services work like crap from Verizion.)

Isn't the Netflix-to-Verizon transit something that we already pay Verizon to maintain?

Yes, but Verizon also wants Netflix to pay them for that same transit.

That's what the whole blog article is about. Verizon trying to invent valid reasons for why they want Netflix to pay them.

They are not inventing reasons. Netflix is serving the content, Netflix pays for it to move. Simple as that. The issue is Layer 3 is not buying enough transit from Verizon to match the disparity in the traffic it wants to send beyond what is covered by settlement free peering.

If you look at the diagram on the Verizon post you can see the link is saturated in one direction and only at 30% utilization in the other direction.

This is the DEFINITION of transit. The fact Netflix is even blaming Verizon for this is ridiculous.

That's funny, I thought verizon's customers paid verizon nontrivial sums of money which happen to be many times of what netflix charges on a monthy basis to connect them to the internet, silly me.

> If you look at the diagram on the Verizon post

The diagram is wrong.

The thing they colored red is not red. The wire is fine. The box inside Verizon's network has a network card in it that's oversubscribed. A slice of their box should be red.

Btw, the definition I'm familiar with for "transit" is crossing a network backbone, it's neither serving, nor consuming. The traffic "transits" your network, on its way from somewhere to somewhere.

I have already paid Verizon to deliver the content. That Netflix doesn't request huge amounts of data from me is beside the point.

Netflix could address the balance by streaming back video of people watching their movies. Would that mean Netflix should pay less? Balance of traffic is just a completely crazy metric.

L3 should not have to buy anything from Verizon. The only reason they're paying them at all is that the blackmail worked in the past.

That is not how peering agreements work.

No, it is how peering agreements work. None of you have ever negotiated a peering agreement which is exactly why you don't understand that Verizon is in the right here, L3 is just providing cheap (and inferior) transit and Netflix is just bitching because it bought a Ford and expects a Ferrari.

This article is readable and explains the basics of 'transit' (I think peering is the correct term in this case).

http://blog.level3.com/global-connectivity/observations-inte...

So ... if movak could costlessly move the traffic to a better route, why isn't that the normal way it's working now?

Why can't Netflix just use that route?

It wasn't costless. The VPN providers paid for the bandwidth rnovak consumed.

But don't people get the same results from free ones? And even if rnovak [1] was paying for it, that's still pretty cheap for the traffic to be shifted like that, with Netflix being aware of the costs, so it's still (AFAICT) hard to reconcile with the claim that Verizon isn't deliberately discriminating against Netflix.

[1] sorry, rn -> blurred into m

Free VPN providers are almost definitely absorbing the cost as part of a freemium pricing model, and cheap VPN providers are probably overselling their bandwidth under the assumption that most of their customers won't use much.

The reason why routing Netflix via VPNs currently works so well is because not many people are doing it. If more people did, then 1.) VPN providers would start to charge more, and 2.) the peering points between Verizon and the VPN services' transit providers would become more congested.

rnovak's "proof" of deliberate discrimination against Netflix was that encrypting Netflix traffic caused Netflix to perform better. That's simply not the reason - the reason is that the VPN traffic avoids the congested Level 3/Verizion peering points. Netflix traffic is not the only traffic that's suffering - any traffic coming from Level 3 is. (In fact, this issue has affected me, even though I'm not a Netflix customer, because I regularly access servers which get their transit from Level 3.) It might be that Verizon is playing extra-hard ball in the negotiations because Netflix traffic is involved, but any discrimination is not exclusively against Netflix and it's not because of deliberate packet inspection. Not even Netflix claims it is.

I had this same experience with Comcast.

See also Level 3's blog post calling Verizon and other ISPs out on this BS: http://blog.level3.com/global-connectivity/observations-inte...

Exactly. Verizon can try to spin this as "Netflix is pushing this traffic on us through inferior intermediate links," but what's really happening is that Verizon's customers are requesting the traffic. The standard response when one of their border links is congested is to work with the peer to upgrade it. Level3 is willing to do so. Verizon would rather try to squeeze money out of Netflix for an unnecessary "direct link". Hopefully people see through this PR.

Yep this is exactly the fast lane that we don't want. The lane is not any faster, just less traffic, says Verizon--really, that seems like a fast lane to me.

Since Verizon is eliminating so much traffic, those FIOS links are free for downloading all kinds of content their customers don't want.

Couldn't it be that your VPN provider has better/less crowded connections to both Verizon and Netflix?

I can see how using a lower grade link to a main source of traffic outside a network can actually shape the traffic and reduce the workload on said network.

If that is not Verizon's doing and if there is some reason why Verizon is not connected to other Netflix's links then I can see it as being out of Verizon hands.

It is more probable, though that the limitations in interconnection between Verizon and Netflix are intentional so as to avoid high utilization of Verizon's network.

> I've tried several VPN's to test out my theory, and with each one, as soon as the ISP couldn't inspect traffic, all throttling seem to magically vanish.

That's not inconsistent with Verizon's statement. Verizon is saying, "we won't accept more traffic directly from Netflix (via their transit providers) until we get more money!". By using a VPN, you're avoiding that direct connection, and traversing a less-loaded link.

Exactly. They took Netflix's bait with the "congestion" message. They're demonstrating that their network isn't congested and now we're closer to seeing that they really are attempting to charge multiple customers for the same traffic.

Assuming you connected your Verizon IP to a VPN server that resides in a non-Verizon network, that test is insufficient to deduce that the ISP looks at the data. As Verizon is saying, Netflix does not pay enough for the data that they send. They don't need to inspect the data. By connecting your VPN to a non-Verizon network, Verizon sees the Netflix stream coming from that network (not Netflix).

How does Verizon know if Netflix is paying enough for the data they send? They are not Netflix's ISP.

I want Verizon to reply exactly to this more than I want them to reply to anything Netflix has said.

I tried the same thing a couple of years ago and observed the same, and then switched providers.

If only more had that option.

That's really the key problem here. There is lack of choice in the US and Canada. Everyone loves to drum their chest about capitalism but in this case it isn't working, and ISPs have clearly colluded to divvy up the territories so they aren't directly competing.

The internet has become a public utility just like electricity, water, or the roads. Businesses and private citizens depend on it day to day, and economic growth is hampered by poor performance in this area (e.g. new types of businesses that depend on bandwidth, like a Netflix competitor, cannot exist if the pipes are slow).

In my view the raw cable and fibre should be confiscated from the big three by the government using eminent domain, and then rented out to competing ISPs (including the big three) on even footing. The rent would pay to maintain the cables/fibre and deploy more network, and the ISPs could add a premium before reselling it to end users to cover support costs and maintaining equipment (e.g. routers, edge connections, etc).

Basically make the final mile public property. It would make real competition as smaller ISPs can now compete with large ones, and there isn't regularly problems for these companies (e.g. getting permits to place new cables).

>Everyone loves to drum their chest about capitalism but in this case it isn't working

Your comment assumes that capitalism is in place in the US. It is not. Cronyism, better known as fascism, is.

Your comment assumes that capitalism does not have an inevitable degenerative end state of cronyism and other ills.

The balance of history suggests strongly that it does.

It sounds like you don't know what capitalism is. Here's the definition Google spit out:

"an economic and political system in which a country's trade and industry are controlled by private owners for profit, rather than by the state."

Seeing as cronyism necessarily requires state influence and capitalism requires the lack of state influence, your conclusion falls on its face.

Capitalism as ideally defined is a period an economy often passes through extremely briefly; one of the exit paths is consolidation of enough power/wealth/influence in a handful of individuals or corporations to begin effectively purchasing government actions as desired.

Unless you would like to argue that no true Scotsman would ever abuse a position to make a buck...

Capitalism is the free and willing exchange of goods.

Because the state has colluded with certain companies to remove the 'free' from the above statement, the US internet is no longer a capitalistic system.

Your statement about all capitalism eventual collapsing to fascism deserves more evidence. If you could truly prove your point, you'd win a nobel prize.

>one of the exit paths is consolidation of enough power/wealth/influence in a handful of individuals or corporations to begin effectively purchasing government actions as desired.

The prerequisite to purchasing government actions is the government having the power/authority to take action. There's a reason lobbyists do not approach you or me.

The prerequisite to purchasing government actions instead of purchasing mercenaries to enforce ones' will directly is the government having the power/authority to take action.

Getting rid of government doesn't change the ability for the powerful to use money to apply coercive force against the powerless. You know that, right? It just means they have to do it themselves. Would you rather $EVIL_CORP own a Senator or own weapons of mass destruction?

The definition of government is a monopoly on violence/force. Furthermore, no monopoly can be sustained without the force of government.

Absent the government I'm not even certain there would be weapons of mass destruction. It is the state who demands these weapons to maintain its monopoly on force.

A business, on the other hand, is interested in making a profit. Absent government coercion, a business must provide a worthwhile good/service to make a profit, all while competing in the marketplace.

Anything an "$EVIL_CORP" might do pales in comparison to actions we know governments have done. One must only tally up the hundreds of millions of lives lost during the 20th century in senseless wars, democide, etc.

The US will implement the very excellent system in which you detailed right after we get single payer health insurance. Never. =/

In SF, WebPass (and MonkeyBrains, from what I hear) are great alternative ISPs. Both do point-to-point wireless. My building shares a symmetric 100Mbps connection to WebPass, and I pretty reliably see 30-40Mbps on my wired network. And I pay significantly less than I had previously paid Comcast. (However, I never used the TV component of my Comcast offer, so the cost structure is presumably different for people who do want cable TV.)

I understand that point-to-point wireless is more realistic in an urban environment than in the suburbs or a more rural setting. Do other cities have good alternate ISPs, or are SF residents somewhat unique in their appetite to try alternate providers?

Point-to-Point (Fixed) Wireless is actually also an attractive solution in rural / semi-rural settings as well.

If the distances are moderate (with line-of-sight) but demand is too sparse to warrant the cost of running coaxial / fiber (and distances are too far for DSL), a common alternative on the high end is fixed microwave, and on the low end Verizon LTE.

A good example is the Ames, Iowa area, where an ISP has set up a tower with good line of sight, and a bunch of directional antennas both on the tower and at customers locations. This setup provides great service out to ~15 miles or so, enough to cover the outskirts of this small town.

Some issues with this setup: storms cause 'rain fade' which blocks the signal, and the wireless spectrum does not scale to too many users. Also, there's a finicky antenna on the roof that can become misaligned in storms.

My building in SF is locked in to ATT Uverse. I'd kill for WP or MB.

Assuming your new provider did not have the problem, which ISP did you switch to?

A well written PR piece, but no new insights here. Verizon feels it is not Verizon's job as an Internet service provider to provide adequate service to parts of the Internet that its customers are paying them to access.

This wouldn't be a problem if broadband access were actually a competitive market - customers would simply switch to a provider that had taken the initiative to give its customers what they want.

Thank you. While almost all the comments are saying the same thing, I think you stated it best.

"Netflix sends out an unprecedented amount of traffic" is one of several misleading locutions in this piece. Verizon customers request these packets.

The ISPs' problem is that they aren't willing to change their pitch to customers. As long as their product is "internet service at ~x gbps," they are on the wrong side of this issue. So they go ransoming the content folks.

The funny thing is that the dream world of the big modern ISPs would appear to be something like AOL circa 1994, where you can have "the internet," but it's a cheap lo-fi interconnect [1], and what they mainly want you to consume is a bunch of low-quality in-house content channels. [2]

[1] http://images.appleinsider.com/leopard-preview-ichat-3.jpg

[2] http://www.jeffwright.com/portfolio/portfolio_aol_main.jpg

We need ISPs and the general public to demand that first statement be delivered as

"Our customers request an unprecedented amount of data and traffic from Netflix."

This is Verizon's problem, not Netflix's.

ADD: Netflix doesn't just randomly send data and traffic onto Verizon's network. That data is requested by Verizon ('s customers).

It's always kind of strange to me that Verizon only wants to provide settlement-free peering "where the relative traffic flows between an IP network provider and us remain roughly equal".

If you were to take that at face value, it means that Verizon would happy to provide free peering if only Netflix modified its client so it uploaded a number of random bytes equal to the size of the stream it was downloading.

But that's nonsensical, right?

No, that's fucking GENIUS.

Please God, if there is a God, let someone from Netflix read this parent post and implement such a thing to take the wind right out of Verizon's sails.

That is actually the basis of our startup is that we can solve the congestion problem by swarming with other video viewers to require less downstream bandwidth usage

That's interesting tech, but I kind of feel like it's just postponing dealing with the problem.

The problem is anticompetitive behaviour from ISPs, and while this would make video streaming better, it would just allow them to be more abusive, while we try to fill the resulting gap with more technology. IMO we should be leveraging the Netflix situation to fix that behaviour.

Regardless, swarm-sourced streaming is something I've been fantasizing about for awhile and I'd love to see your source if it's open. You should definitely proceed with it regardless.

It's a holdover from the old days, when one side of a peering agreement very logically didn't want to carry all the packets of another provider while the other provider doesn't do much to help you in return.

It would make sense in a completely p2p world - at the end, I receive and transmit roughly equal amounts of data - and I am billed accordingly on my end only for the traffic I send, despite generating some cost for the receiving end.

This is not the case with ISPs - you get billed for everything. In the old phone world where this started, it'd be like you having to pay to receive calls as well as making them.

But the connections consumers have are non-symmetrical. Almost none are able to upload at HD-quality speeds.

It doesn't need to be random. Make it torrent Ubuntu CDs or something. Then you can declare Netflix is helping achieve a public good in addition to calling Verizon's bluff.

Trust me, Verizon wants nothing of this sort. They would literally throw a fit if suddenly Netflix customers were pushing out as much bandwidth as they pull in.

Netflix has an open offer to move CDN servers deep into provider networks [1] to avoid peering congestion. This would put most of the Netflix traffic at the bottom right of Verizon's infographic. Why hasn't Verizon accepted this offer?

[1] https://www.netflix.com/openconnect

Not sure why this isn't ranked higher. The entire issue with Verizon is that it refused to do this. They can't complain about intermediate systems when they have been provided the chance to bypass many of them.

Your comment is the most pragmatic thing I have read.

The whole thing of ISP's complaining about Netflix to me is wrong, they are complaining about a peering agreement with a transit provider and just using Netflix as a name people will recognise.

The truth is there are technical solutions to these problems but politics is taking over the issue.

That chart is worthless without numbers.

First off, the chart makes it sound like Netflix is preferring subpar transit providers, and that by simply switching to different transit providers, they'd be fine. But there's no numbers on total bandwidth. For example, if Verizon has 100Gbps to Netflix's content providers, and only 40Gbps to their other content providers, then the additional 20Gbps is going to be a drop in the bucket.

It could be the reverse: Netflix is choosing the 40Gbps provider where they could be choosing the 100Gbps, in which case they could easily triple the available bandwidth they have to send to Verizon.

But, we all know that's not the real issue.

Finally, if peak utilization means that netflix is saturating the router with reduced bitrate video, then increasing that could very well lead to saturation in the internal Verizon network.

EDIT:

I ran some numbers. If Netflix's providers are saturating at 100%, and other providers are only at 44% peak, and then peak incoming traffic is exiting the Verizon router at 56% peak, that should roughly mean that Netflix has roughly 3.6x the amount of bandwidth from "other transit providers" than it does from Netflix's transit providers, which, in turn, means that Verizon has roughly 2x the total bandwidth that Netflix uses available on it's network. What isn't mentioned is what percentage Netflix uses of it's current providers, how those "other providers" are split up, the maximum theoretical utilization, etc...

What isn't mentioned is how that's split up. It's probable Netflix can't use on most of those links even if they wanted to, and there's no word on how many links there are. If it was one giant link, and Netflix could use on it, sure, Netflix should probably try to do that.

Bottom line: I don't think there's anything technically limiting Verizon's capacity to Netflix's transit providers, as according to their numbers, Netflix's transit providers only constitute ~22% of the total bandwidth into Verizon. Down the line, it's probable that, even if these link issues were fixed at the border router, they would pop back up again in the Metro Area router or aggregation router, and really gum up Verizon's network.

Great analysis, thank you.

Adding to your point about gumming up the metro area routers inside Verizon that may be quite possible. Remember this is Verizon's analysis for a single customer. What are the odds that Verizon, to influence public policy, choose a customer that would best represent Verizon's case. For example, a customer in a region that was just provisioned a bunch of extra capacity. So, it's possible that the percentages at the customer/aggregation/metro area are not typical for Verizon, but rather Verizon's best case scenario.

The summary of the article by Verizon is:

> Our network is fine. Netflix is buffering because we won't allow the traffic Verizon customers request onto the Verizon network until Netflix pays us more money.

Verizon's customers request an unprecedented amount of data and traffic from Netflix.

This is Verizon's problem, not Netflix's.

Verizon should be working with Netflix on an amicable solution to perhaps cache data inside Verizon's network... O What do you know! Netflix already has a program for that: https://www.netflix.com/openconnect

To be fair, this problem is non-existant today, because Netflix has entered into payed peering agreements with Verison/Comcast/other ISPs.

They more or less acquiesced.

They're just not happy about it.

ITT: People that don't understand that settlement free peering needs to be mutually beneficial.

When you pay Verizon you pay for access to their network. Not access to Netflix's network. Verizon will has transit agreements with some people to use it's network and settlement free peering with other ISPs of similar size.

When I go to by transit from Verizon to host my website do I expect to get settlement free peering? Well, probably not because I am probably serving much more content than I am consuming, as such my link is unbalanced.

If instead I was another ISP and I was sending roughly as much data to Verizon as I am consuming it suddenly becomes mutually beneficial for us to agree to send traffic for free as then we don't each have to pay transit costs for this traffic.

Not understanding this is why HN and other tech news outlets are in outrage. This is just how peering arrangements work. Unless your traffic is balanced you need to pay the cost of transit, simple as that.

Now, the reason Netflix is bitching is they -are- paying for transit.. they just aren't paying enough to the right people. Their problem is that their transit providers don't have sufficiently large links to Verizons network.

Because of the massive asymetical nature of having Netflix on your network Verizon is not agreeing to settlement free peering to Layer 3 et al. Which is completely fair and how things work, unless it's mutually beneficial you have to pay up.

Now, why does a witch hunt against those ISPs make no sense? Well simply put they are not running their network at 1:1 contention and probably charge prices that reflect that. If you went and bought carrier grade transit from more respectable providers chances are you would be getting 1:1 contention and these issues would never occur, but you probably pay more than 10x what Netflix is charging per mbit.

TLDR: Netflix doesn't understand how telecom works and needs to pay for better transit.

You're missing a point. Netflix _pays_ for their bandwidth. Namely, their network provider. (Or more likely, their peering points).

It's not like somebody showed up at their office and said "Here's a giant pipe, push all you want".

And "they're not paying to the right people" is an incorrect argument. You pay _your_ connection point for money, and the recipient pays theirs. Intermediate connections need to be handled by the respective networks, not the endpoints.

So if Verizon wants money, they need to extract it from their peers, not Netflix. (Or the user). Which they don't do, because, well, turns out they make enough money of it that losing a peering agreement is actually detrimental to their business.

It's completely within VZWs rights to expect money from Layer3 for peering. It's not within their rights to extract it from somebody three hops down. That is the _reason_ we have peering agreements, billing individual customers doesn't make sense.

As far as I know, there has been no claim that Level3 pays Verizon for their peering. The Verizon-Level3 peering is settlement free as both are large Tier 1 networks. What verizon is refusing to do is upgrade the size of the interconnections until Level3 either pays them for paid peering or gets their traffic ratios back to an equal ratio.

Level3 as a Tier 1 who pays no one for peering or transit, and is of course not going to do that. That leaves their transit client Netflix with the choice to either use other providers, who will likely run into the same traffic ratio issue, or pay for peering directly with Verizon.

When Netflix is pushing out 1/3rds of the internets whole traffic, and it's all going in one direction, there's practically no tier 1 network out there that can provide transit for that without breaking their peering agreements, except perhaps the same content providers (Verizon, ATT) offering paid peering to netflix. It's not unreasonable with their traffic levels to expect netflix to peer directly with the largest consumer networks.

> Not understanding this is why HN and other tech news outlets are in outrage. This is just how peering arrangements work. Unless your traffic is balanced you need to pay the cost of transit, simple as that. ... > Now, the reason Netflix is bitching is they -are- paying for transit.. they just aren't paying enough to the right people. Their problem is that their transit providers don't have sufficiently large links to Verizons network.

It is mainly because of 2 things:

1) If you pay for X internet access, you expect to get X internet access. You don't expect to get a restricted version of what Verizon puts out in its marketing. It isn't reasonable to expect a lay person to see it differently.

2) https://www.netflix.com/openconnect

Netflix is willing to let them put their hardware inside their network. Verzion, et al. refuse unless they receive "special compensation" above and beyond what is normal [normal being defined as Netflix's relationship with companies like Frontier, British Telecom, TDC, Clearwire, GVT, Telus, Bell Canada, Virgin, Cablevision, Google Fiber, Telmex].

Verizon, et al. can complain all they'd like. However, they want special treatment above and beyond what a small ISP like Google gets. Their argument is "because Verizon has a monopoly on its customers internet access". :/

Free peering is mutually beneficial: Verizon is receiving payment from customers who want to access Netflix at reasonable speeds. Verizon serves consumers, who are by definition asymmetric users.

"When you pay Verizon you pay for access to their network. Not access to Netflix's network."

I don't see it that way, not do I think, the vast majority of US consumers see it that way either.

If you go to the landing page for verizon's high speed internet service, you'll see nothing that says "You are paying for verizon network service, not internet access. The do not call the server "high speed verizon network service", they call it internet service because they know customers do not want access to verizon's network, but the internet.

Also, verizon knows many of their customers want to watch netflix, and they certainly would rather watch a netflix video with 15 seconds of buffering for every minute of actual video (which, funny enough, is probably still less wasted time than in ad supported channels as such is supplied by verizon's tv service). Thus, but fulfilling their promise to connect their customers to the internet via peering relationships to the parts of the internet that their customers demand, they are creating happy customers, how is that not mutually beneficial?

Lastly, you would have to be devoid of all pessimism to not believe this isn't about money, but control. If you look at the money verizon is charging for these peering arrangements, is quite trivial to both netflix and verizon. But it just so happens that netflix is competing with a core product of verizon. Once the peering arrangements go in, verizon has a platform to begin to influence netflix and their ilk.

I get where you're coming from, but I still don't agree with you.

If that's the case then it would be "helpful" for Netflix to configure their players to send garbage data back to Netflix as fast as possible while playing videos just to get closer to that 1:1 peering that you talk about. We both know this would benefit exactly no one, not the ISP, Netflix, or the consumer.

"When you pay Verizon you pay for access to their network" When I pay my ISP I pay for access to the Internet, not to some other net. The customer doesn't have to be preoccupied about any backroom operations necessary to provide the service contracted. OTOH, when I pay Netflix I expect it to be able to provide the content, including having a link with enough capacity.

What Netflix is doing is "bullying" a bigger company into a less favorable deal in a negotiation table. Verizon now has to deal with customer pressure to make a deal sacrificing their profit/incurring in loss. That's the way negotiations are made, you have to know who are the stakeholders, how can they affect the outcome and how can you act on them. If neither side lied then everything is ethically sound.

The outcome will depend on whether Netflix can endure a competitor who offers a better deal to Verizon and wether Verizon will bulge under customer pressure. From where I stand, Verizon is in a better position because its monopoly is stronger.

Except that I want to pay someone to deliver the traffic I request from third parties (up to a specific rate that I pay for), and just make that work.

Not for the privilege to have traffic delivered to me if the sender pays enough money to the government mandated monopoly that owns my area.

The value that Verizon gets for accepting that traffic is that I continue to pay them money; no need for the person fulfilling my request to pay Verizon for Verizon to accept things on my behalf (which is what they agreed to do). Verizon (and other ISPs) bill their service as accepting any package on my behalf, and tell me that I can have a certain rate of packages accepted and routed to me.

They're simply not upgrading their package receiving centers to reasonably reflect what their customers are ordering because they know that their customers can't do anything about it, and they don't have to provide the service they promised.

Netflix could fairly easily make the traffic symmetrical (send nonsense in the other direction), but that wouldn't be beneficial for anyone.

It would be more interesting if Netflix were able to start doing caching on the client boxes that sit there mostly idle and have a bit-torrent like protocol for distributing it.

If you'd like a different perspective regarding the quality of your broadband connection and the reasons for congestion this is a good read that basically says ATT at least does not provide enough connectivity from the Level 3's optical backbone to their regional networks. I suspect Verizon is probably the same.

http://blog.level3.com/global-connectivity/chicken-game-play...

Here's a letter Level3 wrote to ATT on the subject [pdf]. http://www.level3.com/~/media/Assets/legal/cicconi.pdf

It's so strange, I live in Bolivia and Netflix never buffers for me at all. I have a laughable 200kbps real download speed (1768kbps plan) and I only wait for Netflix to load for about 2 seconds on my PS4 and off I go watching on 480SD quality.

I would expect the US to be much much better.

you actually watch below 720p?

i'd switch back to downloads for HD before watching something with that resolution.

Honestly though, I can't really tell the huge difference from my couch. I guess my TV isn't big enough to actually notice.

Sure my blurays play at higher framerates and look crisp, but I don't mind watching 480 at all.

I sympathize with you, but realize that there are a lot of video geeks here who are baffled some of us plebes still have SD televisions.

No one in the US wants to watch 480 video.

I do. Comcast. Data cap.

Of course, what I would -prefer- is to ditch Comcast, and yes, watch at a higher resolution, and I will at the first available opportunity. But can't. Because monopoly.

nonsense. there is no monopoly in capitalism.

/sarcasm

If there was Capitalism you'd actually be able to compete.

There is Capitalism ... but it's been corrupted. Economic systems are morally neutral. How people employ them determines the morality. The U.S. is, in general, pushing Capitalism to its logical end, in which the system consumes itself in the name of its constituents' "best interests."

I only buy HD on iTunes, but I only watch SD for 99% of my TV shows. Movies it depends. If it's a kid's movie we've seen already: SD. Also if it's the latest action rental: SD. I rarely rent in HD unless cinematography plays a large role in the film.

480P looks surprisingly good on iTunes. Much better than DVD typically. I'm OK with that for the most part.

OTOH Netflix's auto-scaling is clearly the future.

Or can.

My 5mbps AT&T DSL struggles to load Hulu and Netflix on a daily basis, even at just 480p

There are several situations where I would be perfectly fine with 480 if it meant I didn't have to deal with loading issues. Watching a cartoon in a lower resolution doesn't detract much from the experience (IMHO).

On my phone I do.

The argument I see most commonly from ISPs (and seemingly re-stated here) is basically, "Why should all of our customers pay for the upgrades we'd have to implement if only the Netflix subscribers would benefit from such upgrades?"

The idea being that, no matter what they do, the cost of it will just be passed along to the consumer. So their solution is to charge Netflix for the upgrades. That way, when Netflix passes those costs on to the consumers, only the ones who actually have Netflix will be paying for the upgrades, as opposed to all Verizon customers.

It makes sense, once you assume Verizon/ISPs and Netflix will just raise their rates to pay off these upgrades/charges.

But again, you have to take the assumption that, regardless, the costs will be passed onto the consumers, for granted. If you think the ISPs will try and eat the costs, then this argument doesn't fly. But they're saying they won't, so... they probably won't.

I'm not trying to pass moral judgement on this (I gave up trying to convince you guys back when every comment I wrote about this was downvoted to hell and back), just saying what the argument the ISPs are giving actually is. It's just not as "left field" as you might think.

If Verizon feels that Netflix should pass the costs on to the consumers, how would they go about that? Much like Verizon can't know if a user will be using Netflix, Netflix can't know what networks a specific user will come in from, so they'd have to spread the costs over all subscribers, just like Verizon.

That's a fair point, but Verizon is at least claiming apparently that it's Netflix's problem to solve, not theirs.

Maybe varied subscription prices based on ISP. Say, a "base" $5/month you pay to Netflix, and then you can subscribe to "performance" plans that cost an extra $1/$2 per month and get you access to the peering agreements for those ISPs.

Though I'm pretty sure that'd start riots. But at least your ISPs wouldn't be offering "fast" lanes! Your products would do that!

How apropos that the link is showing unavailable after timeout & errors after only 17 minutes of being posted.

<Verizon> It's definitely not us. Yep. Not it.

LOL taking 30 seconds+ to even load their page

They must be using Netflix's network.

The first time the page failed to load, the second time it took 52.76 seconds according to the Chrome Developer Tools. I'm on a university network that is super fast.

Doesn't even load for me. http://i.imgur.com/wT7zV3f.png

Yeah the site/page has been down/slow. Getting ISE 500 errors.

Long story short, Level 3 won't pay us for peering agreements so it's their fault.

The most incredible part is that the facts and anecdotes presented so far don't agree with Verizon's tale, so instead of Verizon decided to respond with a fluffy PR piece instead of fixing the problem or countering with facts. Verizon seems completely unwilling to take any blame in this and does not seem to have their customer's best interest at heart.

There are no new facts in this article other than a useless infographic and repeated claims that Netflix is saturating their link with content Verizon customers are asking for.

The only new fact I see here is that link between L3 and Verizon is not symmetric. That's not surprising really, but it means that there is already traffic asymmetry baked into whatever peering agreement they have today.

It seems like Netflix is the one providing value to the Verizon network so it should be the other way around. It would be like someone saying "I'm not downloading these files from a webserver because they're not paying me to download it."

> We’re Not Satisfied Until Our Customers Are

he misspelled "Shareholders"

can we all reply to his tweet[1] about this letting him know we can all see through his bullshit

[1] https://twitter.com/daedyo/status/487292992504360960

Nobody's getting responses from either him or Libby Jacobson. They just farted in the room and left.

I'm genuinely curious if the author `@daedyo` [twitter] actually believes what he has written?

This blog post by level3 was fairly enlightening, and in stark contrast to his view/version of this.

http://blog.level3.com/global-connectivity/observations-inte...

As an side, living in a country where ISPs (mostly) doesn't have monopoly, it's funny that these congestion issues never occur here.

Seems like another example where lobbying and money is the underlying issue. https://mayday.us/

The real problem is the ownership of the physical lines that interconnect everything. The high cost and the limited right of way access prevents duplication of physical networks leaving one company in ownership thus control and this is a monopoly. This is why most North American cities (Canada and the United States) have one cable company and one phone company in each region. Services from competitors and small companies just access, use and resell from the single company in the region. In BC for Example you have Shaw and Telus. Bell, Videotron, Rogers rents access over the Telus Network in most cases.

What is really needed is to separate the physical network form the service provider. We need a Provincially and State owned network or county or city owned networks with one mandate only. Provide a physical network and charge a price to maintain and grow it. owners of homes and businesses and other buildings would pay for the last mile connection to the network and own that last mile connection. Companies like Bell, Rogers, and dozens upon dozens of small companies offering service be it phone or internet or TV can plug hardware into the network and provide such services. Both the consumer and the ISP pay a network access fee that pays for the network. But this puts every one on equal footing because its the same fee. the physical network can't throttle or control or manipulate any of the content. Would be easy for small companies to expand by adding hardware into data center locations all over the network. Would allow consumers to buy and plug in equipment to allow for point to point VPN connections between family members at a fair price.

It is the only way to solve this issue. I would suggest the same be done with the Cell Phone networks. The Cities and Provinces and States own the towers using all the frequencies and they charge a access fee to the subscriber and phone provider to manage and expand the network. Any company could start up a cell service and compete head to head. As long as the core infrastructure is owned by one entity that has its own motives on how to use it, it will always be a monopoly situation and thats bad for business and consumers and innovation. Take the expensive part out of the puzzle and make it a public owned resource let the ISP and companies fight for customers on equal footing.

This proves the opposite of what they argue. Supposedly the argument is "Netflix users use too much bandwidth so Netflix should pay for our network upgrades". And yet what their chart and argumentation shows is that the Verizon network has all the capacity it needs to handle this user's traffic and yet at the border of the Verizon network, Verizon is not accepting more traffic from Netflix.

If the non-trivial-much-highler-than-netflix monthly cost of verizon's internet service does not cover the cost of verizon connecting their customers the internet, what the hell does it cover?

I also don't understand why they are talking about "Review Shows No Congestion on Verizon Network", people are paying them for internet service, not verizon network service.

I know who I believe, and it's not Verizon!!!

Can anyone recommend networking resources analogous to Learn C the Hard Way or Dive into Python?

High Performance Browser Networking by Ilya Grigorik is pretty solid, and free online. http://chimera.labs.oreilly.com/books/1230000000545

Beej's Guide to Network Programming is still the best.

http://beej.us/guide/bgnet/

Seems the issue (according to Verizon) is that Netflix is using "connections specifically to be used only for balanced traffic flows.". Can someone comment on this and the difference between the two (if any)?

You can always spot a one sided blog post because the ability to leave comments is always turned off. I bet they spent more time building the inforgraphic than they did addressing the customers issue.

Oh, I don't know about that (although I agree with everyone that Verizon is in the wrong here). They could be announcing free puppies for everyone and the first three comments would be Some Dude Complaining About Obama, a paranoid schizophrenic going on about I don't know what, and somebody claiming that their aunt made $85/hour working from home.

Perhaps Netflix would be wise to figure out caching, and let subscribers fully download movies at the highest possible quality with zero buffering. That, or start serving movies with bittorrent.

Why is the "transit providers" split into two segments - inbound and outbound? Wouldn't it be the same logical segment and be impacted based on data in both directions?

> Why is the "transit providers" split into two segments - inbound and outbound?

Because they are trying to illustrate how unbalanced the traffic flow is in each direction, to bolster their argument that Verizon shouldn't have to deliver the streams that Verizon customers requested until Netflix gives them more money.

They seem to forget that the internet is a request based system. It's not a series of tubes going downhill.

The page isn't even loading at all. Just the "his webpage is not available" message. WTG, Verizon.

Why isn't a large ISP like Verizon directly peered with Netflix? Why waste money and bandwidth on transit?

Can't even load the site, the irony !

Ironically, I can't load the website.

Somewhat ironically, the page is taking forever to load.

500 Internal Server Error for me right now

Let the verbal battle begin!

Reply on news.ycombinator.com