Verizon's accidental mea culpa
blog.level3.com
blog.level3.com
That's what I would call a mortal wound to Verizon's arguments.
Verizon's argument is clear:
> Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows
Level3 hasn't refuted this point (the imbalance of the traffic flow). If the traffic is not balanced, the contract between Verizon and level3 probably has clauses requiring Level3 to pay Verizon for the imbalance (in this case, since Level3 is pushing more data, they would be the ones paying). That is the real problem, most likely, and the technical argument merely serves to confuse
No. That's not how the internet works. Nobody PUSHES data. People PULL data.
Verizon has customers who pay for internet access. Those customers make REQUESTS to Netflix for data and Netflix RESPONDS with data.
They happen to be streaming movies which use a lot of bandwidth, but on average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or 50/5 or 100/10 that Verizon advertises for purchase.
If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.
But this data isn't unrequested. Verizon's customers have requested it from Netflix as they are within their rights to do since they have literally paid for it.
This is Verizon wanting to bill their customers once and then their customers' vendors as well. Double billing for a single service is a neat trick if you can pull it off. But it tends not to engender goodwill.
* This would increase throughput and capacity for both Level3 and the consumer networks.
2) Netflix has offered free Open Connect Appliances to all major ISP's.[2]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all their customers. In-network traffic is almost free for the ISP's.
3) Netflix has offered to change their entire distribution model into a P2P model.[3]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all customers. In-network traffic is almost free for ISP's.
.
All of these options have been flat-out refused. The Big5 ISP's are purely after the money. There is no other compromise for them.
[1] http://blog.level3.com/global-connectivity/chicken-game-play...
[2] https://www.netflix.com/openconnect/hardware
[3] http://arstechnica.com/information-technology/2014/04/netfli...
I work at an ISP, can confirm. All decisions about capacity are about how much money we'll make or lose. We're intentionally keeping bandwidth constrained because we know it will severely impact our cable TV business.
Net neutrality simply means they are not allowed to go into homes and figure out what you're using your electricity for, and try to charge you more based on that usage.
It gives them pops, "free" capacity (meaning they have to buy dark fiber, and can resell internally cheaply, or just wouldn't have it at all if they didn't have phone business), shared infrastructure.
So if you do this, effectively it will mean the same any other solution means : more expensive internet.
Given that that will be the result, might I suggest an alternative solution ? Get a basic business internet line from Verizon. See the difference.
like what exactly ?
All of Netflix, in HD, on 24TB.
Netflix could solve the bandwidth problem by just mailing the whole library to customers. "Never underestimate the bandwidth of a station wagon full of magnetic tapes."
They will put a server in a data center anywhere an ISP wants, and they'll manage the whole syncing thing so the ISP doesn't even know it's there. After that, all content comes from that cache.
My personal opinion is that you should cache/co-locate your data close the the edge of the edge of your network whenever it presents a benefit for congestion and price.
http://www.listbox.com/member/archive/247/2014/07/search/bmV...
Its the senders responsibility to deliver data to the local ISP of the recipient. It doesn't matter if the person asked for the data, you still have to pay to get it there.
The issue is that Verizon is both a national transit provider and a local ISP. If you just dump your data on the nearest Verizon peering port, you aren't fulfilling your end of the bargain. You are asking Verizon to do it for you, as a transit provider.
However, if you are delivering it to the last mile, you have "paid for it" it already.
The issue is where are these congestion issues occurring. At the last mile or at the edges of Verizon's transit network.
Verizon absolutely has the right to bill for providing transit.
Even if we agree that Verizon has the right to bill Level 3 for transit within _Verizon's own network_, it seems odd that they are not taking advantage of Netflix's CDN boxes, which would drastically reduce the amount of transit required.
If you just dump your data on the nearest Verizon peering port, you aren't fulfilling your end of the bargain.
So, let's say I build a new skyscraper with 10,000 apartments, and I set myself up as the exclusive ISP to the tenants of that skyscraper. I wire up the building nicely, and have some big switches and routers in the basement. Does every content provider (e.g. Netflix) or company working on their behalf (e.g. Level 3) now have an obligation to build new circuits, and deliver the traffic all the way to my basement?
For years, this is how Verizon/AT&T/SBC/BellSouth/etc had structured telephone networks. And this is probably the same structure they want to extend to shared data networks aka the Internet.
If you were a competitive telco carrier who wanted to deliver a massive amount of phone calls to Verizon customers in a particular region, you couldn't just dump it off at the tandem (which you can think of like a telecom peering point), you had to extend your network to the end offices where the Verizon customers were aggregated.
The thought there being there's finite capacity between the end office and the tandem. And if you're going to use most of it, you should either pay for it or build your own facilities to alleviate congestion.
You're dealing with a traditional telecom company (Verizon) in its telecom mindset (build to me/mid-span meet, and keep a very tight watch on ratios/meet points via accounting, billing, and state regulation), vs. a traditional Internet company (Level 3) in its traditional peering mindset (build to the exchange, then build to me, and keep a loose observation on ratios/meet points with no regulation).
I hope I haven't gone too far off track.
The state won't let you take on calls as a local access provider if they're going to end up going nowhere.
You also have a regulated obligation to provide working 911 service so you need to interconnect with the incumbent's 911 tandem switches. So, off the bat, you have multiple connections with or to Verizon.
Now, if there are some really popular people in your building who attract a certain amount of calls, YES, you can demand that the other telecom carriers extend facilities to your switch (which is located in your basement).
Those carriers will decide if they want to build facilities to you, or work with another carrier (remember the one interconnection agreement you had to sign?), to pay them to deliver calls to you on the agreement that they have enough facilities with you.
You are compensated, per minute for any call that terminates on your skyscraper network through a process called reciprocal compensation, whose rates are laid out in the interconnection agreement you signed.
Is old-timey Verizon's attitude starting to make sense?
How quickly would this change Verizon's policy?
As a customer of an ISP, I pay for access to servers on the internet. My ISP charges me for a level of bandwidth. If they are the bottleneck slowing down my connection to the services I want to access, they should fix it, and give me what I'm paying them for.
For example, you want to download something off a Server in Germany. Your ISP is responsible for getting your initial message to the server. But it's not responsible for taking that information in Germany and carrying to back to the US for you. The Server's ISP has to carry it to the USA and then dump it off at your local ISP.
Now your ISP and the Server ISP may have a peering point in Germany, but they might not. But either way, your ISP isn't under any obligation to pay for transit of data being sent to you. That's just not how it works. And that definitely isn't what is in your agreement with your ISP.
Is that accurate? Or no?
In this case Netflix is being forced to pay for transit from their "house" all the way to mine. Not how the internet normally works.
AWS may be the origin for content distributed via peering, appliances, and CDN partners but your videos aren't streaming from AWS: https://www.netflix.com/openconnect
you can't put your own hardware at AWS.
they are the poster child for using AWS for what it is good for, general purpose compute (storing your play list, the list of all movies, controlling the CDN etc)
but I imagine for streaming you need bare metal
The agreement between Level3 and Verizon probably has some balanced traffic requirement, with penalties for asymmetry. If that is the case, then the nature of the imbalance would force Level3 to make payments to Verizon.
Verizon gave a very clear suggestion for Netflix: work with other providers as well. If Netflix worked with the overwhelming majority of third party providers, then you could argue that Verizon is not playing fair. But I don't see any commentary that Netflix is actually working with most of the third party providers.
So does that mean there's a downloader-pays rule?
If so that means that Verizon should be paying Level3 because it consumes 3x the bandwidth that it provides.
Verizon's needs Level3 3x as badly as Level3 needs Verizon on a bit-by-bit basis. So why isn't Verizon paying Level3 for the imbalance?
There are actually two sides to the ISP game. There's the consumer ISP side and there's the commercial ISP side.
Consumer ISPs are in business to get their customers hooked up to as much of the internet as possible to enable them to consume things. And they charge their customers (normal folk like us) for this service.
Commercial ISPs are in business to get their customers hooked up to as much of the internet as possible to provide services. And they charge their customers (big companies) for it.
When commerial ISPs meet consumer ISPs typically they say "let's trade traffic for free so that way we can both keep our money and not worry about accounting"
What's happened is that a consumer ISP is trying to charge more than just it's last-mile normal customers for data that they use and which it is ostensibly contractually obliged to provide to them without further fees.
What Verizon is saying is "look they need to pay us to upgrade this link" which on the surface doesn't sound too bad. It kinda makes sense. If I used more bandwidth they would charge me more. But the thing is that I am already paying them to make sure that the connection to the internet is uncongested. They are deliberately allowing it to get congested to try and bill both sides of the transit.
They may have a peer balancing agreement, which is not being held up, but they're also still being payed by their customers to provide access to content outside of their own network and Verizon is enforcing a limitation from another agreement (the peering agreement) that is affecting their ability to provide the content their customers want to have access to.
You can't have it both ways. Or... I guess you can, as Verizon has shown.
This obviously gives the Consumer ISPs pricing power in all of their business dealings (including any peering contracts that they force the Commercial ISPs to sign). The fact that Verizon owns RedBox (a Netflix competitor) & sells TV packages is just further motivation for hampering the interconnection.
On the commercial side they provision a link from point A to point B (which is what his software figures out) and it's dedicated bandwidth. They are never over-subscribed.
I'm almost positive that this is not how they do things on the consumer side. Otherwise the Netflix debacle wouldn't be happening. But basically the consumer side is always over-subscribed and by a reasonably large factor. They're counting on consumer traffic to be bursty which it normally is. Netflix isn't bursty the way they're used to.
I sympathize with the idea of expecting one thing when building a model and seeing it play out differently. It sucks and it can cost a lot of money to make good on promises that you've made. But remember that Verizon is the one that made these promises. Their customers didn't put a gun to their head and say "promise me these crazy high download speeds or else!", Verizon did it willingly to try and steal customers from their competition.
Because they did it to themselves I have little sympathy.
See first page about CDN for a description of how they moved from CDNs paying for access to demanding ISPs installed thier CDN for free. http://arstechnica.com/information-technology/2014/07/how-co...
> But when we ask them if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download -- thus filling their upstream networks and nearly doubling our total traffic -- there is an uncomfortable silence. That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs.
> in other words, moving to peer-to-peer content delivery
Verizon is selling me a 50Mbps connection, but if I use it, for even a little, they get upset and say they are over capacity. This is the same company that over-subscribes their network on purpose, as most ISP's do. They will come into a neighborhood with a 1Gbps link, then sell 50Mbps to 100 homes, in hopes that not everyone uses the connection at the same time, nor for long duration.
The marketing papers over this in misleading ways, and that is bad, but if you wanted to pay for your own dedicated bundle of wires, like a businesses office building does, you could get it.
>The marketing papers over this in misleading ways, and that is bad
It's not a weak argument at all and the misleading marketing is exactly the reason it is not. In fact, Verizon has oversold bandwidth (or at least offered it for less than they are willing to accept).
But, what you seem to be effectively saying is that customers are at fault for buying an advertised product which touted the benefits they desired at a price that was acceptable.
If your last mile network was built to allow you to saturate your 50Mbps connection 24/7 your bill would be much larger than it is now. So much that you probably wouldn't pay for internet service, so oversubscribing is a totally normal and rational process when building out a network.
1. Kansas City, KS
2. Chattanooga, TN
3. Provo, UT
4. Austin, TX
These places all have fiber to the house, and supposedly gigabit speeds. Two of them were built out with zero help from Google, so Google can't be making huge losses on the internet to prop up other parts of their business.
Both me and my brother get billed when we call each other on our cellular phones! Help me!
Also, your phone provider does get charged for each incoming call. They just eat/redistribute that cost, since for normal phone user it's likely to be less than a penny per month. But try running a conference service and you're definitely going to get billed for incoming calls (apart from some weird force-subsidised areas where you can get paid to receive calls)
When I sign up with an internet provider, I am agreeing to pay a fee for open access to the internet at large. I'm not signing up to pay for each piece of content I access or each site I connect to.
I've never agreed, to the best of my knowledge to pay for access only to content served at a rate limited by the reverse utilization of the peering connection that content provider's content can be accessed across.
Are you saying that in the US when Ann calls Bob that both Ann and Bob will be charged?
If you are using a cell phone or a satellite phone, you pay for your connection to that network, whether or not you started the call.
You certainly don't make it so that grandma using her POTS has to pay for the guy with the more expensive connection. That is viewed as extremely unfair.
OTOH, using the cell network generally has free calling to anywhere in the lower 48. It's not the old physical wires that are the expensive part, it's maintaining and upgrading all the cell towers, and so people pay for that usage. Once you've paid to connect your call to the wired network, you can do whatever within very wide geographic borders.
http://devnull-as-a-service.com
Imbalance fixed, but I don't think that is what Verizon wants.
Of course, since the NSA would need to listen in on all that new random-bit traffic (since Netflix is obviously the new Skype in the terror playbook), taxes would need to rise significantly in order to pay for more NSA bulk-data-collection servers. But that's just the cost of living in America...
Seems to me it's a win-win for everyone, including poor border routers. Except Verizon then wouldn't make an extra penny from that efficiency, hmm?
Why should they have to do that? Level 3 has open ports now (so they claim). It doesn't matter which provider the traffic is coming from, either Verizon wants it or they don't (and apparently they don't, even though their core is no where close to saturated).
The balanced traffic requirement is bogus anyway. Its not as if Verizon offers their own video streaming service that I can subscribe to as a Comcast or AT&T customer. So they're never going to see balanced traffic in the age of video streaming.
- If you push traffic to my network it is because someone in my network requested it. - Incoming and outgoing traffic are technically no different. - It makes more sense to pay to receive traffic than get paid to receive traffic. That's actually what VZ customers pay for and that's why not many people complain about ADSL speeds.
If I were Level3 I would actively partner with VZ alternatives in the US, announce it to VZ customers so they can move away and then proceed to remove any peering with VZ. Enough bullshit already.
Instead of double billing (i.e. subscriber + content providers) couldn't Verizon just increase their subscriber rates to cover their perceived increased costs?
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts? Claiming that Verizon charges Netflix at extra e.g. $1/subscriber/month to peer?
Would it be legal for Verizon to literally just cut off Netflix? Just return an access denied to their subscribers.
For (2), Netflix could almost certainly do that, but I assume they'd prefer to not pass any cost onto consumers (because, really, they're making a stand against this business practice by Verizon being fair play at all). Among other reasons, if it's seen that Netflix is willing to pay a "ransom," other carriers may try to increase their contracted costs.
1. Verizon, as a monopolist, is already pricing its service at the level that extracts the most revenue while remaining within the envelope of whatever loose regulation they have.
2. Netflix, as a company in a competitive market, can't easily increase prices without losing customers. If Netflix charges Verizon customers $9 and Hulu is only $8, it makes Netflix look less attractive, and they'd lose customers.
Further Netflix isn't the only place to get movies. I can get movies from Amazon (streaming or DVD) Walmart, and other physical stores. I don't have that same kind of choice in ISPs though.
IPSs have more of a monopoly on their subscribers than video intermediaries do. The studios can release movies through a half-dozen channels. I would (figuratively) KILL for the chance to pick from a half-dozen ISPs that have offerings which are reasonable for the modern age.
We think of ISPs as selling us data, but the big ones have realized they can make more money as data market makers. This whole kerfuffle is just them trying to see if they can get away with it. Because they're effectively monopolies on the last mile, it's important that they fail here.
Verizon already increases prices x% per year, so maybe. But for every price increase there are some number of customers who will cancel or downgrade to a lower plan. (Alternately, Verizon could decrease their profits.)
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts?
Sure, but some percentage of Netflix customers would cancel.
Would it be legal for Verizon to literally just cut off Netflix?
Considering that the FCC is debating Net Neutrality right now, that would be a very bad idea.
There's more to "the Internet" than just "the WWW". While HTTP could be considered a "pull data" protocol, there's plenty of application protocols which involve "pushing" data over TCP/IP transport.
In other words Netflix doesn't start streaming data to me for the hell of it. Pandora doesn't stream songs to my computer unless I hit play. Songs don't get downloaded from iTunes unless I request them and pay for them.
This is in contrast to the mail that the USPS delivers to my door. I don't ask for most of it and I have to sort through it and throw all the bullshit away.
I guess what I mean is that most internet traffic is based on consent of some kind. While paper mail is not.
I might not specifically request any particular bits but the majority don't show up unannounced and unwanted.
> there's plenty of application protocols which involve "pushing" data over TCP/IP transpor
Right but by the time a TCP link is established you've already setup a session which can't be done unilaterally. That is akin to consent and quite possibly a request. You can't be "data-raped" because if you decline the TCP session that's the end of it. Someone can send a billion request to initiate TCP sessions but they can't initiate without your (or your computer's) consent.
Given the router/firewalls that basically everyone has between their modems and their computers it's getting very, very difficult to send data to residential ISP customers without their express or implied consent.
But under NO circumstances could you claim to do UDP over TCP/IP.
Were Netflix performing this "data rape" of Verizon's network it would be huge news as a DoS/DDoS attack. But that's not what we're hearing.
CAN people perform attacks? Yes absolutely and it requires no consent. But without some kind of consent it's regarded as an attack rather than as legitimate network traffic.
Yours is probably the clearest explanation I've seen of why this all feels so icky.
As an example I use a website (like Facebook or something) which does live auto-updating. Under the hood it's something like WebSockets or SocketIO and the server does send datagrams to me without me specifically requesting those individual data packets. I couldn't request them, I didn't even know about them until they arrived!
But while that technically is "push" from a high level it's still "pull" in that Facebook (or whoever) doesn't just start sending them to me for giggles. It doesn't start happening until I visit their site, and it stops happening fairly shortly after I navigate away from it. Their computers don't unilaterally send packets to me. I request them by continuing to execute the javascript that their servers sent to me when I loaded the page. Once I stop executing it, the connection is closed and the messages stop.
Uh, I don't think he meant it the way you took it. I think he knows how network traffic works.
To be pedantic curl -d test example.com would push data across the internet.
If it doesn't accept the connection your push goes nowhere. Imagine the webserver is down or the port is blocked. In that case the TCP session doesn't get initiated and you're unable to send the data.
http://en.wikipedia.org/wiki/Transmission_Control_Protocol#C...
EDIT: Considering that most consumer ISPs don't allow their customers to run servers (Google Fiber is an exception) it would be very very difficult for someone to successfully "push" data to consumers. And that's only if I'm willing to accept your definition of push, which I'm not.
Also, that has nothing to do with the point that "pushing isn't a thing on the internet" is a poor way of describing the problem being described, especially since the poster said the internet and not consumer's using their ISP (which still isn't strictly true, but is at least closer to the idea of the asymmetry in most consumer and business interent connections).
That doesn't negate the point that consumers download much more data than they upload.
When I say something incorrect, I expect to be corrected. I attempted to correct something that was factually wrong and don't believe I was mean while doing so.
I don't have any hard numbers but I would guess that at least 90% and very likely 95% or 99% of all internet traffic is pull-style.
Transferring data between two parties on the internet for any reason usually goes hand in hand with one person wanting to receive some data and another wanting to send it. If the sender doesn't want to send, no transfer takes place. If the receiver doesn't want the data, it's regarded as an attack.
I am having a hard time coming up with legitimate situations where someone sends data to my computer without my permission (express or implied) and that doesn't seem like some kind of attack.
But I'm also open (at least I'd like to believe I am) to someone cooking up a situation that proves me wrong.
Also, nothing I said despite it being asymmetric, just that his point about pushing and pulling isn't the best way of describing the issue.
His point about pushing vs pulling is a perfectly fine way of describing the issue. It's limited, but all explanations are.
Not always. Backup is popular today - as I look at my simple home utilization I push about 30% of the ingress up to "somewhere".
> Double billing for a single service is a neat trick if you can pull it off.
This is how every communications network has been ideally built from the beginning and this "trick" has been played out time and time again starting starting with the PSTN. It's not uncommon, but is also isn't ideal - unless you are the carrier.
You're equivocating on 'push' and 'pull'. Under OPs definition, a 'pull' is a customer's valid usage of the service, while a 'push' is something like unwanted or unrequested data transfer from a 3rd party. This definition is a little muddled when you talk about uploads but it should be easily understood. You uploading your backup data is you merely using the service you paid for, even though you're 'pushing' data somewhere.
For example, suppose there are 5 separate networks, run by different groups: A, B, C, D, and E. C connects to A,B and D, and D connects to C and E. In this case, the only way networks A and B can connect to D or E is through the connection between C and D.
Now, if in this arrangement, the D or E networks are sending/requesting a lot of traffic to/from networks A and B, while C is rarely requesting data from D or E, then network C might have a fair argument to make that D is abusing their balanced-flow peering agreement. Basically, they are being asked to be a middle-man for all the traffic going between networks that are NOT theirs. They might ask for compensation for doing this job (carrying traffic between two other networks).
However, this is NOT what Verizon does. All of the traffic between level 3 and the Verizon network is bound FOR A VERIZON CUSTOMER. They are not a middle-man, they are the end destination. Now, if Level-3 is sending data over those congested level-3/verizon gateways that is bound for, say, AT&T's customers, then they would have an issue and could fairly demand payment.
That is not the case, however.
Verizon is charging customers for access to internet content and they're bringing that content in over a peer balanced connection with agreements in place, intended for use in an actual peering environment, by business costumers of Verizon who are paying to provide content in the same way that Netflix pays Level3 to provide their content over their peer connections.
The conflict here is that Netflix is paying to provide the content, Verizon's consumers are paying to get access to the content. That creates a connection chain that is unbalanced in some way. Verizon just wants that imbalance to be double payment for them and nobody else.
But the Internet is a big hierarchy where things flow downhill. Every node can talk with every other node. Historically the networks would trade data freely if they were both doing an equal amount, but even 15 years ago I heard sober network engineers worrying that in the future there might be very unbalanced connections and what would be the fair way to deal with that.
NB: In the end I think Verizon is being purposefully difficult to their customers, but they are not completely unreasonable to think that having a lot of data sent into their networks [1] is just something they should STFU about.
[1] yes, even if you say "data requested by their customers" over and over again
Verizon's argument would be a lot stronger if they didn't attempt to say that Netflix chose balanced flow contracts. They are saying that Netflix went shopping for backbone providers and chose the ones that had balanced traffic flow agreements to deliver video across, which is just silly.
Such rhetorical overreaching is a common debating error, and can ruin an otherwise good argument. Verizon may or may not have balanced flow agreements, I don't know what the contracts are, but now I don't trust Verizon to honestly report on them based on their imputation of Netflix's motivations.
Oh, please. If that's really what's stopping this from being fixed, then I would be hard pressed to find a better illustration of the need for regulating "last mile" providers as public utilities. We're talking about an investment that is not even rounding error in Verizon's accounting, to connect Level 3's network to a Verizon network that, by Verizon's own admission, has plenty of excess capacity that is not being used. So the whole idea that, if the traffic isn't balanced, Verizon is somehow incurring huge extra costs that they need to be compensated for, is obviously bogus, which means their contractual terms are nothing less than an attempt to use their privileged position to extract huge monopoly rents in exchange for no added value whatsoever.
"Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows"
But isn't that a misstatement of fact? CDNs like Level3 do nothing but transfer data, usually media files, onto consumer ISP networks when those ISP's customers' request them.
In what dream world are Akamai and Level3 receiving as much data from Verizon's consumer network as they are sending? Why would any ISP expect a link with a CDN to have a balanced traffic flow?
But it is Verizon customers, who request the traffic and Verizon is not providing the best quality, because they don't want to upgrade their own equipment and the peering with e.g. Level3. In fact, it should be Verizon paying to Level3.
Verizon looks at it as if Netflix and Level 3 are pushing the Data, everyone else clearly sees that they are pulling it.
That's the way the Internet works - since consumers are generally prohibited from running servers (and even if they did, they wouldn't operate at the same scale), traffic from consumers is always going to be orders of magnitude less than traffic to consumers.
All of the data that Verizon is receiving from Level 3 is in response to a request that a Verizon customer has issued. Because the size of a request is much smaller than the size of the payload (response), the traffic will always have this pattern.
Do you have any evidence supporting that Level 3's contracts with Verizon require them to pay? This is the first I've ever heard that implied, and it would be very surprising, since it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this).
Not impossible though. I'd bet money that equal in/out contracts would be cheaper- maybe they just hoped no one would notice, or didn't anticipate the impact of Netflix on already-existing contracts?
It's not just Netflix - the entire Internet works this way. End-users send a request, and servers send a response. There are very few cases in which the response is not larger than the request. It doesn't matter what content you're serving - the raw traffic delivered to consumers is going to be greater than the raw traffic from them[0].
Saying that they didn't know that the traffic would be shaped this way is saying that Level 3 doesn't understand the way the Internet functions on a technical level, which I would say is impossible.
[0] This is (sort of) why residential Internet speeds are usually quoted asymmetrically (e.g. 20 down/5 up). Consumers - well - consume more traffic than they produce, so networks are already optimized for delivering more traffic to consumers than they themselves generate. That said, all of the traffic is initiated by consumers; the traffic Netflix sense is only in response to the explicit request by a paying Verizon customer.
You'd actually bet money, on the idea that Level 3—a major internet company—didn't understand a fact about internet traffic that's blatantly obvious to everybody? What?
If we do that, we find that physical link speeds of any variety have improved by many many magnitudes over. As this statement suggests, 10GBit is absolute jellybean in the networking world. Yet, last mile bandwidth available to customers has stagnated. Imagine a perfectly paved autobahn, and here we are discussing if we should be able to drive 20 mph on it, and who should pay for the extra damage that will cause to the surface. Incredulous.
There are some regulatory systems, where in a goldilockes moment, these basic truths are recognized and acted upon. Consider the EU drastically cutting roaming charges. What was so blatantly obvious that even the senile, crooked decision makers at the EU couldn't miss it? All the providers already roam over the internet, at basically zero cost for both parties if you consider that speeds are limited by the over-the-air interface at either end. Yet they kept charging customers the price for a specially erected microwave link from 1960.
Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.
This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one.
Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon.
And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal.
But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know.
But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.
But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already.
To me it seems like they're abusing monopoly(ish) power.
They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.
"Thus, class-action waivers must be enforced regardless of the consequences for the vindication of small-value claims, state or federal."
also, http://www.cbs46.com/story/24388303/consumer-advocates-fine-...
also http://publicjustice.net/content/tribe-scotus-denying-access...
also http://www.minnpost.com/community-voices/2014/05/forced-arbi...
So any disputes with verizon cannot go to court, they have to be arbitrated. Further, there cannot be a class arbitration. So to vindicate your rights against verizon, you'd have to be willing to bring an individual arbitration against Verizon and bear the costs of doing so which will be exponentially more than the value of your claim. In essence, these clauses immunize corporations from liability under antitrust laws, consumer protection laws; employment laws etc. It is a terrible situation created by a Supreme Court that despises class actions.
Much of net neutrality can be reduced to "don't let monopoly ISPs leverage that into a monopoly in other services".
If Verizon didn't offer TV, this wouldn't really be a problem. If there were 17 other broadband ISPs to choose from, this wouldn't really be a problem. But Verizon does offer TV and they're one of two choices for me and most of their other customers, so it's a problem.
http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...
One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.
EDIT: John Oliver also talks quite eloquently about this as well: https://www.youtube.com/watch?v=fpbOEoRrHyU
Edit: in the past I have had similarly good experiences with YouTube problems. They seem to go away as soon as you start accessing YouTube over IPv6.
Watch how suddenly you get 'connection reset by peer', look around and discover nobody in your packet path has any cause to reset your traffic. Now do a straight http proxy (varnish works well for this) to the same machine, run it all day night with bits of http traffic. No issues at all.
No go back to creating an SSH tunnel between your machine and the endpoint. Run traffic, note the mysterious 'connection reset by peer' when some thing upstream sends your TCP connection a FIN.
It could be that their network just has a really hard time with encrypted packets but some how I don't think so. I have tried a variety of port numbers.
EDIT: and I went back and verified I still have 'ServerAliveInterval 60' in my config file for all hosts.
Their response was that yes, they do disrupt all torrenting traffic, "because it is illegal and they must comply with copyright holders". Nevermind it was various linux distros I was trying to download and evaluate, all torrenting is automatically evil by association.
I had to switch to Cox because they're the only other option in my area. The 250GB data cap on my plan is BS, but at least I can spend that 250GB however I please.
With comcast my data cap is 300GB a month, I should be able to use 300GB of whatever I want Youtube, Netflix, hell even torrents or file sharing. Doesnt matter where it comes from I'm paying for 300GB of data at X speed. How hard is that to get?
Netflix, on the other hand, has paid good money for a very fast and reliable connection all the way to the Level3/Comcast PoP.
For yet another analogy in this thread - if your server was instead sitting in a data center with an open 10GE ethernet port or two, right next to a Comcast router, yes, they should just install the extra NICs and wire the things up without for the price of supplies and installation.
Should net neutrality go away? And is it fair what Verizon and others are demanding from Netflix/Youtube... Also, who should foot the bill, the backbone providers, the content providers or the last mile providers.
You talk of peering arguments yet you dont seem to back them. If netflix is a level 3 customer and level 3 has a peering argument with Verizon, then whats the problem?
L3 did the exact same thing to Cogent before, for the same reasons.
Think about your own ISP. I pay $80 for service, but to get the same bandwidth shared between 2 ISPs I'd have to pay $120 plus now I'm doubling my hardware costs (routers to each ISP).
Or perhaps they could ask Level 3 (who are clearly pissed at Verizon) to make the deal for them. These edge providers already have business with each other (peering), so it should be relatively simple for Level 3 to purchase some bandwidth from another large provider and then route some of their packets (e.g. the Netflix ones) through there.
In any case, the edge providers have good reason to collaborate on thwarting Verizon on this, so why don't they?
Verizon wants to bill it's customers for the bandwidth that they use. And then for some of that traffic, if it's from Netflix, they also want to bill Netflix.
Verizon is the company which is selling asymmetric internet plans so it should come as no surprise to Verizon that their customers are requesting more data than they're sending.
But they use this asymmetry to suggest that what's happening between Verizon and Level3 isn't business as usual (which it is) and that they should be able to double bill.
Verizon is doing a decent job of spinning but if you really look at what's going on it becomes clear that they're really torturing definitions to make their arguments.
I'm a Verizon subscriber and I get apoplectic when Netflix stutters, because I know exactly what's happening. But I also don't know what's the exact right model for peering points, and I think net settlement might be a better model.
The problem is that Verizon has apparently been successful in spinning the story that peering between Verizon and Level3 is supposed to be balanced. As others have already said in these comments, that doesn't make a lick of sense. Peering between backbone providers should be roughly balanced, but Verizon isn't acting as a backbone provider, they're acting as a residential provider, and there's no way peering with Verizon will ever be balanced. That's a fundamental consequence of the asymmetric plans Verizon sells and the fact that their customers are expected to download significantly more than they upload. This means that Verizon will absolutely be receiving a lot more traffic into their network than they send out.
Basically, I doubt Verizon actually has balanced traffic with any other provider, except perhaps other residential providers. But they're only making a stink about Level3 because they're trying to use this claim to double-bill.
Note again, Verizon made the intentional choice to service the residential market, and to sell asymmetric plans. They knew going into all of this that they wouldn't have balanced traffic with providers like Level3. And that's perfectly fine, because the only reason they're receiving this much traffic is because their paying customers are requesting it. Verizon has already been paid to receive this incoming traffic.
Also, and this is something I haven't seen anyone really address, balanced peering agreements between backbone providers is typically meant to prevent one provider from routing traffic through a second provider's network, destined for a third last-mile provider. In that scenario, the second provider doesn't gain anything from the traffic, and hasn't been paid for it. That's why the balanced peering agreement exists, to ensure that no provider gets taken advantage of that way; if you one provider wants to route traffic through another provider's network, they have to be prepared to receive just as much traffic.
But that argument doesn't apply to last-mile providers. The traffic isn't being routed through their network to a third destination. The traffic is being delivered to the network because that's it's destination. If the traffic was unsolicited (for example, a DoS), then it's reasonable for the last-mile provider to try to charge the sender for it. But if the traffic was explicitly requested, which is generally the case (and certainly is for Netflix), then the last-mile provider has no justification for charging the traffic sender. The traffic has already been paid for, by the subscriber who requested the traffic.
That's not true. I pay Verizon $X/mo to deliver the internet to me. And most of what I want delivered at 100mbps is netflix. Now instead of Verizon taking the millions (billions?) in revenue and building out what we're paying for, they're trying to charge netflix (for what they already charged me).
But the real question here is, is this a peering relationship anymore. And Verizon has a legitimate argument that it is not based on the sustained differences in traffic flows. I don't know what the answer is to this problem, but I'm not convinced by either Level3/Netflix or Verizon.
Personally, if I ran Verizon, I'd be meshing with Google/Netflix/Amazon's networks as much as possible. But I'm not, oh well.
Verizon and comcast have been selling asymmetrical residential links since always. Verizon is advertising 500/100 on their homepage right now. They know damn well that people are pulling way more data than they're pushing, and the interconnect links from level3-to-ISP have always been hugely asymmetrical.
That would be up to the arrangement you have with your internet provider.
What's happening here is that they are not making the good-faith effort. They want to get paid for the transit from Level3 onto their network and from their network to your front door.
The thing is according to basically all precedent of how the internet works when you pay for internet you're (theoretically) paying for all costs Verizon incurs while providing that service to you.
If they're trying to bill someone else for entry into their network while billing you for the exit then you should be getting a discount to the tune of however much that service provider is paying. But you're not. Which is why people are getting all up in arms.
EDIT:
> Level3 stops becoming a peer and starts becoming a subscriber
Yes, that point is when Level3 is DOWNLOADING more data FROM Verizon customers than UPLOADING to them.
Verizon charges their customers every month for plans where they download more than they upload.
As it stands Verizon customers are downloading 3x as much data FROM Level3 and Level3 is downloading from Verizon.
So by that logic, Verizon should be a Level3 customer and pay Level3 for any imbalances.
Honestly, it's a case of externalities, so we're not going to see much improvement until government either takes over peering or last mile.
If that's not a settlement-free peering agreement because of the disbalance, then Verizon should be the one paying part of their users subscription fees to the rest of the internet backbone.
>Q8: Comcast says that Level 3 sends it 5 times the traffic that Comcast sends Level 3. Is that true? If it is true, why shouldn't Level 3 pay for the traffic it sends to Comcast?
Comcast goes on to argue that Level3 pushes 5X more data onto the Comcast network than Comcast pushes onto the Level3 network. What I find ridiculous about the whole claim is that every single one of Comcast's residential internet packages has 5X higher download speeds than upload speed. (e.g. 105Mbps/20Mbps, 50Mbps/10Mbps, 20Mbps/4Mbps)
Comcast provides internet speeds that force customers to download 5x faster than they can upload and somehow Comcast is surprised that their customers are pulling down 5X more data then they're pushing to Level3?
[0] http://corporate.comcast.com/comcast-voices/20-qs-with-accur...
Regardless, if Level3 keeps insisting this is a peering point, and then demonstrating that it's more like a transit to ISP link. Which ISPs pay for.
But after reading this... am I the only one that thinks peering agreements are done bass ackwards? If Verizon's customers are _requesting_ Netflix access, shouldn't Verizon pay Level3, not the other way around?
When you buy business backhaul from Verizon you get guaranteed bandwidth provisioned on their network from point A to B. They don't over provision because they can't: contractually. If they don't meet their SLA the customer is out of the contract and can go elsewhere.
On the residential side things are very different. We have no SLA other than "pfft whatever we'll roll a truck next Thursday I guess"
Verizon might share long distance backhaul between residential and commercial sides of the company but if so it's done with strict quotas to ensure that the commercial side of things isn't impacted. It's not as though the core network division lets everyone have a free-for-all on the available bandwidth through the core.
If I had to guess I would venture that the consumer group either runs largely or entirely separate equipment. They might share fibers but I'm not all that confident about that. They're very, very close to entirely separate businesses.
I'm not getting why I'm being down voted for posting the cached version when I was getting the 404.
edit: this one seems a bit better:
http://webcache.googleusercontent.com/search?q=cache:DBHDyx7...
now i just torrent everything so i can watch uninterrupted when i want. I consider i am still legal since i'm paying for both subscriptions yet and they pay the content. if not, not my problem.
but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?
as i just showed, they are the reason lots of people are going back to torrents that do not make money to the studios like streaming services does. Why do they rather sue john does instead?
Because it presupposes that whether or not in fact, streaming via Netflix is profitable or not, the studios didn't have to be dragged kicking and screaming to allow it at all.
But, assuming they really did need to throttle traffic in order to protect their own network. For one it wouldn't make sense to control your throttling by running your routers at 100% capacity. But also any throttling should be equal among all of their peers and not selectively chosen based on Verizon's business interests.
By definition, the only way they can be 'crushed' is if they're not using the funds they get from their customers for maintenance of their network.
Even with Net Neutrality it's fully within an ISPs right to say X GB/mo for $Y and throttle or block over-use. The problem is an ISP deciding that, regardless of what is going on in the network, regardless of what a particular user has used or is trying to use, traffic to Netflix will be slower than traffic to Vudu.
Verizon, Comcast, et al have plenty of available tools to prevent congestion without violating Net Neutrality.
They're intentionally trying to confuse the issue with usage, because it's an easier PR sell.
But Level 3 claims it would be very inexpensive for Verizon to increase capacity there. Just a few port cards and a few thousand bucks for a major city like L.A. They even offered to pay Verizon's costs.
This is Verizon wanting to double-bill for traffic. Plain and simple.
> Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic".
That's Verizon's job. To go out and get by any and all means necessary enough bandwidth to satisfy their customers, namely the residential customers who seem intent on using Netflix. At no point do they have a moral imperative to hold their customers hostage to extort money out of someone else that their customers are trying to access. It's already been paid for!
All of your comments keep ignoring or misunderstanding what peering is. It is supposed to be roughly equal data traffic for both sides. Once it is lopsided in one direction, it is no longer peering.
It doesn't matter who requested what, the fact that L3 traffic is consuming a majority of the bandwidth now is the issue, and Verizon wants L3 to pay to fix that.
If anything, Verizon should be L3s customer, and L3 giving them SFP is doing Verizon a favor.
Verizon is a Tier 1 provider: https://en.wikipedia.org/wiki/Tier_1_network
Yes much of the traffic ends up at end user destinations, but this doesn't negate the fact that the peering arrangement is no longer roughly equal.
Those companies couldn't be more different.
At the peering point the arrangement is no longer mutually beneficial for both parties due to one side of the peering arrangement now using more than its fair share or traffic. This is what Verizon has issues with.
You can start talking about peering and Tier 1 ISPs but Verizon Residential is not the same company as Verizon Business. Verizon Business is a Tier 1 ISP, Residential is not.
L3 can't consume any bandwidth on Verizon's network because their customers don't unilaterally push data to Verizon customers. If there is any traffic from L3 to Verizon it's because Verizon's customers have requested it. That means Verizon should make a good-faith effort to acquire sufficient bandwidth to sate their customers requests all the way up to their theoretical contractual obligations. Only once the bottleneck is COMPLETELY out of Verizon's control do they have any room to suggest that "well our service is only up to X Mbps and in this case it's out of our hands"
In this case they have the power to fix the problem and they are choosing not to, to the detriment of their customers. It's shameful.
EDIT: If the link were 100% saturated in the other direction then you could potentially make the argument that L3 is causing the problems. In that case their customers are requesting so much data from Verizon's customers that the link gets congested. And in that case it would be paramount for L3 to figure out a way to get Verizon to upgrade their side of the peering agreement. But that's not the situation here.
There is a peering location. At this location traffic is supposed to be roughly equal in both directions. At one time it was, and both parties were happy. It no longer is the case. L3 is now consuming >50% of the bandwidth. Verizon states that since they are using more than their fair share they should pay cash to upgrade the network. It doesn't matter who requested what, or from where.
Also your edit is correct. If 90% of the traffic at the peering point was outflow from the Verizon network, then they should be paying to upgrade the bandwidth.
I will sketch out my argument as follows:
1. Data transfer requires mutual consent or else it either a) doesn't happen or b) is considered a network attack
2. On the internet today the majority of transfer flows are asymmetric; regular users download more than they upload. I don't know if this is the natural way of the universe or it happens this way because of the way residential broadband is structured i.e. more download capability than upload
3. People who have more outflow than inflow pay to be connected to the internet (i.e. servers)
4. People who have more inflow than outflow pay to be connected to the internet (i.e. regular users)
5. Somewhere this traffic has to meet in the middle and transition from a network where the sender is paying to the network where the receiver is paying
6. This is typically called a peering point and the transit here is usually settlement-free not because it's usually balanced (though in many cases it is) but because it's the point at which the sender-pays network meets up with the receiver-pays network. In other words the sender pays for the network transit from their servers all the way up to the peering point and the receiver pays for the network transit from the peering point onwards to their house.
7. When looking at the problem this way it's hard for me to determine who is "using" more network bandwidth because again, data is only transferred by the consent of parties on both the sender-pays and receiver-pays network
I understand the argument that Verizon is a Tier 1 ISP and thus deserves to be paid for transit. The Tier 1 ISP is actually Verizon Business a largely separate division of Verizon from Verizon Residential. But Level 3 is also a Tier 1 ISP and thus by that logic also deserves to be paid for transit. http://en.wikipedia.org/wiki/Tier_1_network#List_of_tier_1_n...
I also understand the argument that "all peering needs to be roughly equal or else someone needs to pay" but again when company A has customers who send more than they receive and company B has customers who receive more than they send I can't understand why A and B don't agree to let the ratios move away from balanced. I understand historically the ratios were close to even but when they're both selling asymmetric service to their customers why insist on symmetric peering?
EDIT:
From Wikipedia "peering" is sometimes called "sender keeps"
"In computer networking, peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the users of each network. The pure definition of peering is settlement-free, "bill-and-keep," or "sender keeps all," meaning that neither party pays the other in association with the exchange of traffic; instead, each derives and retains revenue from its own customers."
I think you should read over this site which has tons of info:
I don't believe that I am. There is no natural law of the universe (like gravity or electromagnetism) that says that peering MUST be symmetric or roughly symmetric.
A symmetric or roughly symmetric peering arrangement naturally arises when two companies which have both traffic producers and consumers on their network interconnect their networks at a peering point. But if one network operator doesn't have the right mix of producers and consumers on their own network the balance can get pushed in one direction or the other.
From their website:
"The Peering Coordinator Community put on a debate on the rationality of peering ratios as a peering discriminator at NANOG 35 in Los Angeles. During that debate, and during the subsequent informal debates afterwards, the consensus was that this metric was neither technically sound nor business rational."
http://drpeering.net/white-papers/The-Folly-Of-Peering-Ratio...
Peering isn't about getting exactly identical amounts of traffic both ways. A lot of times that is the outcome, but it is NOT required.
Peering is about the point at which two companies interconnect in such a way as to keep their paying customers happy without incurring additional expenses.
Please read (and re-read if necessary) that page and look at the diagrams and make sure that you understand them before you continue to comment on this topic. Your comments thus far indicate that you don't really understand how networks work. It sounds like you've heard "peering" before and kinda understand what it means. And this dispute between Level3 and Verizon doesn't fit into the neat and tidy little "peering is symmetric" box that a naive understanding of peering would yield. I am sorry if this sounds condescending but you're making a lot of posts with no real bearing on what's actually happening or why.
As well, Netflix will do the same -- literally stick a disk array of videos inside Verizon's network.
If most of your traffic is residential, then you're not a peer, then you're a subscriber - residental ISPs have to pay others for their Internet access. Verizon has also a valid transit&peering infrastructure, but if we would be looking at their residential business separately, then it would have to pay for the customer<>Netflix traffic on their interconnections.
The haggling over semantics here doesn't really change the substance. Ok, "Netflix" isn't saturating the interconnection.. "customers using Netflix" are saturating the interconnection. And it would still reportedly cost Verizon very little to fix it.
I think that points the finger pretty strongly at Verizon.
I wonder if Netflix or Verizon would lose more customers...
They should continue to allow access, but they should also keep saying when the problem is with the network provider. Customers love Netflix (NPS of 54) more than Verizon (NPS of 32), so they'll be more inclined to believe Netflix.
I actually don't know. Is that legitimate?
For transit providers, asymmetry = Paid.
For local ISPs, they are typically willing to asymmetrically peer because they'd otherwise have to pay transit.
Since verizon is a T1 transit provider, it doesn't want to give it's service away for free.
I think the fair answer in this dispute depends on whether L3 and Cogent are forcing Verizon to provide transit for free or are they just delivering the data the local network for the customer (like a local ISP peering arrangement).
I'm not sure which is going on. If L3 is just dumping Netflix packets on the nearest Verizon peering location, well then Verizon is providing them a transit service. If L3 carries the data all the way to the local network, then it's not transit at all.
I'm not sure what is actually happening. I'd imagine a mix of both.
From Wikipedia:
"However, the most common definition of a tier 1 network is a network that can reach every other network on the Internet without purchasing IP transit or paying settlements."
Not really, for the consumer ISP side of its business Verizon absorbs more traffic than it produces by definition: consumers don't typically produce much data outside of p2p.
Furthermore, Netflix has offered putting server inside Verizon's network which makes this argument moot.
VZ just wants to double-bill.
But honestly, it couldn't be done. Not with the upload restrictions at the end of the pipe.
What if Netflix got that number to go up from say 30kbps to 500kbps? Still only 10% of upload speed link utilization on a 50/5 Mbps customer line.
Even if the ratio only went from 3:1 to 3:2 suddenly you're much, much closer to balanced. Takes a lot of wind out of their sails. Instead of "ZOMG THREE TIMES AS MUCH!" it's "FUCK IT'S 50% MORE!" which doesn't sound nearly as good.
The providers that Netflix does not use do not experience the same problem.
This needs to be put in front of the FCC since it is the bare minimum language they should be able to understand.