Hacker News Reader

Comment by amalag | Hacker News Reader

I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level 3 and Verizon is lacking.

Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.

View on HN

Yes, it absolutely is a simple fix.

This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one.

Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon.

And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal.

But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know.

But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.

But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already.

To me it seems like they're abusing monopoly(ish) power.

> Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.

They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.

Yeah that's a good point. What I meant was "without any charge to Verizon"

Absolutely, and you made the point well. I think when companies start butting heads like this, some people don't recognize that it's not just some corporate bank account, but the results of the dollars they've already given for the services they've signed up for. I think that disconnect is responsible for a lot of the lack of outrage around this.

I wonder if Verizon's customers could form a class action lawsuit accusing it of failing in good faith to deliver bandwidth from Netflix due to intentionally congested handover points.

Verizon has been given immunity from the threat of a class action by virtue of its arbitration clause class action waiver. Same is true comcast, AT&T etc. You can thank the Supreme Court for this.

Hmm, I've heard of immunity over class action wiretap related issues, but bandwidth delivery too? Any references? "class action immunity verizon" just has too much noise.

Aren't such waivers automatically null and void at least in some states? In EU, such disclaimers are simply a waste of ink, but I recall seeing that some USA states had gone that way as well.

http://media.mofo.com/files/Uploads/Images/130725-Employment...

"Thus, class-action waivers must be enforced regardless of the consequences for the vindication of small-value claims, state or federal."

also, http://www.cbs46.com/story/24388303/consumer-advocates-fine-...

also http://publicjustice.net/content/tribe-scotus-denying-access...

also http://www.minnpost.com/community-voices/2014/05/forced-arbi...

So any disputes with verizon cannot go to court, they have to be arbitrated. Further, there cannot be a class arbitration. So to vindicate your rights against verizon, you'd have to be willing to bring an individual arbitration against Verizon and bear the costs of doing so which will be exponentially more than the value of your claim. In essence, these clauses immunize corporations from liability under antitrust laws, consumer protection laws; employment laws etc. It is a terrible situation created by a Supreme Court that despises class actions.

Verizon's advantage should be obvious: they also sell TV service which is in direct competition to Netflix. If Netflix works poorly, that's more customers potentially signing up for FiOS TV.

I feel like this is what people often forget in the net neutrality debate. The pipe providers have a horse in the race (FiOS TV On Demand, XFinity On Demand) and we need to make sure they aren't rigging the race so to speak.

Going beyond that, they typically have a monopoly or duopoly as an ISP, while they're subject to competition as a TV provider.

Much of net neutrality can be reduced to "don't let monopoly ISPs leverage that into a monopoly in other services".

If Verizon didn't offer TV, this wouldn't really be a problem. If there were 17 other broadband ISPs to choose from, this wouldn't really be a problem. But Verizon does offer TV and they're one of two choices for me and most of their other customers, so it's a problem.

Furthermore, if Verizon degrades connections to whichever ISPs Netflix uses, then they have leverage to get Netflix to pay Verizon to peer. If they make it so that the only way that Netflix can offer their service is to pay them for it directly, then that is more money for Verizon. Very clear motivation.

It's even stated parenthetically in the blog posting when comparing European and US Level 3 and Verizon peering utilization, the former ~18%, and the latter ~100% "(where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses)"

I'm not a lawyer, but when you put it that way, it almost screams "RICO violation," doesn't it? ;)

They just want the same deal as Comcast, that Netflix, in a moment of stupidity, made:

http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...

It's not stupid at all. By giving into the racket, Netflix demonstrated that it's not a technical issue at all, so that argument no longer flies. It's clear that the ISPs are capable of supporting the service, they just don't want to.

Well not really. They can just claim that without being paid extra to do so, it wouldn't be cost-effective for them or whatever. I mean that's their public argument anyway - that these services cost them too much and they need to pay up.

This is not and has never been a technical problem. It is purely in the business and politics areas.

One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.

Its not always great on Comcast either. It has gotten better over the last 6 months though, I will admit.

I bought a VPN and it fixes it. Sad that it is necessary.

Which VPN did you go with?

I used privateinternetaccess, i am happy with it. I don't even use it all the time but the speed tests hold up against my 50Mb/s connection. Hulu has blocked it if that is a concern, there is some other VPN that has more egress IP's that works with Hulu, but I didn't know about it and it was more expensive anyway.

You can run your own VPN on a VPS and have a dedicated IP without needing to worry about Hulu getting blocked.

I did this on Comcast as well, but its interesting to note they do shoot downs of ssh tunnels. Using IPSEC hasn't been an issue though.

Can you describe that ssh tunnel issue? Are they monitoring your traffic and actively disrupting anything that looks like ssh?

Sure, set up an SSH tunnel, make it a SOCKS5 proxy (easy enough to do) to a host in the 'cloud', now proxy you traffic through it (like Netflix).

Watch how suddenly you get 'connection reset by peer', look around and discover nobody in your packet path has any cause to reset your traffic. Now do a straight http proxy (varnish works well for this) to the same machine, run it all day night with bits of http traffic. No issues at all.

No go back to creating an SSH tunnel between your machine and the endpoint. Run traffic, note the mysterious 'connection reset by peer' when some thing upstream sends your TCP connection a FIN.

It could be that their network just has a really hard time with encrypted packets but some how I don't think so. I have tried a variety of port numbers.

EDIT: and I went back and verified I still have 'ServerAliveInterval 60' in my config file for all hosts.

When I was on Comcast, I felt like they were doing this to gaming traffic as well. I have no evidence to back it up except anecdotal experience though.

When I was on AT&T U-Verse they did this with torrents. I replaced my router, OS, and modem thinking the problem lay somewhere on my end, and once I ruled all of that out I called up AT&T to complain and call out the fact that it happened only when I torrented.

Their response was that yes, they do disrupt all torrenting traffic, "because it is illegal and they must comply with copyright holders". Nevermind it was various linux distros I was trying to download and evaluate, all torrenting is automatically evil by association.

I had to switch to Cox because they're the only other option in my area. The 250GB data cap on my plan is BS, but at least I can spend that 250GB however I please.

I had problems a while back with any long-running downloads or streaming video on ATT DSL. Basically made some uses impossible - any large remote data that couldn't resume at arbitrary points. Went completely away when I switched to a third-party ISP - using the same ATT-owned DSL lines.

I signed up for a VPN service right at the end, and that cleared everything up for me and completely confirmed they were intentionally shaping and disrupting traffic.

I currently have a 6in4 IPv6 tunnel through Hurricane Electric. At least netflix.com seems to load over IPv6. I wonder if my Netflix experience is fine precisely because my traffic appears to go to HE first, then to Netflix.

Edit: in the past I have had similarly good experiences with YouTube problems. They seem to go away as soon as you start accessing YouTube over IPv6.

Interesting. I'm going to have to start trying Netflix and Youtube over IPv6.

I have Comcast as well and it's gotten because of the deal Netflix struck with them... http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...

EDIT: John Oliver also talks quite eloquently about this as well: https://www.youtube.com/watch?v=fpbOEoRrHyU

Verizon probably needs to pay Level 3 for the bandwidth - so apart from the cost of the hardware (cards + cables) there is a possibly large cost which is not mentioned in the Level 3 article.

Netflix has offered to directly connect to Verizon without the need for Level3 to be in the middle.

and Verizon wrote in their post that they "working" on this: "we are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network." But "working" could mean: "waiting for Netflix to pay for it".

Netflix already paid Verizon's ransom and now Verizon is building connects to Netflix but such connections take time to install. Level 3 is arguing over old news at this point, but they're trying to set a precedent to prevent other ISPs from trying the same thing.

This is a good point, but for a different reason: IIRC, Netflix is paying Level3 for connectivity. If Netflix pays Verizon for direct connectivity, Netflix still pays someone for bandwidth, and L3 loses their cut.

Verizon probably doesn't need to pay Level 3 for the bandwidth. Typically peering points like this are settlement free. In other words it's a voluntary exchange of traffic and each side of the line keeps 100% of the revenue from its customers using the interchange.

And Verizon claims that peering points are intended to be roughly equal in directional usage. Which this is not. Are there examples of peering agreements where companies don't care that usage isn't balanced and still don't require some sort of net settlement?

Peering links between backbones (like Level 3 and Verizon Business) are supposed to have equal ratios. Peering between content providers and end-user ISPs don't care about ratios.

Right. And the latter has to pay for it.

The latter what? I think content peering should be free because there's no fair way to set a price for it.

Why not? There's plenty of price data for high volume bandwidth applications, and even if there weren't, the parties could negotiate for it.

An end-user ISP like Comcast has a monopoly on access to their customers, so if they're allowed to charge for peering there is no competition to limit how much they charge. This tends to result in the ISP allocating 99% of the surplus created by peering to themselves.

That's very true. The only limit would be losing customer (HA!) because they didn't connect to the right content providers. There's some interesting network effects here that smell like positive externalities, which would explain the underinvestment. But also suggest that there isn't a good solution.

Right. If they really cared about ratios they wouldn't offer unbalanced service such as 50/15. They have built their network around users predominately consuming content and then complain when there is too much consumption from specific providers that happen to compete with services they also offer.

I'd say a lot of the peerings at DECIX and AMS-IX.

Good to know. And these all remain settlement-free?

iirc it is part of the peering culture there, but I am not sure exactly, never having worked in that field

I'd say that wmf addressed it already. Peering points with one side being primarily consumer services are always going to be unequal due to the nature of the end user and the ways in which consumer ISPs provide bandwidth to their customers. Lots of download, and very little upload speed as an example. As such it's nearly impossible for many peering points primarily used by consumer services to use an equal amount of traffic on both directions. Both sides of the agreement know this and it's still usually in both of their best interests to peer settlement free.

Reply on news.ycombinator.com