Verizon made an enemy tonight | Hacker News Reader

Verizon made an enemy tonight

iamnotaprogrammer.com

133 pointssudonim49 comments

Verizon, TWC, Comcast. All of these providers are local monopolies.

This is the root of the problem.

Here in Australia I can choose any number of retail ISPs that will service my Fibre to the Home connection. If I was not living in the building I am now I would be able to choose any number of ADSL ISPs. This creates competition and fixes the problem.

Funnily enough this access is legislated here in Australia. Yes. Legislated.

There can be no ISP monopolies in Australia.

Better yet we have embarked on building a Fibre to the Home national network called the National Broadband Network which functions under the same scheme, ensuring we won't have any bullshit monopolies for the foreseeable future.

Americans should campaign for the same solution, enforce last-mile wholesale and legislate separation of ISP retail from ISP wholesale business units.

Everyone wins.

Everyone wins.

No. The rich can't get obscenely richer.

I firmly believe markets should be regulated for competition. I'm sure there are examples of companies merging and creating an improved experience for their customers, but I can't think of any. On the other hand, I can list half a dozen companies I've done business with, where a merger has reduced the quality of service and while increasing its price.

Tons of markets that have been "deregulated" have become more competitive - national airlines in Europe, Ma Bell in the US - but they are almost all about breaking up a government sponsored/created monopoly. The US market telecom market has done an amazing job re-merging, to the point that a few more mergers and it'll be Ma Bell all over again.

> Better yet we have embarked on building a Fibre to the Home national network called the National Broadband Network

Just feel I should add that while we did embark on building a Fibre to the Home network, after the last election we're switching to a Fibre to the Node network instead.

I think this issue may be more subtle than we want it to be.

I'm no fan of monopolistic ISPs, but it is true that when you have a peering point with unbalanced traffic flows, the side sending more traffic is supposed to pay the side receiving more traffic. You pay per byte sent, just like how the sender of a letter pays postage. This has always been the case, and as long as Verizon charges Level 3 the same amount per-packet as they do any other sender, there's no violation of net neutrality.

So Level 3/Netflix is making the argument that all Verizon has to do is install some additional network cables and everything would be fixed. Well that's a convenient argument, given that Level 3 sends way more traffic than it receives on that link, and currently (it seems) isn't paying for that difference. I hate to say it, but I think Verizon may actually be in the right here, at least insofar as that Level 3 should in fact be paying for that peering arrangement. Level 3's blog post seem deceptive on this point.

That said, Verizon has a video service that competes with Netflix. Verizon's video service presumably doesn't have to pay transit fees to reach Verizon customers. So what's to stop Verizon from charging unreasonable fees in order to stamp out competition? Presumably, this is the real problem: Verizon and Level 3 cannot agree on a price, because Verizon has no reason to offer a reasonable price.

So Level 3 and Netflix are waging a public campaign to shame Verizon, presumably as a bargaining chip to get the price lower. But neither side is really being truthful with us.

I'm just happy that in my area I can get sonic.net, an ISP whose only business interest is delivering packets.

I don't understand why ISPs can just be ISPs? Especially the telco ones are investing a lot of money to try to move up the value chain into things like video, IaaS, home security, probably more that I can't think of. Meanwhile they fail to innovate at actually being an ISP. Google seems to be the leader there with Google Fiber and their clever sofware for managing their data center networks.

> Well that's a convenient argument, given that Level 3 sends way more traffic than it receives on that link, and currently (it seems) isn't paying for that difference.

That is true if Level3 was sending traffic through Verizon to say Cogent or HE. But in this case, the OP is requesting this traffic through VZ and he is paying VZ to carry this traffic for him.

>requesting this traffic through VZ and he is paying VZ to carry this traffic for him.

That's a very disingenuous way of putting things.

Let's get all the actors in play here:

Netflix - The website/service you're trying to access

Level3 - Netflix's ISP

Version - Your ISP

You

So to get to Netflix you'er going through 2 different entities.

If you consider Level3 and Verizon to be on the same level ( after all, they're both ISPs), then your argument works the same way the other way around : Netflix is paying Level 3 for the connection between it and you, so why shouldn't Level 3 be responsible for this? Why is only Verizon responsible?

The second way of seeing things is that Verizon is responsible for the connection between you and Level3 (i.e. Level3 is a Verizon customer). Level3 wants to just plug in more cables to get better speeds on Verizon.... well who wouldn't expect to have to pay more for that?

In no situation is Level3 entitled to the maximum speed technically available to it for no cost. The counterpart to this is that it's Verizon's responsibility to offer reasonable pricing for upgrades (and not have content discrimination and conflicts of interest).

If Verizon offers the same pricing structure for backbone infrastructure to everyone who asks, it's hard to fault them on this. Level3 is in a particular position given the size of the traffic served.

But if I were a mailer for something like Amazon and were responsible for 35% of packages sent during a day (fake number), it's not like I could just ask FedEx to let me use more of its trucks and get higher priority for free.

> after all, they're both ISPs

No, Level 3 is not an ISP. So the argument doesn't work the other way around. Also, you've framed things to make it sound as if Level 3 is initiating the traffic, when it is Verizon customers who are pulling data from the Level 3 network.

If I am a paying customer of an ISP, I sure would expect them to get me the content I want at a reasonably high speed, even if that means they need to bring up additional capacity.

You could say Verizon is overselling their bandwidth to customers and that's why they have degraded connections to Level3. Is it not Verizon's obligation to their customers to upgrade their connections to handle what they've sold to their customers?

> ...it is true that when you have a peering point with unbalanced traffic flows, the side sending more traffic is supposed to pay the side receiving more traffic.

Up until very recently the only time you paid for peering was when you were actually buying transit. Level 3 is definitely on the right side of this argument, not only logically - but also from the perspective of precedence. Anybody interested in learning how the internet actually works should read "The Internet Peering Playbook".

Hmm, I guess I don't understand why sending packets to an end user over an ISP is not "transit", but I'll go ahead and admit I know very little about this subject and am probably wrong.

It seems that the peering arrangements typically pay for transfer through one network to another, not to get data to the end consumer.

I don't know enough about the issue to say whether that's true or not, but that seems to be the argument they're making.

If it were considered "transit" in the same sense as delivering packets from one network to another across large distances, then consumer Internet access should be free because it's always biased in favor of receiving rather than transmitting bytes. Verizon should even be paying its customers to accept the extra bits they get.

That doesn't seem entirely obvious, since last-mile infrastructure is a completely different beast from backbones.

But yeah, I clearly don't know what I'm talking about at this point. Sorry.

I think you understand more than you admit ;-). The last-mile infrastructure is indeed a completely different beast, and Verizon is last-mile infrastructure. So backbone-to-backbone might be symmetric, but backbone-to-last-mile should never have been expected to be symmetric.

ISPs pay for transit, tier 1 networks do not (by definition). Level 3 is definitely a tier 1 network, but Verizon is acting as both a tier 1 and an ISP. It isn't a very complex problem, tier 1 networks don't pay transit.

I guess that makes a certain arbitrary kind of sense... :)

Peering is when you send packets from your network (or your customers' networks) to my network (or my customers' networks). Transit is when you send packets from your network (or your customers' networks) to my network, and I agree to deliver it, regardless of if it's within my network, my customer's networks or beyond my network (via peering or my own transit connections).

Peering is historically settlement free at public exchanges and in private connections with balanced traffic. When a private peering is not balanced, historically the peering would be disconnected, but in the mid 2000's, paid peering started showing up more frequently as an option. Paid peering is often less expensive than transit, but it depends on the networks involved and the volume.

Transit - I send you data and you deliver it to another ISP or backbone provider.

Not transit - I send you data and you deliver it to someone in your own network.

Yeah, the thing is, I don't consider myself to be part of my ISP's network. I consider my ISP to be "transit"ing packets from the internet to my network. :) But I guess that's not the accepted definition.

I was pretty sure the precedent was that high volume peering was for 'balanced flows'. At a public peering point with a shared LAN, balanced flows aren't usually as required, but capacity would usually be less as well.

If Level3 (or Netflix) would read the Peering Playbook, they would see that they need to attract some upload traffic from Verizon to balance their flows, and get Verizon to upgrade the links (then, once the links are in place, they can drop the upload traffic). If I were Level3, I would find a backup service or image/video hosting site and convince them to let me advertise their IPs to Verizon/Comcast to balance the flows.

I looked for old level3 peering policies, and actually in 2004 they didn't mention balanced flows which was surprising, because most "Tier 1" networks have required balanced flows for peering for a long time.

If you don't like Level 3 paying Verizon/Comcast, consider the situation where Verizon has no backbone, and pays for 100% transit from a hypothetical Tier 1 provider. If Level3 doesn't have balanced flows (within usually a 2:1 or 3:1 ratio), most Tier 1's would depeer them as well (or refuse to upgrade circuits).

Flow ratio balance will fall apart for whoever has the shortest route to Netflix, so it is certainly a very interesting problem. Verizon has demonstrated that it isn't interested in fixing the problem though (by refusing Netflix cache devices). So even if Level 3 were to somehow balance the traffic, you can be sure that Verizon would still screw with Netflix traffic - they're simply using the balance argument as an excuse.

I'm no fan of monopolistic ISPs, but it is true that when you have a peering point with unbalanced traffic flows, the side sending more traffic is supposed to pay the side receiving more traffic.

Such an arrangement with a consumer ISP is so preposterous I have to assume that Level3 and Verizon never expected flows to be symmetric.

Level3...isn't paying for that difference.

Right, because Verizon's customers pay for the difference.

The beef of the problem is that Verizon is both a consumer ISP and a transit provider.

Except in the case of Netflix, it's purely an ISP.

It looks to me like the OP is making more of a beef about Verizon's advertised speeds being significantly different from their actual speeds. That Verizon is continuing to advertise speeds that it intentionally is not delivering, IMO, constitutes fraudulent advertising. This can and should be the basis of, and resolved by, a class action suit.

It's no different really than an ISP that advertises 100Mbps, has ethernet to the home, but only a T3 to the Internet.

I'm sure their customer contract has clauses which specify that "actual speeds may vary". This does not, however, make their advertised speeds any less knowingly false.

Of course, their actual advertisements include disclaimers so they're not actually making claims about their upstream connections - legally, they would be able to get away with a T3. The problem is that there's no economic reason for a competitor to lay their own last-mile lines to the vast majority of consumers and compete on upstream features, since not enough people care enough to switch for them to recoup the capital investment. It's truly a first-mover-takes-all scenario.

As I understand it, Verizon customers are requesting data from Netflix and Level3 has to carry that data from Netflix to the edge of the Verizon network. Verizon has got a profit from their customers so I would say that in any case VZ are the ones who should pay L3 to bring the traffic to their edge.

Or, you know, stop the bullshit and keep the Interwebz working as they've been for decades.

I think you are ignoring the crux of the argument. Verizon has agreed with the customer basically to "go fetch" what the customer asks for. Netflix is not sending data to the customer, the customer is the one who has asked Verizon to "go fetch" at a price agreed between them.

Verizon operates on the agreement with the customer that they can afford to meet the customers data needs at the price agreed.

Your ISP is like a courier service who agrees to collect and deliver your parcels from the parties you order from, with the understanding that you are bearing the full cost of the delivery.

At no point do they state that if the party you are ordering from will not pay them a fee, they can wilfully restrict the amount and the speed at which the parcels are delivered.

The OP's experiment demonstrated clearly that Verizon's didn't lack the ability to deliver the packets at the speed he had 'ordered' regardless of source, but was deliberately throttling the speeds when they realized they were from Netflix. By routing the connection through the VPN, Verizon lost the information the throttling was based on and the downloads arrived at a higher speed.

Any one who hosts a server knows that you can transmit terabytes of data for a tenth of the price ISPs charge customers.

It is purely corruption and venality on the part of the FCC and the legislators when entertain the arguments from the ISPs on net neutrality and ignore the breach of faith and contract that the ISPs attitude towards the customers entails.

The proper solution is the for the customers like the OP to bring class action suits against the ISPs and force them to reveal the financial and technical details to see if they have any substantial bearing on the ability of the ISPs to deliver on the services they have promised.

@kentonv With regard from the customer end, the issue of peering arrangements is basically a false one. Netflix shouldn't have to pay a penny to anyone. Verizon's agreement from the customer viewpoint requires them to build their pipes right up to Netflix's doorstep if that is what it takes them to collect and deliver the packets the customers have requested.

This whole debate is basically about the legislators and the FCC giving Verizon et al an opportunity to rip off both customers and service providers.

Are there any consumer-facing ISPs which send as much traffic to somebody like Level 3 as they get from them? I doubt it: most consumers recieve a lot more data than they send. So by your logic, all people peering with Verizon should pay them (except maybe TWC or Comcast).

So if Netflix clients would start pushing random bits upstream to balance their downstream, everybody would be happy. Even better if they push a bit more, then Verizon should start paying Level 3, no?

"Verizon, you're my enemy, I'm going to continue paying you every month while I work around your stupidity with my VPN..."

That's just how fucked up this situation is.

This is clearly a "dame pan y dime tonto" situation (“Dame pan y dime tonto” literally translates to “give me bread and call me fool”. The idiomatic meaning is “I don’t care what people say as long as I get what I want”.)

They have trained their agents on how to respond to complaints about speed... https://twitter.com/CyrisXD/status/489950468483731458/photo/...

The funny thing is even though the rep claims that speedtest.net is fake, the Verizon speedtest service at http://my.verizon.com/services/speedtest/ is powered by Ookla, the company behind speedtest.net

I do the same thing as the OP, I use an OpenWRT router with OpenVPN back to a VPS I have and it improves streaming video quality and speed for Netflix and more. Highly recommend it.

Wonder what location he tunneled from / to to get around Verizon's bottleneck.

Hi! OP here. I'm in NYC. I tunneled to Washington D.C. and that seemed to do it. I figured I'd try a location as close as possible first.

Tried the same thing on TWC from NYC with no difference in speed. Will experiment with other setups.

Also would like to note we pay for (and can get) 50 Mbps down. Netflix comes in at ~560 kbps and just stays there.

Cool. You may want to test different VPN services in case that's it. I've been using vyprvpn and I'm quite happy with it, but there are probably a few others that would work well.

It's not the location that matters, it's which uplink to Verizon a given VPN goes though.

If Verizon's recently released data is accurate, location probably doesn't make much of a difference. The key is that packets from the VPN provider to the user need to not arrive via the congested Level3 <-> Verizon interconnections.

If I were him, I would change ISPs.

To who? TWC? Comcast Xfinity?

Many Americans don't have that option. The physical wires either in your house or connecting your house to the street trunks are often owned by the telco themselves. That's right the copper wires in your house you own, might be property of Verizon or comcast. They get exclusivity to these wires.

Reply on news.ycombinator.com