Hacker News Reader

Comment by sliverstorm | Hacker News Reader

What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced network.

Then Netflix comes along, and suddenly 35% of Comcast's customer traffic is all going to a handful of Netflix DSNs? Suddenly Comcast has to redo the whole connection. They have to buy top-shelf hardware just for Netflix.

I don't know if this has happened, but it's an interesting thought experiment to see how "just one company" could place disproportionate burden/cost on the ISP, as compared to the case where traffic travels to & from many different entities.

View on HN

It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude.

And then there's this: http://arstechnica.com/information-technology/2014/04/bittor...

"[W]hen we ask [ISPs] if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download—thus filling their upstream networks and nearly doubling our total traffic—there is an uncomfortable silence," Netflix CEO Reed Hastings wrote last month. "That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs."

> It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude.

That's because around the time Comcast was building up their networks customer demand was wildly asymmetrical. People simply download far more than they upload and Comcast wisely dedicated more of it's bandwidth to downloads.

I got the data in the following paragraph from a talk on the subject at an Outerz0ne or maybe a Phreaknic years ago:

At the time all cable companies were were building up their data network, they were building that network over top of their existing video distribution network. That network was designed [ages prior to this point] to have tiny, tiny, upstream channels for use by system control and maintenance devices, and huge downstream channels to shove 100+ analog TV channels. Because it would have cost a metric shitload to replace all of that hardware, the cable companies were forced to give their data subscribers pretty large downstream links and relatively meager upstream links.[0]

Frankly, the nerds of the day (remember, this is the mid-to-late 1990's) would all have been running home servers, and would have been demanding a symmetric connection so that they would have a substantially cheaper alternative to ISDN.

(Note that nerds that remember those days consider asymmetric home Internet connections to have been more harmful than the Eternal September.)

[0] Note that this property is pretty meaningless in an HFC network where the cable company could run fiber directly to the home, but chooses not to. [See also ATT U-Verse.]

But then doesn't that mean that it's also easier to put a cache in front of those overused switches and solve the problem much more easily?

I guess the core issue here is that, from a customer's point of view, the ISP is selling me inet access, regardless of where the data actually goes, so it is the ISP's responsibility to keep their end of the bargain (to provide inet access). As a customer I should not care how is their topology planned, what kind of equipment they use, etc.

So basically the ISP though that they could offer T amount of bandwidth, advertised it as U (with U > T) because most customer won't use even close to U, and now that more and more users are actually using U amount of bandwidth (or closer to it) the ISP's infrastructure is suffering because they simply where not prepared for it, i.e. they oversold their capacity.

OT: By the way, why downvote the parent? I really thought downvoting is for when the post is off-topic or offensive or something... but more and more I see HN modders downvote because they disagree... really? so you make the post go into an ureadable color (grey) just because you disagree? I don't think it's supposed to be that way, is it?

Good point about U > T. It also happens in Gym business where they oversell their gym capacity betting that 80% of their customers will be too lazy to come exercise everyday. Now imagine if I sell a motivation pill that makes you go to gym everyday. It would be absurd if Gym companies send me a cease and desist letter because i am overburdening their facilities because everyone who takes my pill wants to use their gyms everyday.

Actually, now that I think of it, we could probably name a lot of other services that rely on this business model and do a thought experiment like the gym one.

An airline for example; now everyone actually caches their flight because Google's self-driving cars always make it on time, and then they demand that Google stops bringing passengers on time?

Or a buffet restaurant, since now everyone goes to the gym, they can actually eat a lot more, thus restaurants complain about, either the gym or the pill maker?

I don't know, it would be a strange world to live in if this happened in other industries. Maybe prices would rise too much?

Playing the devil's advocate: what if the higher contention ratios lead to 50% reduction in gym membership fee? All other gyms follow suit and consumers and gyms both benefit from the realization that not all members will be in the gym at the same time. Marketing departments start vying for consumers who they know are unlikely to attend everyday and lure them with annual sign up discounts.

Fast forward 3 years later, you come out with your pill and throw a wrench in the works. Now gyms realize they have to go back to the old pricing model because the economics have changed. Consumers will no doubt become upset at the prospec of having to pay more. Some gyms take a wait and see approach to see what other gyms will do. Your pill started it but really, it's not your fault or your problem.

What should the gyms do?

>>What should the gyms do?

Be Honest. They can invent a new business model based on transparency. Imagine if price is not fixed but you are charged per use based on the traffic at the moment. It could be similar to what Uber does, price based on demand.

In terms of ISPs, I think there should be a micro transaction exchange where I can buy X minutes of data transfer. If Verizon is clogged I switch my router to connect with ATT etc. All networks should create a single broker device thats given to each consumer.

Netflix even makes it relatively easy to do this as an ISP too

https://www.netflix.com/openconnect

Netflix isn't "doing" anything to Comcast. Comcast's customers are demanding (and paying for) the asymmetrical traffic. If they have to redo the connection, so be it -- they promised their customers connectivity, and if that means installing new hardware to achieve connectivity so be it.

Edit: rewritten because I misunderstood your point.

Funny, I actually just added an edit to my post above that discusses this. Comcast adds interconnect capacity as needed. So in at least one way, it's actually easier this way, right? Just one peer to make sure they have capacity for.

But either way, it's Comcast's job to run a network that's optimized for its customers needs, not some abstract concept of fairness. And in this hypothetical, it's not on Netflix that streaming lots of content is not compatible with how Comcast happened to build its network. Comcast needs to fix its network so that its customers can get their data.

I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.

> just a small handful of nodes

But Netflix is not "just a small number of nodes". They're basically their own CDN; they have servers all over the Internet, precisely in order to not have the problem you describe.

What's more, Comcast has a deal with Netflix to directly connect Netflix nodes to Comcast's network; in other words, Comcast directly controls how many paths there are for Netflix content to get into its network. If there aren't enough such paths, they can just open up more.

Thankfully netflix expose their stuff via bgp (as well as other nice things) so you can have multiple paths into their network (and out of yours) without any single giant router. allowing you to balance your traffic.

You can even ask them for a freeish box to run inside your network to reduce requests to their CDNs

Oops, my bad. Rewrote my response.

What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced network.

Consumer ISPs have never expected symmetric bandwidth. If they had, they would sell us 50/50 connections instead of 50/10 or 50/2.

Note: It is definitely not built under that premise, hence why your consumer bandwidth is asymmetrical...

The Tier 1 always wants to ask for more bits than a lower tier operator. If a lower tier operator asks for more bits and tilts the traffic balance, things get interesting. Cogent is not a tier 1 but slings 35% of the internet around at will.

It does cost a lot of money to route Netflix's bits and the narrative is hugely helpful to Netflix. I'm not saying this is correct, I'm simply trying to point out that Netflix has a ton to gain from carriers upgrading their network connections and routers.

Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source.

Netflix is demanding huge network upgrades without any additional revenue source.

I think a lot of us would have more sympathy with the ISPs' situation if they hadn't consistently and for a very long time been promoting their services as ever more valuable by offering ever higher bandwidth, yet keeping quiet about little things like massive over-subscription and high contention ratios that meant if all their users actually tried to use the service that was purportedly being offered then no-one would get anywhere near the advertised speeds they were paying for.

With the rise of things like streaming video services and Internet telephony, some people are actually trying to pull significant volumes of data down the pipes relative to the advertised size of pipe available. Everyone's known this was coming for a long time, and no-one more so than the ISPs. The way I see it, if the current situation hurts ISPs that failed to invest or set a realistic pricing model, they'll just have to take their medicine and offer a more realistic deal to customers next time (or fail because they can't compete effectively).

If Netflix closed its doors today, presumably Amazon Prime and Hulu would pick up the slack. The problem is not Netflix providing content; the problem is users consuming it. Users want better service than the ISPs are inclined to provide.

> Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source.

Sounds to me like just another way of rephrasing "ISP Customers are demanding huge network upgrades without any additional revenue source."

Netflix does not send packets to you unless you request them.

silverstorm didn't say anything about symmetrical. If Comcast assumes that the various interconnect points are relatively even in traffic usage, but then the few that go to Netflix start getting congested (leaving the rest relatively light), they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic.

Edit: not sure why I'm getting downvotes since silverstorm clarified exactly what I wrote

> they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic

You're missing a key element of the situation: Comcast made a deal with Netflix precisely to address this problem. Netflix now has direct connections to Comcast's network, so Netflix traffic does not have to go through a third party. In other words, Comcast now directly controls how they connect to Netflix, how many connection points there are, what the average network distance is between the closest Netflix connection point and their customers, etc. So they have a much cheaper way to even out Netflix traffic: open up more connection points.

(Which, btw, AFAIK they already did as part of the deal; they certainly did not come out of the deal having "just a few spots" where they are connected to Netflix. Netflix basically is their own CDN; they have servers with their content on them all over the Internet. So Comcast has all kinds of ways to shorten the average network distance between each of their customers and the closest Netflix endpoint.)

Asymmetrical peering has been the rule since the 90s. Think of it as fan-in and fan-out, each of which has different dimensions.

Reply on news.ycombinator.com