Hacker News Reader

Comment by valas | Hacker News Reader

Why do you think it would drive down Netflix costs?

In old world there were a dozen of "tier1" providers who would all peer with each other more or less settlement free. It is conceivable that one of these providers (I think it was Cogent in this case) was offering "low rates" to take Netflix traffic and deliver that traffic to Comcast over their settlement free peering. As traffic flow between Cogent and Comcast was increasing, Comcast refused to upgrade the interconnect to accommodate the flow. Then: links get hot, traffic gets dropped, customers complain, Netflix+Cogent asks Comcast to behave, Comcast offers to unblock the pipes via direct Netflix<->Comcast peering for a "fee".

Now I don't know whether that "low rate" that Cogent was charging Netflix was lower than a "fee" that Comcast is charging. It might be that you are right and the fee overall reduced Netflix costs. What I do know however, is that Comcast is a monopoly and if unregulated, they can start extracting higher and higher rent from Netflix (or any other content provider for that sake).

View on HN

Peering agreements existed in the form they did because of the underlying assumption that traffic would be roughly symmetric. Services like Netflix that send massive amounts of traffic in a single direction violate this assumption.

This is one of the underreported issues in the whole net neutrality debate. In the late '90s I worked in the WAN engineering group doing capacity planning at a big regional CLEC, which actually was a Tier 1 backbone provider. It was widely assumed in the industry that the flat rate pricing model -- pay $X for Y "guaranteed" bandwidth -- was living on borrowed time and would be replaced by actual usage-based pricing, paying for packets like you pay for gallons of water. The flat rate model only worked because we generally could keep available capacity well ahead of demand. We could do that because we were adding customers like crazy, and customers sucking down high bandwidth were outliers. If those prerequisites ever changed we'd be screwed.

Well, here we are in 2014 and ISPs like Comcast are not adding customers like crazy anymore, because most of the people who want high-speed data from them already have it. But the customers they do have are, on a per capita basis, using much more bandwidth now than they were five years ago. And this is only going to get worse. The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever, but those requirements are no longer being driven by customer acquisition. Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.

But wait, doesn't Netflix pay for all their bandwidth already? Well, probably not: unless industry practices have changed drastically, even those backbone trunks are actually oversubscribed. If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. Netflix breaks that assumption.

I don't know quite what the solution here is, and I certainly don't want the kind of dystopian "the ISP only gives you access to web sites that you pay for" future the sharpest critics paint. But I don't see this as a step toward that dystopian future. What I see it as is the ISPs trying to figure out how to work usage into pricing models without saying "screw it" and actually adopting a usage-based pricing model.

If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7.

Whoa, whoa, whoa. This is exactly the assumption commercial customers paying for unmetered links are making, otherwise they wouldn't be paying for unmetered links. Not being oversubscribed is supposed to the entire point. If ISPs' business models depend on them being able to break their promises to customers paying for continuously saturated links that's their problem, but they're still on the hook for what they're being paid to provide.

I'm looking forward to chipotle_coyote response on this, but I don't think he's referring to the circuits that commercial customers pay for, but instead the ISP peering circuits - which are usually expected to run at much less utilization.

Isn't flat rate pricing completely orthogonal to net neutrality? Net neutrality doesn't mean you can't charge me for usage, it just means you can't charge me differently based on the kind of usage, as opposed to the volume.

I personally think the resistance to usage-based pricing in the tech community is really weird, although I mostly see it on the consumer end. Hard caps are bad, and caps that only apply to some services are really bad, but charging more if you use more is completely reasonable.

The idea is that usage-based pricing doesn't make sense for customers because there is a hard limit on consumption simply based on the terrible state of last mile connections. Simply put, charging 2014 dollars for usage on a line with a capacity that was last relevant in 1985 is not motivated by actual capacity limitations, but a simple money grab.

(Not to mention that there are no inherent variable costs to transporting bits. Water needs to be purified, electricity is obviously usage priced - but you can be pushing bits forever once you have the hardware setup.)

I think the resistance to usage-based billing is practical rather than economic. With a flat-fee scheme you can look at your bill and say, "yup, $89.95 is what I agreed to pay each month."

With usage-based billing, the amount will float up and down and it's hard to understand how the amount is calculated. Usage-based telephone bills are broken down by calls, which you can probably remember.("What's this 46 minute call to 555-1234? Oh yeah, I called Mom last week.")

With IP traffic, how do you itemize the charges? Temporally? I used to get cell phone bills that included pages and pages of stuff like "Jan 27, 4pm, 239789987345 bytes". It was completely useless.

How about reverse DNS? That has other problems. • You use BitTorrent? Here's 247 hosts with incomprehensible names on your bill. • You use a VPN? A whole bunch of traffic is lumped together. • "Honey, what's all this traffic to nastyporn.com?" • It puts pressure on web sites to optimize their deployment strategy for billing clarity

The "correct" way is probably to log everything, then provide analytics software. But I can't imagine my Mom generating reports just to understand her internet bill.

Economically, sure, usage-based billing is a great idea. Alignment of incentives and all that. But practically, it's a pain for everybody involved.

I think the analogy to water or electricity billing is actually a very good one.

My electricity bill shows only X kWh per month. There is no "itemization" of usage. However I can purchase watt meters that can monitor power anywhere (including the main grid connection) if I decide the bill is too high.

I can definitely imagine a similar service sitting on the router or provided by the provider that does bandwidth accounting to deliver a picture of what is going on. I'm sure there would be many third party device manufacturers willing to do this traffic accounting as well.

The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever [...] Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.

I dunno.. being profitable enough to buy time-warner cable for billions doesn't sound like a nightmare scenario from their POV. It sounds like they'd rather spend the money becoming a monopoly than improve their infrastructure.

You forgot to mention the part about how it actually does balance out because Comcast is receiving $18B billion a year for internet bandwidth from the its customers. Customers are requesting more data, but Comcast wants to be paid on both sides because they have a monopoly over the customer.

If it is really expensive to accept Netflix's offer of free hardware and free bandwidth then the additional cost should be passed to the customer. The problem is that it's not actually more expensive and Comcast knows their customers can't choose someone else so it's a much better strategy to squeeze the content providers.

Comcast is prepared to provide that bandwidth to its customers subject to it's fair use cap of 300 Gigabytes. What they aren't prepared to do (for better or worse) is install larger circuits to Cogent so that Cogent's customers have unblocked access to Comcast's customers.

Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast. This will probably set a very bad precedent for them when it comes to renew those agreements in 5-10 years, or however long they will be. Maybe they are hoping for some competition to enter the US environment by then.

> Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast.

How would that work? Comcast customers by and large can't switch to another broadband provider and Netflix customers can certainly switch to another video provider (Comcast!). It's not like this is a brand new issue, Netflix has held out for a long time. The Comcast / TWC merger is what finally did it, 30M combined customers is too big to risk on a public game of net neutrality chicken.

Surely they're still working on lobbying efforts to make it illegal for a company like Comcast to simultaneously have a monopoly on selling broadband and compete with services using said broadband. It's pretty obvious at this point that internet access is a utility and it should be treated as such.

Almost nowhere does Comcast actually have a monopoly on broadband.

http://nymag.com/daily/intelligencer/2014/02/why-comcasttime...

""" But there's a silver lining for people who hate this deal: By the government's own standards, the Comcast—Time Warner deal should be blocked. To understand why, you have to know that there's a thing in antitrust law called the Herfindahl-Hirschman Index, or HHI. In very basic terms, the HHI is a way to measure the concentration of a given market. And the HHI is the test the government's lawyers apply if they want to see whether a proposed merger would create monopoly-like conditions. """

A duopoly or oligopoly isn't much better. There's still a marked lack of choice for the customer, which results in lock-in and higher prices.

The Hacker News readership includes many people capable of starting ISPs. Comcast's large profits, which have been used to fund a string of high cost acquisitions, are a juicy target for potential competitors. Yet you never see discussion of ISP startups on Hacker News because the barriers to entry protect the current carriers. This wasn't the case in the 90s, and had Hacker News existed back then ISP startups would have appeared frequently.

The issue of telecoms not having competition has been the case for years. Back in the BBS days Jack Rickard used to harp on this in Boardwatch [1] magazine. I recall him saying something like, "Two guys with a screwdriver should be able to start a phone company."

[1] http://en.wikipedia.org/wiki/Boardwatch

> This wasn't the case in the 90s, and had Hacker News existed back then ISP startups would have appeared frequently.

Indeed. The late 90s / early 2000s were the golden age of the CLEC. There were certainly pain points (like getting copper pairs provisioned), but I remember not caring that my building didn't support Comcast Internet because I had an SDSL connection from Telocity and an ADSL connection from Speakeasy.

A friend of mine had a pretty nice BBS setup in the early-mid 90's... It took him a dozen requests and 2.5 years to get extra phone lines in. That was definitely a pain point.

I actually knew a lot of small-mid sized ISPs that started off as BBSes... Have a T1 split for phone + data, with several phone lines, and a data pipe upstream. I used to subscribe to Boardwatch magazine as well.

That said, most were either bought out, or went under within a couple years. It was a rough time for commercial SysOps in the mid-90's... not that I really feel bad for them. Times and environments change... The BBS and art scenes are still around today, but not nearly what they were in the early-mid 90's...

I miss it a lot... since most BBSes were based in a local area, there were a lot of get togethers, and you'd meet and talk to people you wouldn't necessarily do in today's internet, which is based in interest groups.

This is true if you have some antiquated definition of broadband that includes a competitor offering "up to" 896kbps while Comcast offers everything from 1 to 20+ mbps, but false for everybody else.

Question: If Comcast is such a monopoly that everyone here keeps repeating ad nauseam, why hasn't it been broken up ATT style yet?

AT&T was broken up into a bunch of smaller regional monopolies that did not compete with one another. (Almost a century after its founding, by the way, and decades of on-and-off litigation.)

The cable industry has evolved into a bunch of regional monopolies that do not compete with one another.

The effect of breaking Comcast up AT&T-style would be... a bunch of regional monopolies that do not compete with one another. And then, as with AT&T, proceed to consolidate via M&A into larger regional monopolies... that do not compete with one another. Same as now.

Welcome to the American telecommunications industry.

http://en.wikipedia.org/wiki/Regulatory_capture

That doesn't discount the fact that laying cable costs lots of money is a natural monopoly.

Wait, so in one post, you challenge HN's readership to prove that Comcast is a monopoly, and then two posts later you assert that cable is a natural monopoly. I'm confused what point you're trying to make.

Comcast is a natural monopoly, and there isn't anything wrong with that. Competitors are free right now to lay cabling to areas where Comcast already serves, they just choose not to because it isn't a smart financial decision.

Completely agree with you here. It's time that the United States realizes that it's approach towards broadband was flawed and tries something new.

What they aren't prepared to do (for better or worse) is install larger circuits to Cogent so that Cogent's customers have unblocked access to Comcast's customers.

That's totally backwards. Comcast's customers are paying for access to Cogent's customers, including Netflix.

That has never been how the internet works. You pay a part of the costs, the other party pay a part of the cost. When the other side opts for a cheap, poor quality connection (like Cogent), you suffer no matter how much you pay, and it's not your ISPs fault.

It's not Comcasts responsibility if Cogent did not have peering arrangement or had paid for sufficient capacity to handle their end of the connection, that their customer was paying them to provide.

If it was, I'd love to provide Netflix connection much cheaper than Cogent. I'll provide it over my 10Mbps uplink, and blame consumers ISPs when everything falls over.

It is Comcast's responsibility because I was paying them $1xx a month to take care of it. Bulk transit bandwidth is significantly cheaper than the price I was paying, so there's no justifying the claim that both sides have to pay.

Cool, let's be partners. I'll provide a service you want to access at 10Gbps, for an hour a week. I've got no connections to Comcast. I'll call up my brother, who runs a little ISP. I buy 10Gbps from him, and so have 5 other customers. You try to access my site. Packet loss ensues.

Since you've paid Comcast, they're now required to go to my brother's ISP and arrange more peering, right? Also, since you're paying, I shouldn't have to pay my brother, and he shouldn't have to pay anyone else, right?

Come to think of it, why isn't all hosting bandwidth free? Why do Azure and AWS charge for outbound bandwidth, but not inbound?

Sure, if your brother's ISP has a 50Gbps pipe at a common peering location, and 10,000 Comcast customers all want 5Mbps video streams from his customers, Comcast should be peering with him.

Let's bring this down to earth. You're confusing transit and peering connections.

Netflix pays level3 for transit service (you pay your brother). Comcast pays level3 (your brother) for transit service. Notice how level3 is neither the source nor the destination of the traffic.

Netflix's traffic increases from, for example, 500Gbps to 1Tbps. Comcast's level3 ports fill up because netflix increased in popularity.

In general networks that pass a lot of traffic like this end up peering directly and avoiding transit costs.

Comcast has two choices. They can either pay level3 for another 500Gbps or they can transition the full 1Tbps to free peering directly with netflix. And yes, I consider it Comcast's responsibility to do one of those two things with their subscriber revenue, which, incidentally, most other ISPs in the US are doing.

Instead they are choosing option 3 which is, leverage their local access monopoly to force netflix to pay them, and protect their cable and xfinity services.

Very, very well said.

Let's do some simple math to prove your point.

Assume Comcast has about 10 Tbps of traffic at peak, single-counting all traffic that transits AS 7922 (Comcast's national backbone network). [1] Now assume that a full 50% of that traffic is originated by Netflix, and the market price for transit is $2/mbps. The percentage is likely quite lower than that, and given 5 Tbps of traffic volume and coordination between the two networks to reduce the number of miles the bits need to be hauled by both networks, the per-megabit price is almost certainly quite a bit less.

But even with these obviously flawed guesses, we're only talking about $120mm/year in costs. Netflix's worldwide revenue in 2013 was $4.3b [2]; Comcast's 2013 revenue from their US HSI business alone was $10.3b. [3] If 50% of Comcast's $10.3b business was in jeopardy, don't you think they'd find a way to absorb that $120mm?

[1] http://as7922.peeringdb.com/

[2] Netflix Q4 2013 earnings release, http://ir.netflix.com/results.cfm

[3] Comcast 10-K filed 2/12/2014

(edited: formatting)

"If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. "

This may be the cases for ISP-ISP connections, but every transit circuit I've ever purchased as an end-user has two elements, a Non Reccurring Charge (NRC) to set up the connection, and then a monthly recurring charge (MRC) consisting of, a port speed, and a 95th percentile metered usage charge.

e.g. I might get a gigabit port set up for an up front $5000 NRC, and then a 24 month contract for $2000/month port speed, and then pay $5/megabit @95th percentile of 5 minute averages sampled over a month.

The ISP samples the usage of my port every 5 minutes, (8640 samples in a 30 day month), drops the top 5% (432 samples), and then charges me the cost of the next sample (sample #8208) - If the sampled speed at #8208 was 1 Gigabit, and sample #8207 was 0 kilobits, then I am still charged $2000/month + 1000 * $5 = $7000/MRC for that month.

Cogent has always had a reputation in the industry for being the cut-rate player (to the point of other ISPs just dropping them as peering connections, resulting in companies literally not being able to communicate with their customers) - Netflix was trying to rely on Cogent to send data to their customers, presumably because they were charging significantly less than "Tier-1" transit-providers, and finally came to the realization that yes, you do get what you pay for.

None of this surprises me, and none of it appears relevant to the conversation regarding Net-Neutrality - this is a peering discussion, pure and simple.

That's one way to look at it. The reason there even is such a thing as a cut-rate service like Cogent is that ISPs like Comcast have a captive audience and can increase their transit fees with impunity. The CDNs that go along with that extortion are the ones that don't get labeled "cut-rate."

You forgot to mention that network technology has leapfrogged last mile connections many many times over. In many ways, the problems have gotten considerably easier for many ISPs, and there are no capacity problems at big exchanges, and certainly no barriers to scaling up.

> Services like Netflix that send massive amounts of traffic in a single direction violate this assumption.

Netflix does not send any data. Comcast customers request lots of data from Netflix. Significant difference.

Another example (just in the other direction): A backup provider needs to pay up big-cable because their customers are uploading a lot of data?

Your second example is actually really fascinating - backup and photo providers get awesome deals on bandwidth because that part of the circuit is typically highly underutilized. So, ironically, it's a case where backup providers argue that their circuits should be charged at much less than market rates because the traffic is going in the opposite direction ,and their provider usually has circuits going idle in that direction.

I'm willing to wager that backup providers haven't run into any upload issues with Comcast for this reason.

DS3 line? What is this, 1996?

Each individual Comcast customer has in theory paid for a DS3 worth of bandwidth (modulo the "cap"), which should be a reality-check that some of your assumptions from that era need updating.

The defense (and it was always a thin one) of ratio clauses in peering agreements was hot potato routing. The ingress traffic is what you actually have to haul a long distance, so the agreement says "you must meet us everywhere we peer, and you must maintain ratio." Together, those rules mean each individual path will be asymmetric but the overall load on the WAN will be shared between the peers. It was always slightly silly, though.

Hot potato is now over. CDN's are desperate to do all the work and bring traffic as close to eyeballs as the monopolists will allow because the cost of actually building a network is irrelevant compared to the rent the monopolists are extracting. The monopolists are selling so-called "transit" to haul packets 1 mile. They're really charging for access to their captive eyeballs, not for moving bits. If you want to send a few megabits to Asia, fine, whatever. You can do it for the same price. They're not even mad, "glad we could make a deal," etc. The situation is nakedly broken and warped, and everybody actually in the business knows it.

Peering has always been a mix of market power, shaming, and political excuses ("ratio" was one). What we're now seeing is the shaming and excuses fall away thanks to apologist bloggers and weak politicians, and market power emerge as the only thing that matters. Compared to this power, traditional outdated notions of neutrality are irrelevant: there is no need to give traffic less "preferential" treatment when you can simply depeer them and then try to sell them so-called "transit" to go 1 mile. "Paid peering" == "well, how much money you got? Let's make a special deal just for you".

I used DS3 as an example. But no, even in 2014, you've only paid for a DS3 (or whatever) worth of bandwidth in theory. It's not just the last mile that's oversubscribed -- so are all the trunks leading up to it. If that's changed between then and now, I would lay pretty good money that it's changed to make things worse, not better.

Without meaning to express either approval or cynicism, market power has always been the only thing that matters in this business. But CDNs have paid big ISPs special rates for years, because they really do put special strains on infrastructure. And my impression remains that that's what this particular battle is ultimately about.

you are some 5 yrs late... ISPs already put usage in the contracts.

The contention was that the change in traffic caused by the Netflix flows put Comcast & Cogent's peering out of balance.

Generally speaking settlement free peering requires that the arrangement be mutually beneficial. If the traffic exchange becomes inequitable then the side that disproportionately benefits would need to pay to cover it.

In this case Cogent was sending vastly more traffic towards Comcast than they were receiving (because of Netflix) so they were the misbehaving party.

Consumer ISPs always have received more traffic than they send, it's the nature of the service they provide. I'm not sure that the settlement free peering model used by upstream providers makes sense when applied to ISPs.

I'll admit my knowledge of peering is minimal, but consumer ISPs have always been something of an anomaly if memory serves. The traffic flow is highly one sided, but peering with consumer ISPs is obviously necessary if you want to actually reach consumers.

Yep, that's loosely accurate for consumer providers but Comcast does more traditional business traffic that one would think.

For example, they are the 30th largest provider globally going by the CADIA rankings (http://as-rank.caida.org/?mode0=as-info&mode1=as-table&as=79...) and have large traffic sources as customers. From the CADIA list this includes Peer1, Dreamhost, SoftLayer, Hivelocity, and Liquid Web.

As a eyeball ISP I occasionally get offered cheap transit from hosters looking to balance their traffic ratios. From the CADIA list it seems like Comcast may be doing the opposite.

I work in a totally different environment -- a very large enterprise IT environment with 200k+ users and probably 2,000 physical locations.

When we do business with cloud providers or other third parties, we usually start with internet based access. When the relationship gets bigger or needs to serve a large portion of our base, we typically either peer with that provider (in the case of big cloud providers like Amazon, Microsoft, etc) or establish dedicated connectivity between our networks.

I think Comcast, Verizon, TWC, etc has a point here -- as a relationship grows with another party, you should have a more robust connection. Netflix hasn't wanted to do that -- it prefers to use a shitty ISP (Cogent) for cheap, and augment it by co-locating content on end-user ISP networks for free. Services like Netflix and Youtube stretch the net-neutrality argument, because they aren't good citizens.

That said, the end-user ISPs cannot be allowed to discriminate, which their monopoly power will almost certainly enable them to do.

I agree. I am not arguing that Comcast should be allowed to go unregulated either. I strongly support net neutrality and do not want to see it thrown under the bus.

Assuming it wasn't harmed is also speculation and we should definitely be concerned about the deal and what it means for net neutrality.

Reply on news.ycombinator.com