The Case Against ISP Tolls
blog.netflix.com
blog.netflix.com
"Your video is buffering because Comcast is slowing your connection down. Click here to contact Comcast customer support. Click here to find Netflix Certified providers in your area."
Netflix and Google and Amazon and Apple all have great brands with high customer satisfaction. They should utilize all those eyeballs to funnel even more hate toward Comcast.
The two networks are directly connected now, which eliminates that bottleneck. That's why performance spiked. There wasn't a capacity issue on either side of that Cogent link; Netflix is willing to pay for its bits, and Comcast's network has adequate capacity to deliver them. This routing change actually occurred two days before the deal was announced; you could see through a traceroute already that the route between Netflix and Comcast had shortened and no longer went through Cogent.
Courtesy Ars Technica on February 21, two days before WSJ reported the deal: http://i.imgur.com/pZPVTeR.png
Why would they want to pay Comcast instead of Cogent? Because Comcast's fight with Cogent was resulting in poor performance for Netflix customers. They had the option of waiting for it to get bad enough that one side or the other gave in on the negotiations and increased capacity, meanwhile losing customers frustrated with buffering at peak hours, or choosing an alternate route to get their bits to those customers.
It also nicely sets up barriers for entry to new video streaming services, which is something maturing companies tend to do. Netflix has grown big enough to buy transit directly from ISPs instead of through middle-men like Cogent, while startups don't have the capital for that.
Netflix was paying Cogent to link it to the internet. Now Netflix's servers are directly linked to Comcast, so Comcast is connecting it to the internet, and that's who they're paying instead.
Comcast is able to fight with Cogent while their own customer's connectivity suffers precisely because the customers often don't have an alternative ISP. Netflix argues that if the TWC merger proceeds the situation will get even worse.
Cogent's thing is selling massively cut-rate transit. They've been involved in constant peering disputes over their method of doing so since at least 2002. This isn't new for them - Netflix is just big enough that people really took notice this time.
That doesn't make sense to me. Comcast's customer isn't Cogent; Comcast's customer is the end user who sends them a check every month for access to the internet, regardless of where it comes from. If Cogent is being paid by Netflix on the other side, then Cogent and Comcast are exactly aligned (each wanting to deliver the Netflix packets because their respective customers are paying them to) and should peer without any money changing hands. If Comcast wants Cogent to pay it so that Comcast's own subscribers can get access to Cogent's customers' content, then they're being shady.
Should Comcast cache my personal website for free so that it's super fast half way across the country? Or am I responsible for making my website faster (by using CDNs and such)? If it's the latter, why should Netflix not have the same experience I have?
[1] http://www.forbes.com/sites/ericsavitz/2012/06/05/netflix-sh...
It's not transit. They're not delivering your traffic to some other ISP. They're delivering it to their own customers, which is what those customers have already paid them to do.
Netflix wants to distribute it's costs back onto everyone else (everyone but themselves). Even though this new arrangement reduces their costs vs. using Cogent.
That's messy and Comcast should fix how it distributes the cost of its customers using the internet, but what does it have to do with Netflix paying for anything? Similarly, you don't pay Comcast to route the packets from your website to your visitors, even though only a subset of them use your site, thus asking everyone else to take on those costs. How is this different?
Any plan where I don't directly pay per byte I download seems destined to run into problems like these - Comcast doesn't want to deal with the increased traffic Netflix is pulling into their network, but really what are there customers paying for but that traffic?
A better solution to what? There isn't a problem to be solved here. There is no danger of Comcast becoming insolvent as a result of Netflix traffic going to Comcast customers. The threat to Comcast of people using too much Netflix has nothing to do with Netflix using too much bandwidth and everything to do with Netflix taking too many customers away from Cable TV. Comcast doesn't seem to have any trouble with bandwidth consumption from people watching their on-demand video services.
The idea that there exists some large collection of people who would prefer metered billing is nothing but a talking point in favor of a policy that damages the ISPs' video service competitors. AT&T did studies in the 1970s that showed that people prefer unmetered plans over usage-based billing even when usage-based billing would save them money.[1] Because it's not worth $10/month to have to incur the mental cost of having to constantly measure your usage.
[1] http://arstechnica.com/tech-policy/2012/04/why-we-should-wor...
This new arrangement improves performance vs. using Cogent. And not because Cogent has a slow network.
Followed by:
"I'm sorry, there are no Netflix Certified providers in your area."
When Comcast is rated as one of the worst companies for customer satisfaction, customers aren't staying with them because they are ignorant when it comes to the alternatives. They are staying with them because there are no alternatives.
[1] 30% of Americans can't choose their ISP http://www.extremetech.com/internet/178465-woe-is-isp-30-of-...
And it should be treated as such.
Humor me for a second and get past the fact that such a law would benefit cable companies. Is it a bad idea on its own merits? I don't think so. Municipal broadband would crowd out private investment in broadband. Given the state of other municipal infrastructure, there's a high probability it would be underfunded and low quality. Would that be a better state of affairs? Being stuck with an overloaded municipal system that never received enough funding because voters don't really care enough allocate a lot of money maintaining and upgrading it, that at the same time makes it impractical for private systems to operate?
I don't think the solution is to allow municipalities to do more things that undermine competition. The solution is to deregulate at the state level. Force municipalities to make the kind of "hands off" deal Google demands in return for rolling out Google Fiber to any company willing to make the investment.
This is really foolish. A company that wants to be a monopoly will provide fantastic service, but a company that has succeeded in establishing a monopoly no longer has any motivation to treat you as anything other than property. Why would any rational citizen let their municipality make such a deal?
(Q1) Regulators should encourage competition? True/False
(Q2) All government services are monopolies? True/False.
(Q3) All municipal services are either (utilities, monopolies)? True/False.
These are all absurd formulations.
Public Services are a thing. Some examples:
Ex 1> My public library is an ISP.
Ex 2> My public library is Tax Filing Center.
Ex 3> My public library is a copyright depository.
Ex 4> My public libary is a copy/fax location.
Competitors:
Ex1'> Comcast
Ex2'> TurboTax
Ex3'> Netflix
Ex4'> Staples
I don't think the solution is to allow municipalities to do more things that undermine competition.
Again, this is absurd. Public services do not undermine competition by implication. This formulation is akin to saying: "a good competitor undermines competition by being good." Or, "a new competitor undermines the competition, by entering the competition."... Um, no.
This is not to say single payer healthcare would not change the business/policy landscape in the USA.
But to even suggest that any municipal provisioning of communications services (public radio, public TV, public internet access) would be aking to "single payer healthcare" in scale/scope/completeness etc is simpley a rhetorical exaggeration.
I think its interesting to debate the pitfalls here, butnot with such extreme poles as a starting point.
Limited scope incremental services are fine, IMHO. Feasible, cost-effective, and market & competition extending.
And the existence of national parks does not make the government a "land monopolist" or what not. =D
The rest of the stuff about "fair" x or y is not really relevant. Markets are supposed to be "free" and not "rigged", but neither of these two concepts imply "fair" in the sense you are using it. Every company on the NYSE hasi its own cost of capital--is that 'fair'? no. its not fair to the owners. But that does not imply that the market is not free for capiral or that the market for capital is 'rigged'.
There are plenty of ways that government services can be provided to extend markets geographic reach, variety, and product depth without getting into some nasty discussion of socialism/totalitarianism etc. There are some services that are more-efficiently organized through the government. In that respect, a government entity is just like a non-profit that competes with a for-profit or private foundation entity (eg universities). And more ...
And "fair" absolutely is relevant, from a market-wide POV. Of COURSE some companies have natural advantages over others. That's great. They should win. When the better competitors DON'T win, you have a market fairness problem. If government is providing a service at higher cost to the taxpayer than if a private company was providing it, then there is a problem. And this happens all the time, because the sticker price of government services is always low, or zero, even though the real cost can be very high.
Furthermore, I don't think it's controversial to say that municipal services tend to be least common denominator affairs. Again, look at schools. I might personally be willing to pay twice as much property taxes to get well-funded schools that hire well-paid teachers, but good luck getting 51% of voters to agree to such tax increases.
Look also at power, water, and sewer utilities. Why are they so badly underfunded, and in a continual state of life support? As a yuppie, I'd be happy if my sewer bill went up by 500% (from $10/month to $50/month) if that meant a high-quality sewer system that didn't dump raw sewage into the river when it rained. I'd be happy to pay a lot more for water that wasn't flowing through old leaded pipes. But good luck getting any of that through a rate setting board!
Public utilities are under political control. Depending on the industry, this can be a good thing or a bad thing. I think for telecom, it's not a good thing, in that it won't lead to an equilibrium that I think techies would be happy with. Say you wanted to build municipal fiber in San Francisco. What's the body politic that will control that system? Do you think it'll be techies? Of course not. It'll be the folks that are out there picketing Google. The discussion will shift from provisioning high speed service to figuring out how to subsidize low-income and high-cost people. Or, take Seattle, which is not very dense and is quite sprawling. Do you think the political consensus will lead to installing fiber in the high-density, high-income, low-cost areas first?
It's also interesting you use public libraries as an example. Municipalities all over the country are cutting library budgets to the bone. Why? Because people don't care about them. I think what you'll find is that municipal broadband will be in the same bucket. People won't care enough about it to appropriate the substantial funds that will be necessary to maintain and continually upgrade the network.
If you want to treat broadband as a utility, you have to pay for it as such. Which means metered billing, which I don't think anyone wants.
Seems like a natural place to put "click here to contact your state/federal representative"
While I grant that cynisim is definitely warranted, at the end of the day, these companies have monopoly jurisdiction because we the people allow them to.
"Sorry but we need to tear up this highway and disrupt traffic for the next couple of weeks to lay some fibre. I know someone else did the same thing last week but they were another company and it's illegal for the local government body to stop any of us laying stuff on public land when we want"
Obviously that extreme is untenable. You may be after something a bit more useful like unbundling, but in the US you have to explain how forcing a private company to let competitors use it's cable is not outright stalinistic communism.
Obviously if they wanted to run wire through a house they'd have to get the owner's permission, or the local government's permission to dig up a road. The thing is, as the law currently stands they're forbidden from doing so; even if every household in they city signed a contract saying "sure, let company X run cable through our yard", company X still wouldn't be allowed to sell them internet over that cable.
Maybe a company could innovate and develop some form of fast wireless mesh network. But even if they had the technology and finances to do so, local laws stating "the only company allowed to sell fast internet here is Comcast" or equivalent prevent them from doing so.
The first is that a legislature can't easily bind its future self - laws passed by a majority can be repealed by a majority or circumvented by a majority in the future. So even if we pass Law 2014-003 saying what you said, there's nothing stopping a 2015 (or 2020 or whenever) law from saying "Law 2014-003 is hereby repealed" or more narrowly "Law 2014-003 doesn't apply to this law". There are some edge cases where you can pull it off but by and large it'd take a state/federal constitutional amendment.
The second issue is that the federal government can only regulate "interstate commerce". While you'd think that anything involving the Internet would automatically apply there, you couldn't stop a state law that covered only a local business dealing with in-state customers. So such a law could be used as a shield for Verizon/Comcast/etc while state laws restricting ISPs would only apply to smaller in-state competitors. My parents and a number of my friends are served by a local Wireless ISP because that's the only alternative in their neck of the (literal) woods. Under your system, their ISP could be subject to innumerable regulations that Verizon/Comcast would be immune to.
Why not require something similar?
What it boils down to in most places is that internet access is not (legally) a public utility and therefore skirts public utility regulation.
How does that solve anything, though?
Judging by the lack of competition, it seems it's not profitable to compete.
Now, you could force Comcast to lease out its network capacity to competing companies, but this is really a shortsighted solution: the consequence is that potential competitors, who are contemplating establishing competing last-mile connectivity, cannot make a solid calculation to see if it would be profitable, because it might risk being forced to lease out the network capacity it establishes, thus it avoids doing so.
The problem here is simple: digging down cables is expensive. No amount of legislation is going to change that. The real solution is reducing the cost of the last mile connection. This might be possible through opening up legislation (removing barriers to entry), though. Like opening up more wireless frequencies, to carrier last mile traffic to consumers.
Nobody is allowed to compete. No company will be afforded the same benefits given to the first cable company that actually layed the physical infrastructure. Local governments give all kinds of benefits to the first company to bring in cable. They won't for the second. They will see it as wasteful duplication of infrastructure.
Companies should not be granted monopolies on the last mile lines that they were granted all kinds of exceptions for. Telephone companies seem to get by without it, why are we tolerating it for cable companies, especially when they are competing with services offered online?
Comcast is holding all the cards.
In truth, it's both. It's a problem where a company has co-opted well-meaning regulation to snuff out competition, which is a thing capitalist companies do. Meanwhile, citizens point the finger at each other instead of removing Comcast's ability to monopolize.
Either the regulations that make competing as an ISP difficult need to be stripped out (my personal favorite), or the last mile needs to be public property. Either way, Comcast getting to be the only one sitting on that last-mile connection is what has given rise to its stranglehold on the internet.
We need to support fixing this on a local/state level. Support smaller ISPs in their attempts to disrupt the incumbents, and support legislation that actually encourages innovation in the marketplace (even if that means striking policies from the law books).
May I suggest a link in your profile?
(And, if I'm ever successful and allowed to dream, Seattle would be my next target)
Thanks for the suggestion, I updated my profile.
I fully support all alternatives to Comcast.
(Should I have acknowledged my bias?)
The process so far has involved :
- Figure out how to get access to right of ways [Completed] - Figure out how to get internet access to sell [Completed] - Figure out how much it will cost to start a CO [Mostly Completed] - Acquire a place to put a CO [Completed] - Acquire high quality, redundant backhaul [Under Construction Now]
- Figure out who's willing to pay to switch away from Comcast [I have some names, but this is definitely not completed] - Beg them for money [Some have subscribed, but definitely not enough yet]
There's a couple of big things that work against me:
City + State taxes for right of ways are pricy. We're talking multiple thousands of dollars per mile per year, on every single mile, just in taxes. (And I'm lucky, my city still owns a lot of their own right of ways. A lot of cities just wholesale gave them away to AT&T/Comcast).
Backhaul is expensive. Most ISP's are monopolies, so they can safely buy in scale. I have zero scale, so I pay full retail for everything. This is obviously very pricy. (I'm guessing I pay at least 3 times more, per Mbps, than Charter pays for backhaul, and probably 4 - 6+ times more than Comcast pays)
Fiber lines are dirt cheap. Install is expensive. Easily 95% of the cost of the network is that initial installation, and it can run into past the million dollar mark even in the smallest of neighborhoods.
If you want more info, let me know. I'd happily talk more about this, and provide solid numbers.
It wouldn't surprise me if speed/pricing like that existed around major network hubs (like Chicago).
I'm paying $8/Mbps. If I were to afford five times more backhaul than I have now, that price eventually drops to about $3.8/Mbps.
Places like Charter and Comcast pay less for their backhaul, in part because they are their backhaul, in part because they are large enough to negotiate their own peering agreements.
Recently, that's swung even more in their favour. Netflix, for instance, pays Comcast money for the "privilege" of paying the costs for all of the backhaul Netflix uses over Comcasts lines.
People sometimes call this "double dipping". But it's actually "triple dipping". Comcast charges subscribers money, Comcast makes Netflix buy the backhaul to Comcast, and Comcast charges Netflix again, to allow them to connect the backhaul they already bought, to Comcast's network.
Makes sense though, I'm sure if you had to get the fiber to your location that would easily push the price up a ton.
Seattle is one of my favourite places. Very expensive though.
If this somehow ever becomes successful in Michigan, Seattle is the very next place I would head.
It seems to be working. Seattle's new mayor recently talked about how the city needs to make it easier to build out internet infrastructure.
http://murray.seattle.gov/murray-seattle-must-be-a-national-...
> Another possible solution includes granting internet companies access to utility poles at little or no charge, so that building more infrastructure is not cost prohibitive.
> We are considering [changing policies] which make it nearly impossible for internet providers to expand existing services without an unusually high super majority of support from neighbors.
Here's hoping that G thoroughly topples every last one of them.
Google Fiber is a contended service, just like every other retail ISP. They're not doing anything particularly different.
Here's the average transfer rate on their 1 Gbps headline product:
http://www.speedtest.net/isp/google-fiber
Yes, 190 Mbps. Fast, but still contended.
There were fights at the municipal level too, but by that point, only a few medium-sized places like Hull in the UK, and Rochester, New York had phone companies that weren't owned by NTC or AT&T.
All of this seems strikingly familiar today, except that there seem to be a lot fewer politicians willing to propose any of the real profound changes like the ones that came a century ago.
Which regulations are we talking about here? This seems like really low hanging fruit, but of course it probably isn't.
Broadcasters get a government monopoly on a finite resource. Broadcasters don't provide a good signal to all customers. Some customers want to use Aereo (that's not the only reason, but a real reason) instead of paying for cable.
But no, the broadcasters want to be paid for not-providing a signal to those customers. Instead of investing to improve their infrastructure, they want to be paid for not improving it. With a straight face.
Of course it's especially ridiculous in the case where it's Comcast NBC, and they are also an ISP. If Aereo wins at SCOTUS, and grows big enough, Comcast NBC can get their pound of flesh as an ISP like they do with Netflix.
It is all well and truly fscked.
This is the best one paragraph summary of the Aereo case I've heard. Certainly makes it absolutely plain why it needs to be allowed to continue.
However, even on the slowest plan you can still watch youtube. The only difference between the plans is the speed.
http://finance.yahoo.com/news/netflix-switched-p2p-video-str...
It's going to take a lot to fight the likes of Comcast in this one, which means Netflix will also need support from the community - so Netflix, please stop it with the boneheaded DRM push on the web. Guess what, if you are going to use P2P, you're going to need a plugin or native app anyway. So just stop trying to corrupt W3C already in a pointless move that can't and won't stop piracy anyway (which I'm sure you already know).
https://www.freepress.net/sites/default/files/resources/Comc...
"Facebook is a little slow because my ISP might be throttling, but I'm not sure, it could also be my computer"
"I cannot access Facebook at all because Facebook blocked my ISP"
Getting users pissed off at web services is not a good way to effect change. We need users on our side, so we can't have them think of us as crybabies.
>> "In this way, Comcast is double dipping by getting both its subscribers and Internet content providers to pay for access to each other."
..and this:
>> "Put simply, there is one and only one way to reach Comcast’s subscribers at the last mile: Comcast."
...should be illegal. This really torques me. This smacks of how AT&T acted in the 80's when I was in my 20's and starting out on my own and how Microsoft acted in the 90's before they got busted, which is what lead me to Linux.
And Facebook is the only way to reach Facebook's customers. Should that be illegal?
It's a large ISP here, and it throttles access to youtube (and probably other content providers). It has been so for years. For some reason, there is no public awareness about that, people just say "my internet is not working well".
It's an act of aggression against their customers, to me, since all ISPs are marketing their offers around connection speed. And it's violating net neutrality, too. I don't get how it does not become a major PR problem for ISPs doing that.
That's the reason why I've switched to the new ISP services from ovh, though, as they made net neutrality their first marketing point (but for the same price, you don't have tv, if that matters).
How much did Netflix's Cogent bill go down? How much of the "work" that Cogent was doing is Netflix now doing themselves? How many hops of the traceroute are owned by Netflix, and how many hops by Comcast?
When I rent a rack in One Wilshire in LA, CA, it comes with power, cooling, and a network link. When I pay let's say $1000/mo for that 1Gbps link, it's not just the link from my rack to their router that I'm paying for. And when I send 1Gbps of packets up through that link, it's not just CoreSite at One Wilshire carrying the cost of delivering those packets.
At some point, yes, I do expect Comcast to get their "fair share" of that $1k/mo. If Comcast is PAID zero dollars for delivering that 1Gbps, then I want this free bandwidth gravy train to propagate all the way back to me. Comcast, please sign me up for 10 of those free 10Gbe links you're giving out, ok? Just tell me where to rent space where I can get those free 10Gbe cross connects to your network.
But then, you can also flip the perspective and ask the question, why doesn't Comcast have to pay for all their bandwidth? Shouldn't they be paying $1,000/mo for their Gbe's?
So why does bandwidth cost money, or more specifically when should bandwidth cost money? Price, as always, is just a means of efficiently allocating scarce resources. Price can only possibly be zero if there is no scarcity. Is it possible there's no scarcity of bandwidth on Comcast's network? Now that would be comical!
I think the answer lies partially in another development. Ask Netflix to chart the cost in terms of Watts and Rack Units to deliver 1Gbps of video streams. My theory would be the cost in terms of space and electricity to generate 10Gbe of video streams has dropped orders of magnitude. This means bandwidth will be an increasing portion of your bill no matter what. It also puts more pressure on scarcity of bandwidth when the ease of which you can generate 10Gbps of useful traffic is down to just a few servers.
Comcast gets paid $50/month per subscriber for that bandwidth. It costs Comcast less money to peer with Netflix than to get it from cogent. Comcast wants Netflix to pay to solve a problem that Comcast's customers have.
> Is it possible there's no scarcity of bandwidth on Comcast's network?
Absolutely! In fact, that's part of what the chart shows; when Netflix peered with Comcast, the video quality (which is a proxy for bitrate to the consumer) went up significantly, which indicates that there was no bottleneck on the Comcast network.
> Price ... is just a means of efficiently allocating scarce resources.
Not in the case of a monopoly. Comcast has a monopoly on a large fraction of the broadband market. They also compete directly with Netflix, so they will leverage that monopoly to make it as expensive as possible for Netflix to continue to compete with them. This is Comcast creating an artificial scarcity of paths from Netflix to Comcast's customers.
On another note, I have referred to Comcast subscribers as "customers" above; if agreements like the above continue, then Comcast subscribers are no longer customers, but rather the product that Comcast charges for efficient access to.
How does it costs Comcast less money to peer with Netflix than to get it from Cogent? Does Comcast pay Cogent too?
Of course the whole confrontation was manufactured, I don't think Netflix was ever arguing that Comcast needed to boost its entire buildout just to support their video streams, and by the way they shouldn't pay a cent. Comcast doesn't have to wait till its pipe are full before controlling traffic flows.
Netflix should be paying Comcast ZERO!
There's the rub. As a cable company, Comcast are not just selling internet access, they're also selling access to TV and Movies, infact historically this is all they sold and they're still stuck in this mindset. Netflix threaten to disrupt this lucrative business.
Everyone knows cable make money from bundling a whole bunch of rubbish channels together with a handful of high quality channels and charging a lot for it. As soon as someone (Netflix) offers to let you just watch the good stuff, for less dollar, Comcast's profit line is going to take a hit. It's no coincidence that it's Netflix who has to pay extra whilst the non-threatening Google/YouTube/Facebook/Porn get past without hassle.
Spot on.
You can stream video with basically nginx serving static files, you just need a 10G network card with TSO.
the CPU power needed is an order of magnitude lower than what you need to run full stack server side frameworks.
Just make sure the content fits in your page cache (RAM is cheap), and you will fill the pipe.
In the upcoming month, you will be able to do this with an ARM SOC sitting on a 10cm^2 board, for 5W TDP.
If we consumers have to pay to access to content why shouldn't ISPs?
Why is what? Are they any different? Are you balking at the word "drone"? How about "unmanned aircraft", which describes both Google and Facebook's approach? It's not like Facebook will be using the same "drones" the Air Force uses to bomb ground targets.
Hope Google provides the service as I prefer them as a company. Yes, both are data mining beasts but at least Google does some cool things and they have services I actually use. Not one Facebook service comes to mind that I use...
I would wager that there are more drones out there being used for entertainment than there are being used for killing... we call them quadricopters and you can buy them at Wal Mart.
Oh, the Hindenburg is a bad analogy as it's purpose wasn't to target people from the sky to kill, it was an accident. Now, if Google is going to be using Hydrogen instead of Helium in their balloons like the Hindenburg then you would have a point.
The lower you are, the lower the latency. Think of a laser-based optical network that connects the drones, with laser links going down to earth as well.
When you think of the control required to ensure a quality of service and full coverage, the balloons seems like a terrible solution.
So why does Netflix get to complain? They have already spent the money for the ridiculously one sided peering(Netflix sending tons of traffic and Comcast sending close to none). You would think Netflix would be happy to spend this money to better their service! Right? Or is it so detrimental to their cash flow that they would rather complain? They should have just told their customers "hey, we heard you were having some problems with our service, look what we did for you!"
In the end this is just a smear campaign to distract Netflix customers from the price hike that is coming. It is obvious Netflix was going to raise its subscription prices eventually. So why not point the blame to a company who is already disliked. The majority of HN seems to be pretty smart people, it would be a shame if you fell for this rather underhanded ploy.
In any case all Comcast's traffic is for their customers, when peering needs to be balanced to be free is when to parties are offering transit to other networks beyond themselves.
But actions like that can be easily spun out in the media (media of which has very close ties to Comcast) to make Netflix look bad.
What would make it better is for all of the major players (Google, Facebook, Twitter, Apple) to collectively blackout Comcast, whilst educating the public on how they are being used and screwed at the same time.
If ISP customers paid per-GB and/or per-Mbps, then the ISP shouldn't care if it's Netflix or Hulu or YouTube being consumed.
The issue is that today, it's difficult to get a good feel for how much bandwidth is being used at any given moment. Unlike a water faucet, our devices don't really have a knob we can adjust the speed with.
None of this has been a problem here in Sweden, where the infrastructure is solid and easily supports end users with 100 Mbit home connections without me hearing any complaints.
Do you have a choice of multiple ISPs? If yes, then that's the critical difference vs the U.S., not the infrastructure. If no, then why has your monopoly ISP built such good infrastructure?
A huge and easy justification for that is that Orange's network had mostly been built using public funds since it was a public company at the time.
Since then, several competitors have built their own network, but most still use Orange's sharing for remote place, small villages etc where it's not worth it for them to deploy their own.
So your question is hard to answer. Why has the monopoly ISP built the infrastructure ? Because it was a public company, and if you wanted your house to be connected to the network they had to do it. Why have the others built their own ? Because it was much cheaper for them long term, especially since they also use it for their cellphone network.
And from what I see, it seems to be the same in a lot of Europe, the national companies made the original network, and competitors piggy backed on it until they had their own.
(I've oversimplified of course, but that's the gist of it)
remember the 'series of tubes' comment by Senator Stevens: http://en.wikipedia.org/wiki/Series_of_tubes
"I have confirmed with my own eyes that the Internet is many things, in many places. But one thing it most certainly is, nearly everywhere, is, in fact, a series of tubes."
Comcast's argument is that they have built a network at great expense for carrying content that others make a profit from, and they want to capture at least some of that value.
Netflix's argument is that they have built a CDN to deliver that content to Comcast's doorstep, and that Comcast won't accept it.
In my mind, Netflix charges money for their service and is not entitled to a free ride. The fact remains that running an ISP is not free, and you can't scale out a network faster than the cash flows (of which, yes, profits are a part) allow. But think about this: Netflix has a 36% gross profit margin, and Comcast's cable/Internet business has an 11% gross profit margin. Netflix's cost structure is lower in large part because Comcast does a lot of the expensive work for them (i.e. last-mile network maintenance and distribution.)
Netflix drives significantly increased Internet traffic usage, and Comcast's investors should not have to accept lower profit margins so that Netflix's investors can enjoy larger margins because of investments that Comcast's investors have made.
I would have a lot more sympathy for Netflix if their margins weren't so high. It just comes off as greedy when they have a 36% gross margin.
The truth of the matter is this has absolutely zero to do with costs, and everything to do with extracting rent via market position.
I know it's not a popular opinion on HN, but I don't think it's unreasonable for Comcast to ask Netflix to shoulder some of the cost. It also has the added benefit of incentivizing Comcast to provide faster service. If they can charge Netflix more for pushing more data, then it directly leads to more profit if their users consume more data.
Actually it gives them an incentive to provide slower Internet service. This encourages content providers to pay to use the Comcast private network instead of the Internet.
We can comfortably say that 99.9% of the services on the internet are not going to pay Comcast. The big players might, but there's an extremely long tail.
Thus, for most of the internet, Comcast encounters a conflict of interest. Its actual paying customers want higher speeds. But it has every incentive to reduce speeds for services that aren't paying for more.
I pay my ISP for access to the entire internet. If they screw with my access to a service simply because they didn't pay, that makes no sense.
If I'm paying $75/month for high-speed internet access, that means the entire internet, not just the parts they get kickbacks from.
I pay my ISP for my access, Netflix pays their ISP for their access, and we get to talk to each other. That's how this stuff works. Comcast charging Netflix is double-dipping, and is directly opposed to their supposed purpose of serving me, their customer.
(Hypothetically. I'm no longer a Comcast customer, thank goodness.)
Edit: I usually hate real-world analogies, but I think it could be instructive here. Let's take the USPS as an example of something in the real world that looks a lot like an ISP.
I can pay USPS $5.60 for a flat rate Priority Mail envelope, and they will deliver the contents anywhere in the country within 2-3 days. Doesn't matter who's getting it, and doesn't matter what's in the envelope as long as it fits.
Now, let's imagine that some company ends up receiving a lot of mail. Providing mail service is expensive, right? It takes a lot of trucks and postal workers to deliver all this mail to this one company. Surely they should be charged for what they're using, right?
Of course, this company says no. These people who are sending us mail already paid USPS. It's their job to deliver mail to its destination once they get paid to do so. Why should they get paid twice to deliver the same piece of mail?
In retaliation, to "convince" this company to pay, USPS starts slowing their mail. Those trucks and mail carriers are expensive, we need some of them elsewhere. You'll just have to wait.
And as a result, because USPS is demanding that this company pay to receive mail that I already paid to send them, my "Priority Mail" starts taking 4-5 days to arrive instead of 2-3.
I hope it's obvious that USPS is completely unreasonable in this hypothetical scenario. I see no difference between this and Comcast charging Netflix for service I'm already paying Comcast for.
Netflix doesn't pay an ISP. That's the whole point of the arrangement, it goes straight from Netflix to Comcast. There is no middleman. So their peering arrangement is the equivalent of "paying their ISP".
To say that Netflix should not have to pay for internet traffic is ludicrous. I want a piece of that. Comcast should hook up their network to my servers so that my personal website is fast for Comcast users. For free of course.
In other words, you're describing a situation in which Comcast blackmailed Netflix for access to Comcast's customers as one in which Netflix claims they "should not have to pay for internet traffic". This is detached from reality.
Now they're attempting to shift costs onto everyone else. And they're using the public's distrust of ISPs to get the public to pay for Netflix's business expenses.
In other words, we're already in the double-dipping situation I'm decrying. The "we each pay our ISPs" scenario that I described and that you quoted is now, sadly, a hypothetical.
Now they are using the biased tech media to create a campaign whereby they will lower their costs even further and shift them onto others (including you and me).
In a reasonable free market, none of this discussion would matter. The agreements would all shake themselves out in the marketplace. But give a company a monopoly and suddenly they're not subject to the same forces. Between Netflix, Cogent, and Comcast, only one of them is a monopoly.
You pay for guaranteed bandwidth within Comcast network. Not the whole internet.
Which gets back to the whole issue at hand. If Netflix wants to improve their service they need to pay for better transit. So they cut out the middleman and go directly to the source of the majority of their viewers.
This is no different than Twitch, Google, or Microsoft direct peering with Comcast. They see the value in allowing people to access their services as fast as possible, so they spend the money to do so.
This is internet 101. Netflix just wants everything for free.
I accept that they can't guarantee bandwidth once traffic gets out of their network, but as long as both sides and everything in between has adequate bandwidth, shouldn't I be able to use it?
From my point of view, if they're allowed to let their service degrade so that they can turn around and charge Netflix more money to make it go faster again, how is that an incentive to provide a faster service? To me, this is an incentive to provide slower service so that they can charge companies like Google and Netflix more money.
Edit: Funny this sentence can be turned around on you by clarifying who requested those streams.
Prior to paying Comcast for transit, if Comcast Customers asked Netflix to push 4k streams to them, that decision carried no repercussions for Comcast Customers.
This is not true. Netflix has to pay for transit of the bits leaving their servers just like everyone else. They just pay their transit provider for this, not Comcast (well, except when it is Comcast, but that's beside the point).
The right ways for Comcast to keep their costs under control re: video streaming would be to either meter pricing or drop rate limits. The problem is that metering is unpalatable to their customers, and rate limit drops would expose the lie that the entire cable modem marketing engine is based on (that last mile downstream throughput maximum is the only number that matters).
Netflix doesn't have a transit provider any more. They are piping directly into Comcast and wanted to do so for free. I want my website to be delivered to all Comcast customers for free, do you support that?
Shouldn't Comcast be required to purchase enough capacity to provide high-quality service to their residential customers? SFI between a content provider and an eyeball network should be a mutually beneficial cost reduction, but Comcast has enough market power (captive broadband subs) that they can get away with intentionally not buying enough capacity to serve their customers.
A peering agreement is free when both sides are accepting traffic for the others, but that's not what is happening in this case. Netflix -> Comcast is one way.
Still waiting for an answer to my previous question, should Comcast delivery my personal website to their customers for free?
It seems like Comcast entered a shitty peering arrangement with Cogent and decided not to upgrade the interconnect links in retaliation. Netflix gets caught in the crossfire and ends up having to pay Comcast directly to avoid a few million angry customers. Is there any transit provider that Netflix could have gone with that has enough interconnects with the Comcast network? If not, Netflix is forced to pay whatever price Comcast names for their customers to receive adequate video streams.
But Netflix was using them for almost 100% of their traffic delivery. That's insane. There are other transit providers they could have paid more to and gotten better service, but that would have meant higher costs.
The whole irony of this situation is that I've heard rumors that Netflix is paying less per gigabit to Comcast than they paid Cogent while getting better service than they would get from a top-tier provider. If that's true, other companies will be lining up to get a similar deal because they'd just be cutting out the middlemen.
> I want my website to be delivered to all Comcast customers for free, do you support that?
Sure, if your website has a peering arrangement with Comcast. Comcast isn't required to offer one to you, though.
As a cable subscriber, I expect that when paying for N Mbps of bandwidth, I'm entitled to N Mbps of bandwidth of the content of my choosing. If Comcast's pricing model needs to change to a cost-per-gigabyte model in order to cope with the increased quantity of data customers consume, so be it. But sneaking the costs onto Netflix's tab effectively shifts Comcast's costs to all Netflix customers, allowing Comcast to artificially lower their prices relative to smaller ISPs without the market share necessary to effectively extract rent from Netflix.
100% of the time ? not going to happen
So let's say I'm an ISP and I bring a fiber to your home, and gives you a gigabit ethernet port. You cannot expect all current and future customers to be able to use 1 Gbit/s at the same time. You would need a big non blocking switch with as many ports as subscribers, the technology for this does not exist once you reach a large customer base.
Now do you want me to shape your link to 0.1Mbit/s, because that's the only thing I can guarantee if all customers uses their link at the same time ? Or you'd rather have the 1Gbit/s possible bandwidth ?
Which one is better:
1) guaranteed 0.1Mbit/s for 10$, 5Mbit/s for 100$, 1Gbit/s for 4000$ 2) possible 1Gbit/s for 20$ ?
If you take the globalized approach, you cannot have business like Netflix, they destabilize the equation.
In order to make that happen, of course, we'd have to live in a fanciful world where shifting last-mile delivery cost to content providers wasn't an option so the painful process of exposing this cost to customers couldn't be hidden in a rat's nest of perverse incentives that benefit the most entrenched corporations. (Ironically, and despite its protestations, Netflix's ability to pay this rent is a barrier to entry for its own future competitors.)
With adaptive streaming technology, there is not such thing as "today streaming demand"
If whenever you raise the pipe size, it is automatically filled by an higher resolution/quality/bitrate stream, it is a never ending game.
What I mean is that, whatever the state/quality of ISP networks is or could have been, we are doomed to face that "Netflix problem". If networks were better, then Netflix would already offer 4k or dual/triple HD streams and so there would be saturation anyway.
> (Ironically, and despite its protestations, Netflix's ability to pay this rent is a barrier to entry for its own future competitors.)
Actually, moving all Netflix traffic to private pipes will free the shared ones, so a new freeloader can use them and cut Netflix prices ;)
If customers are requesting more data than the last-mile pipes can handle, there are a number of fair ways to handle the overuse:
- At peak times, customers are throttled in direct, formulaic proportion to past (peak) usage
- Customers are charged more for usage at peak times
- Customers are limited to n% of the pipe at peak times
Notice how none of that singles out Netflix, just high usage in general.
If we're going to throw out some numbers, Netflix's net income was just $112M on $4.3B in revenue last year (2.6%). Comcast's net income was $6.82B on $64.66B in revenue (10.5%). Netflix's spend on content acquisition and production is increasing, which means that 2.6% is shrinking; they're already cash flow negative IIRC. Not much room to start doubling up on bandwidth fees they already pay.
I understand that Comcast doesn't like this because it makes them a simple seller of bandwidth instead of a "provider of premium services", or whatever marketing mantra they're using this week. But if customers do not want their "premium services", but just want bandwidth, the solution to that is to build better premium services, not to try to get paid twice for one stream of bits.
If the usage pattern of Comcast's users is changing such that Comcast's internet offerings are no longer bringing in enough revenue to support usage, that's a problem with the offerings. They need to charge more, or charge differently.
However they change it (if indeed they need to change it, and it's not just a money grab, which seems more likely) then they need to charge based on how their users use the service, not what specifically they're using it for. If peak usage is a problem then charge more for peak usage, but there's no reason that A should pay more for the same peak usage as B just because A's peak usage involves Netflix.
Because the customers have to pay for the bits being transmitted to them over Comcast's network, but Netflix has to pay for the privilege of competing with Comcast's own video services. It isn't coincidental that this became an issue after the FCC's Open Internet Order -- which specifically forbade discrimination against competitors to a broadband provider's own video services -- was struck down. This was pretty much the prime example of what it was intended to prevent.
Because Netflix wants a special arrangement that makes their bits deliver more quickly. Why shouldn't they have to pay for that?
To provide the service of access to the internet, Comcast must connect with other networks that the stuff on the internet is connected to. This is the business they're in. That's specifically what customers are paying for when they buy internet access: Comcast gets money from customers, that money goes to connecting Comcast to the internet, then connecting Comcast to the customers' homes. Thus internet service is delivered.
One of these networks they must connect to in order to sell internet access is Cogent's. Behind Cogent were hundreds of thousands of sites and services, including Netflix. These customers all pay for their internet connectivity. Cogent's being paid to connect these customers to the internet (Netflix's bandwidth bill was at least in tens of millions per year), Comcast is being paid to connect its customers to the internet, they both need to connect with an adequately sized pipe to meet their obligations to their respective customers. They're both paid handsomely for the service.
Comcast was being derelict in its duty to its customers by refusing to maintain a wide enough connection between its network and Cogent's network to reliably deliver access to that part of the internet. Customers are paying for that service (unlimited access to the internet at 15+mbps), but if the part of the internet they wanted to reach was Cogent's, they couldn't get anywhere near the advertised speed, and videos sometimes buffered or downgraded bitrates at prime time.
Netflix should not have had to pay to correct this, Comcast should've. The connection between Netflix and Comcast customers wasn't poor because Netflix refused to send the bits fast enough, but only because Comcast refused to accept them fast enough. That's Comcast's responsibility to address. Providing that network on their end is the business they're in. Netflix is not trying to pay for any special "fast lane" speed that other sites don't get, and its' already paid for its internet access, it's just trying to avoid going out of business as a side effect of Comcast screwing its own customers by failing to provide the service it's been paid for.
> Netflix should not have had to pay to correct this, Comcast should've. The connection between Netflix and Comcast customers wasn't poor because Netflix refused to send the bits fast enough, but only because Comcast refused to accept them fast enough.
One thing to add to that is that Cogent was happy to deliver the bits fast enough.
Comcast's CEO is a smug bastard who doesn't think his customers want faster speeds, or cheaper internet, or higher level of service than Comcast currently provides. The CEO doesn't care that his customer satisfaction and service ranks dead last -- he things that's just dandy. Comcast's CEO thinks they were ranked the "Worst Company In America" not because they might be the least liked in general but because a few, vocal, not-representative people on the internet were mean.
I've seen nothing of this low level of service or high cost; I can get a 150mbit connection for $89/mo. It's not Google Fiber, but it's really not bad at all (especially compared to FiOS, which I have and hate with a passion). I mean yeah, their customer service is shit, but the Internet service itself is good.
As others have commented, consumers already pay for both services.
If it's infrastructure cost that you're concerned about I've read somewhere (Bloomberg?) that Netflix offered to co-locate its equipment in Comcast's data centers but the latter refused.
Routing each bit to customer houses is the difficult part.
The offer to put Netflix's servers in a Comcast data center does not help.
That's what CDNs are for. Which, as the article shows, is how Netflix delivers video to subscribers of every other major ISP. Why is it they can somehow route all that traffic to customer houses but Comcast can't?
Take "last mile" literally and divide the US in a big grid of 1x1 mile squares.
If Netflix was able to dispatch its traffic up to each of these squares, there wouldn't be any problem. but there wouldn't be any Netflix either :)
That's not how a CDN works.
Look at the diagram in the article, between the fourth and fifth paragraphs. Every other ISP uses transit networks to distribute Netflix traffic (like all other traffic) to its subscribers. Netflix puts its content on CDNs that are conveniently located to make the transit networks' job easier, by making "netflix.com" endpoints closer to large populations of subscribers. Somehow all the other ISPs manage to "route traffic to customer houses" using that arrangement. Why does Comcast have to be different?
In other words it is acting like a monopolist or a shrewd business.
The fact that Comcast is eschewing the Netflix offer of free Open Connect CDN appliances (https://www.netflix.com/openconnect) tells me that this isn't an issue relating to upstream congestion, but rather with Netflix competing with Comcast's own VOD service.
As I said, generating traffic is easy. If there is no drawbacks in raising streaming bandwidth on Netflix side, they will continue to do so, 4k streaming, multiple streams per account, YouKnowWhat... all the way until all ISP subscribers will use 100% of their available bandwidth at all time.
I may pay for a XXX Mbit/s access, but I know that no ISP can sustain that XXX limit for all customers at the same time. And I certainly don't want my ISP to shape my fiber access to 10kbit/s because that's the guarantied bandwidth they can offer me.
Note that I'm in Europe, I don't have an resentful/hateful relationship with my ISP, but I can understand the US case is different because cable companies (used to) have bad business practices.
I do agree though that nowhere in the commercials it is said that you are paying for a peak rate.
I think you mean "still have bad business practices." Comcast regularly gets rated as the worst company in the country by consumers (in some cases the worst entity, public or private, beating out the IRS for that coveted position).
nearly 1/3 of the US have only a single option for broadband internet, and that company is typically a relatively unregulated company with low business costs that charges high prices. That's just a recipe for bad behavior.
But the customer is paying for that to happen. The last mile is payed for by the customer. The deal between Comcast and Netflix is designed to handle the macro level bandwidth, which is only painful when you need to cover distances.
By putting themselves wherever Comcast asks, Netflix removed all but the last mile costs. Those costs are paid for by end users.
When you start looking at the question like this, you're abandoning any technical argument and instead move the conversation into "well, look at all this money moving near my proximity, I should be getting my beak wet." This is a pretty good warning sign someone is/has started framing the conversation on something other than the issues at hand.
Fairness in profit margins is a red herring at best. I like the idea of network neutrality, but I have a much much much stronger fundamental belief: nobody is entitled to a business model. That kind of thing shouldn't even be given the slightest bit of consideration when setting policy.
Unless the explicit purpose of a policy is to create a business model, as is the case in Intellectual Property. Creation, improvement, destroying, or diminishing of business models is a lever that is rushed to way too quickly in some nonsense idea of "fairness" for people that are too used to cooperate welfare and don't believe in anything but entrenchment of middle of the road "compromises" that just serve as regulatory entrenchment of the status quo.