"This is about consumers not getting what they paid for from their broadband provider. We are trying to provide more transparency, just like we do with the ISP Speed Index, and Verizon is trying to shut down that discussion."
If Netflix is 35% of global internet traffic at peak capacity (as per the Akamai CEO's comments at a number of events), is it really fair to treat them like every other company? That is to say, if Netflix is really the sole driver of Network upgrades, why does Verizon have to subsidize their costs of business?
I'm not taking the stance that this is correct, only querying as to why the dynamics here are such that we automatically assume Netflix is in the right. Guilt tripping Verizon into adding more routers is a major net positive to Netflix's business.
Yes, ISP customers are paying for access but the business reality is that the ARPU per subscriber is decreasing every year at a rate lower than additional subscriber acquisition can sate. As Wall Street demands growth, it has to come from somewhere, and since Carriers are not able to sell consumers additional services (no matter how hard they try), they need to find another set of wallets (content providers). They're not utilities and have a profit motive, right?
Again, I'm not saying this is correct, simply trying to add another viewpoint to the conversation.
Customers are requesting that data, through a pipe which they have already paid for.
It is unfair for Verizon, or any other ISP, to ask to be paid twice for the same bits. What bits I am asking for, as a customer, shouldn't make a damn bit of difference.
To my understanding, the upstream ends are linked largely thanks to peering, which is based on mutual benefit. At the upstream level, nobody charges other providers because everybody is interacting with everybody- sort of like, "I bought this round, but I know you'll be buying the next" so you don't worry about balancing the books.
Then Netflix comes along, chews up 35% of internet traffic, and basically takes a disproportionate piece of the pie. If I've got everything in my head right, this is against the spirit of how peering is supposed to work, and so it doesn't surprise me if spats start to develop upstream.
If Verizon is unhappy because they have to develop their last-mile network to meet promised speeds, that's Verizon's problem. But if Verizon is unhappy because the upstream is becoming distorted and they are becoming beholden to just one peer (Netflix) instead of an environment of free, balanced exchange- maybe that's not Verizon's problem.
A thought experiment: Let's pretend it's not Netflix for a moment. Let's pretend there's a bunch of competing streaming media services which happen to take up, combined, the same amount of bandwidth that Netflix does.
In this case, Verizon would still have the same problem (if you buy that they have a problem at all, which I personally do not), except they couldn't play at being mafiosos, holding one in particular to what would be a protection racket in any other context since the load is the same amount, just distributed among many much smaller players.
What would they do in this case?
The fact that it's one large company gives them the ease of threate^H^H^H^H^H^H leveragi^H^H^H^H^H^H^H^H negotiating with one entity instead of upgrading their damn network like they're expected to do.
If Comcast wants to offer their customers video over a traditional cable network, they pay content providers.
Why should the Internet be different? Shouldn't ISPs be paying people who provide the content their customers are buying?
The Internet is different in many ways, but notably because cable companies aren't doing peering and don't have to deal with the complex settlement of international data trafficking.
The reason ISPs can charge both the customers and the content providers is because we have no other choice. Where I live I have a choice of two providers: Comcast or Verizon. My choices are: use one of these two, or have no Internet. Not much of a choice.
It's possible that if Netflix didn't exist someone would fill that vacancy but it's also possible that the aggregate total volume of bits transferred during streaming would be lower if Netflix did not exist.
ISPs typically possess less total bandwidth then they sell. The problem is that the typical users usage is rising faster than the ISPs are willing to handle. If I pay for 50Mb/s, and the other end of my communication has enough bandwidth to provide it, then I should be able to get to that speed.
As far as I am aware, there has been no accusations that Netflix does not have enough bandwidth to supply their customers, but that the intermediaries have oversold their networks to the point where they cannot deliver on their promises.
If they choose to reduce that speed for any reason, or throttle any kind of connection they are entitled to do that but they can't complain when someone else says that is what they are doing.
Personally, I think given how much public money has gone into infrastructure and the amount of money I pay in taxes it seems insane for me to first pay for the ISP's infrastructure with taxes then have to pay them again. So while they have a right to throttle, they don't have a right to throttle my connection on infrastructure that they did not pay 100% for. Which probably accounts for the majority of their infrastructure.
It's also very hard to prove that Verizon is throttling, versus, say, simply physically underprovisioning the connection.
With respect to public money, I think you've hit the nail on the head. The crux of the Net Neutrality debate, to my mind, is whether these networks are public or private. If they're private, Net Neutrality is bullshit. If they're public, it's obvious.
It is in consumer interest for the networks to be public, but the same could be said about the socialization of any number of industries. I'm constantly fascinated by the way folks consider socializing the telecom industry to be incredibly reasonable and socializing, say, the healthcare industry to be insanity.
Edit: Maybe I need to tag all of these posts with #DevilsAdvocate to avoid being downvoted into oblivion.
Can't I want both to be socialized?
* Water
* Sewer
* Power
* Internet
* Healthcare
* Fire
* Police
* Roads
* Education (added per /u/CodeMage)
Feel free to chime in if you can add anything I forgot.
Cheers.
I'll note that we should have explicit public subsidies for Internet and that I would much prefer to have public networks with private operators (to my mind that's heaven for competitive markets but politically impossible in the US).
Orphanages
Mental institutions
Defense
Still a good list to have around so we can argue about where the line should be drawn. Kudos!
Healthcare is arguably a public good, at least as far as vaccinations and contagious diseases go, since herd immunity is a large way to fight them (especially for contagious diseases for which we have 80%-90% effective vaccines). Not so sure about other illnesses (like cancer) though. I'm personally for socialized medicine for other reasons (humanitarian), but I agree that the natural monopoly argument is not applicable there.
Education is tricky, and honestly, I don't know any better than the next guy how to solve the issue. What's important to one parent may not be important to another. Second language? Art? Music? Sports? Recess? Lectures? History? Religion? Critical thinking? What outcomes are important? Good test scores? Writing?
Health care has become a perverse industry because its market properties were destroyed not by government, but by malicious corporate conglomerations out to maximize their own profit at the expense of human life. People forget that one of the purposes is government is to prevent private organizations from growing too powerful and interfering with the general welfare of the people, because the narrative is rarely framed this way, but it's a very important governmental function. I don't think socialization is the answer, but something has to be done.
I think health care needs to be brought back down to earth, and we should outlaw all non-catastrophic health insurance products. The insurance racket has completely destroyed that market. There are other elements that have caused the health market to get completely out of whack, but insurance is at the very core of the issue, and it has to go away.
When almost half of the country's bankruptcies are due to health care debt, simply sick people trying to stay alive, we have a real social problem that we need to address.
Of course, we did the opposite of this and made it illegal not to have health insurance, only further inflating the size of the monstrosity that has leeched and destroyed the system in the first place. I don't believe writing a blank check, like the other countries have done, is the way out. I believe the way out is going back to a what-you-see-is-what-you-get approach, and forcing medical providers to charge rates people can actually afford, as their business will immediately plummet if they don't seriously rework their billing structure to become affordable again since people will actually be paying it now.
Most European countries offer health insurance, and total healthcare costs are much lower (compared to the GDP), so I don't think insurance is the problem (also, AFAIK most countries are bankrupt because of bad bank loans, falling GDP/taxes, and social payments (pensions), not health care costs).
The real problem is that doctors get paid when people are sick, so they are not incentivized to heal them, long-term (without regulation). If doctors were paid when the patients were healthy, you wouldn't have this problem. But I don't know how to implement that.
Insurance, classically understood, is risk sharing, not cost sharing. In other words taking it from a "not everyone will use this service" risk sharing model to "everyone will use this at some point" cost sharing model is what causes the problem of hidden pricing.
You can have it both ways (risk sharing and visible prices) because most healthcare costs aren't unforeseen. In other words, the number of accidents where the hospital needs to be relatively sure insurance is involved pales in comparison to the cost of routine visits/care which could be paid for out of pocket and prices would then be visible allowing for competition.
However, there are also many medical costs that can be foreseen (even planned for) but are either "beneficial" for the society (e.g. childbirth) or still risky and essentially no-one's fault (e.g. cancer - most people will get it, but some will get it at 50, others at 95). Incidentally, it's cancer and the catastrophic treatments (e.g. heart surgery) that are most expensive. Possibly also because it's in the big-pharma's interest to keep it this way (at least for cancer).
Insurance is absolutely the wrong model for something with this kind of usage pattern. The insurance industry is notorious for its antics precisely because they're one of the only industries that can only make money when they don't provide the service you're paying them for. The fact that medicine is so widely used makes this especially egregious with regard to medical insurances. They'll do anything they can to deny your claim because aside from basically accidentally having a healthy group of people in the insurance pool (due to the way corporate policies work, and the new health legislation), denying claims is how they make money.
Firemen are even more heterogeneous, in that they have arrangements for big fires and catastrophes like tornadoes where adjacent units come to help. After my city got hit by an EF-5 tornado, as I was exiting my trashed apartment complex, there were fire units varying from a small Indian tribe that's right across a state border from us (and the city e.g. treats the sewage of their casino) to one 40 miles away. Next day, the markings on my apartment door were from a fire unit the came from 160 miles away.
What is needed is something like unity of command/hierarchy. Having two or more units responsible for the same territory leads to the sorts of problems you cite, e.g. a friend told me the NYC city police and subway police once got into a big firefight. This would have been in the '70s or earlier, now they've been unified.
For example, here in the UK, we have the formerly nationalised telephone network provider required to allow individual ISPs access to key parts of its infrastructure via local loop unbundling, so those ISPs compete for the contracts with individual customers. And while that infrastructure is indeed common and so in a sense a monopoly at that level, there are also multiple cable companies and mobile networks with completely independent infrastructure all the way down. There have even been a couple of impressive success stories where installing high speed broadband infrastructure was too expensive or risky in a certain geographic area for any of the big commercial providers to take it on but the local community devised their own solution, sometimes even finishing up with a better connection than what most of us get in big cities.
In other news, prices for Internet access in the UK seem to be dramatically cheaper than most places in the US.
This is what I'm advocating for. A public last mile that any service provider can use.
These rents comprise something like 5-10% of city revenue. And so everybody gets what they want:
* The city gets a free 7% budget bump, and doesn't care who pays for it as long as it's too complex to be an election issue
* ISPs get a monopoly, and don't care that they play the scapegoat because consumers have no other choices anyway
* Citizens get better government services at lower tax rates. Sure, they pay for it via a backdoor tax on internet access but nobody really notices that and they direct their anger at the ISP rather than the elected government
Really the only way to throw a wrench in this system is something like Google Fiber. Nobody wants to be the elected official that turned Google away, because that is actually a sound-bytey enough issue that could come up at re-election. And Google, as a company that doesn't really care about being a profitable ISP, will only move in to a city if it gets a pretty good deal on the rents.
Part of my point was that there is no physical monopoly here, even on the wires.
BT (formerly the national telephone provider) has one set of infrastructure.
Cable companies have their own, literally installing a different box in your home with an independent physical connection to their own network.
If you use one of the mobile providers, which is viable for a home connection in the 3-4G era if you're only using a modest amount of data, there is no physical wiring to the home at all.
There are even a few special cases where people have literally built their own infrastructure as a community, only hooking in to some big provider's network higher up the hierarchy, presumably on some sort of customised, one-off deal.
So while we do have local loop unbundling and a shared-last-mile kind of deal, and we do have ISPs competing for that same physical connection, even that is not a monopoly provision.
IMO non-catastrophic insurance should be abolished, the government should overcome the supply chain cycle ("Sorry, I have to charge $500 for this MRI because this machine cost me $4 million") by producing its own functional medical equipment and selling it at or under cost, and health providers need to get back to reality and charge face rates that real people can pay directly. And we all need to come to terms with the fact that some of the extravagance probably needs to be toned down.
However people are not willing to accept the consequences of hospitals that can't surge in catastrophes. When an EF-5 tornado hit my city 3 years ago, it took out one of our two hospital complexes. Things got ... rather intense at the other (http://stormdoctor.blogspot.com/2011/06/first-response-mode-...) but it being big, and stocked with otherwise excess inventory of all sorts of things, kept the death toll at ~160 instead of a likely 1,000 or so.
My list would include: non-elective healthcare, food, housing (same as for food), electricity, water, local/regional transportation (e.g. for commuting to work), internet access and possibly basic internet-capable mobile phones.
Basically "natural monopolies" are certainly all candidates for complete socialization.
It doesn't matter either way, in the end the experience is the same.
Is it really the same people arguing for that? First and foremost, the telecom industry is nothing like the healthcare industry, and they shouldn't be reasoned about in the same way. Common carrier status has precise laws and legal meaning surrounding it. Please don't confuse it with socialized medicine.
You lost me there.
"There is only one valid definition of a business purpose: to create a customer." ~Peter Drucker (whose writings contributed to the philosophical and practical foundations of the modern business corporation)
“On the face of it, shareholder value is the dumbest idea in the world. Shareholder value is a result, not a strategy… your main constituencies are your employees, your customers and your products. Managers and investors should not set share price increases as their overarching goal. … Short-term profits should be allied with an increase in the long-term value of a company.” ~Jack Welch (former very successful CEO of GE)
Jack Welch was hugely successful in creating and competing in new markets specifically because competing in saturated markets is not profitable.
Those models are great for industries in the 0-80% saturation growth period, but aren't really applicable to high saturation markets.
The negative shareholder value argument depends on a summary characterization of all investors as people who only care about money. Not all investors are like that. You can create shareholder value out of anything that your shareholders value, which is not to say that "shareholder value" should even be a serious metric. It's just a narrative that anti-corporate people have drummed up.
And yet financial statements are reported in dollar units, not in customer units.
A business is a balancing act of many "should"s and "could"s.
But when your top priority, or god forbid, sole priority, becomes "maximizing" money, versus contributing in a meaningful way to society, your business no longer deserves to exist.
2) Agreed.
3) All of the big Telcos are public, but your point stands.
To be explicit, I wholeheartedly agree with you about humans making pseudo-random decisions. I'm trying to explain the dynamics of the Telco position, from their vantage point because I think it helps the discussion to view things from other angles.
Also, intentionally physically underprovisioning the connection is throttling, plain and simple. You are limiting the amount of traffic that can traverse that network boundary to 10G/40G/Whatever your port speed is. What difference does it make if I do it in software or simply refuse to bind more ports to the team?
Hard to prove in a court of law, perhaps, but the speeds of access to the businesses being extorted can tell a pretty convincing story.
This chart: http://appledailyreport.com/backend/wp-content/uploads/2014/... paints a nice picture of how ISPs want to extort businesses that are in more profitable lines of work: notice how speeds change from before Comcast started trying to extort from Netflix, to during the negotiations, and after Netflix payed their protection money.
Level3's article also called this out: the ISPs are purposefully, physically, underprovisioning their connections, because it gives them leverage over companies like Netflix.
As to why the internet is different: the internet is not like healthcare. Socialized or not-socialized medicine is obviously not that big of a deal to the future of a nation; both the US and Britain were economically competitive while having very different systems.
In contrast, the Internet obviously matters a great deal to the future. And if various players (hollywood; last-mile ISPs) can manage to appoint themselves gatekeepers and engage in rent-seeking behavior on the internet (in the United States, at any rate), well, HN is acutely aware of what that will mean for the future of innovation, and the future of nations as well.
I understand that it would be a negative impactor on the future of nations and innovation; I'm asking if that's enough to actually drive change. My sense is that, historically, with respect to ISP behavior, over time they get what they want. I'm not saying this is right, I'm simply noticing a trend, just as many have noted that less-private social networks win over more privacy focused businesses.
Sounds sufficiently sophisticated.
http://en.wikipedia.org/wiki/Last_mile
"Last Mile" is a common telco industry term. It's not about framing, it's the correct term for the connection. It does not connote any specific value to the connection relative to others, nor is it derogatory. It's simply saying where in the network the connection lives. A "last mile" connection is not an inter-network connection, it's a specific link within the network that your ISP has built from the rest of their network to you, the individual customer.
Last mile is from the POV of the carrier, and implies that something is being delivered to you.
First mile is from the POV of the customer, and implies that this is the first mile of your internet connection.
The direction data is flowing is irrelevant.
How is that viewpoint even remotely considered valid ?
I've yet to meet any carrier who wants competition, for anything. Their ideal world would be that they would decide which phone the customer can have, what limited service the customer can buy, and how much it is, and take-it-or-leave it. That's how AT&T worked, and every major carrier today is a spin-off from AT&T and still thinks pretty much the same way.
The word "promise" is not an accurate reading of the legal concepts at play in bandwidth purchase. Dedicated bandwidth for consumers is expensive and completely unavailable on cellular networks.
Verizon also doesn't have to let iPhones connect to their network and only does so because they benefit financially from the relationship. Trust me, putting a device onto a major cellular network in the US is highly political, full of peril and financial alchemy.
Source: I do lots of telecom stuff specifically around wireless.
Wouldn't intentionally throttling and/or not keeping up your capacity intentionally be a violation of this "best effort" doctrine? Not arguing, honest question.
Ideally, you're right -- if netflix (and internet video in general) drives growth in internet traffic, maybe it isn't fair to require service providers to upgrade their service out of hand.
But then I think that rule primarily applies for a free market. I'd argue that internet providers are not currently participating in a free market and it causes real problems. Without a free market... companies need more supervision.
I'll put it this way: 2 competing providers. Average internet usage keeps rising exponentially. People want faster internet. The providers either lower prices or upgrade infrastructure. Netflix is not forcing them to do anything. But then it's probably reasonable for Netflix to say "Your connection is crappy because of your internet bandwidth, it's not us..." But hey, consumers can choose.
Problem here is... what are their choices? Even just one other company participating in the market that can make a similar offering (eg cable speeds) would change the whole conversation I think. Suddenly they'd be working harder to keep your business (or take you from the competitor) and they'd stop talking about more and more ways of screwing their customer and taking advantage of de facto monopoly status.
The incentives are all messed up and it encourages gouging instead of innovation.
Are the networks public or private and why can't we decide?
I don't see that argument going very far. Netflix has an ISP and pays for their own bandwidth. Why should they pay twice for their bandwidth?
If verizon has a problem with the amount of data their customers are requesting from netflix they could always disallow that or institute data caps etc.
If the ISP's customers are making requests to Netflix' servers, then yes.
ISPs have been miss-selling Internet connectivity for many years - "UNLIMITED!!" "FAST!!" "CHEAP!!" - and now they ( and their customers) are reaping that harvest.
Interesting question. I guess from my perspective I buy broadband from AT&T and they tell me under no uncertain terms that I have a cap at x amount of gigabytes. As a customer I expect to use all of that.
>Guilt tripping Verizon into adding more routers is a major net positive to Netflix's business.
As a customer it shocks me they aren't doing so. Yesterday it was Netflix, today its every kid with an Xbox1 or PS4 buying games digitally as buying physical media is out of style. What's a typical game nowadays? 20gb? That's the equivalant of many netflix movies right there, all at once.
What does tomorrow hold? Why aren't they prepared for this?
Customer revenue is decreasing and because the market is saturated they cannot add enough customers to combat the atrophy of revenue. As shareholders expect growth every year, ISPs are forced to look for new revenue streams.
If you don't think they should be doing this as a private company, reclassify them as a utility. Otherwise shareholder demand is what gets satisfied.
Carriers have very specific equipment amortization patterns in their heads and they are financial models that banks accept. If carriers have to change their amortization patterns (more routers, faster) they have to change their financing which damages the value of the stock.
Sure we can do this, but what does "fair" mean here? I'm trying to say that the argument is not as black and white as people make it out to be.
Welcome to the world of business. Change is constant, you can't just tell your customers, "Whoops we didn't plan for that."
I also disagree that its as hopeless as you make it out to be (in many nations 100mbps or even 1gbps unlimited internet is trivial to get). They can raise prices on customers, lower caps, throttle netflix users, etc if they're in such financial dire straits. Their competitors will love this as customers will migrate to them. Their competitors may have a more nimble management structure and can provide the data at this level.
What I don't get, assuming they really are in dire straits and not trying to monetize artificial scarcity, is why don't they just push out a notice, "Due to network limitations, we are throttling streaming video users to 4mbps which is enough for medium quality 720p streaming. The investment to support 1080p streaming is impossible for us. Love Verizon."
Actually, I know why they won't do this, because Comcast will pay for the router upgrades and steal all their customers once they find out. So they go around their customers' backs and try dirty dealings with Netflix and others. These guys being duopolies in most markets (substitute AT&T with Verizon in some states) just decided to not compete and simply change the rules at the FCC, which is pretty much a textbook description of collusion.
This is the biggest part I think people are missing- they are not thinking about the upstream end of the pipe. Maybe it's the same story even when you do consider the upstream, but I'd at least like people to acknowledge that aspect.
Yes it is.
If you solve their problems in the general case, then that solution will also work for the next big user, and for the small users as well. If you special-case handling for them, then it will be harder for the next new entrant (who could even be a Netflix-upending upstart) to become established.
They do use more resources, there is no question about that, but they also have higher costs to their ISPs. This cost could be mitigated by them running their own internal network and peering directly with other backbone providers (they already do this to some extent). But in an architectural sense that option is available to anyone (in practice, of course, nobody is doing BGP for their home network).
If you believe in the original promise of the end to end network (a promise which is currently lying in the street gasping for breath) then you don't want netflix special cased.
It's no surprise that Verizon (with POTS roots) and Comcast (hub and spoke cable provider) don't have end-to-end in their DNA, but rather a tollbooth model of the world. It doesn't mean we have to go along with it.
The ISPs are punishing consumers for their choices - it's just ridiculous.
There's a conflict of interest: Netflix has a vested interested in provided lightly-compressed, very-high-resolution, bandwidth-hungry movies to its subscribers, especially if it obtains 100% free peering that it demands, as bandwidth would be free for them. On the other hand, ISPs are forced to upgrade network any time Netflix unilaterally decides to increase the bitrate of the movies, without this bringing any benefit to their bottom line.
For instance, if Netflix halved the bitrate of all their movies, subscribers will complain, but surely they would be able to deliver them without any buffering to everybody, and at the same time to keep free peering agreement and whatnot; they would go from 35% of total US bandwidth to 17%, and ISPs would be satisfying their customers' choice.
So where do you draw the line? Surely Netflix can increase the bitrate at will, at any time; should they be "allowed" to do it any time they want, while keeping a free peering agreement? Should the ISPs be forced to upgrade their network every time?
Come on, they have been doing this since the internet started. And in the beginning of internet some services were hugely popular as well. Keep in mind that upgrading the network costs money, but once the infrastructure is in place, moving bits is really cheap.
If Verizon tripled its total bandwidth today, how long before Netflix begins raising the bitrate because "the Internet has evolved and we can now deliver bluray-quality streaming at the same $9/mo?". And then they fill up pipes again.
The whole issue here is cause by the impedance between the cost structure (pay by the gig transferred) and the price structure (pay by the peak speed, and I approximate the average gig transferred).
In a perfect utopic world where everybody was willing to pay by the gig transferred as that is the real cost factor, we wouldn't be here having this discussion.
> moving bits is really cheap.
Moving bits is cheap WITHIN the ISP network, but once you go outside they have a cost (depending on the peering agreements).
Forced to upgrade? Maybe not.
OTOH, should people be made aware that the ISPs failure to do so is a contributing factor to the unsatisfactory results they are getting so that they can make informed market (and political, insofar as government policy impacts the competitive landscape) decisions? Absolutely.
This is an excellent point. Why doesn't NetFlix at very least offer the ability for their users to queue content for download / local caching? If customers had a slow Internet connection at least they could plan ahead to avoid suffering streaming problems.
Ironically isn't streaming-only perhaps a violation of net neutrality? NetFlix artificially limits the speeds my ISP can deliver by only offering streaming. I have a 100Mbit/sec connection yet NetFlix chooses to limit me to streaming speeds.
No, not at all.
Because this would almost certainly result in the content industry screaming bloody murder.
As others have mentioned, the ISPs woefully miscalculated the growth of the internet and consumer bandwidth usage and as is the case with capitalism, those mistakes should cost them.
When you install a server in a colo, you don't get free transit. You find a transit provider (like Level3, Cogent, or NTT) and you pay them for bandwidth. You can understand this very intuitively as most hosting services (like AWS) will charge you for bandwidth.
As a service provider (like Weebly), you are paying your own transit provider for a certain amount of bandwidth, and they are covering the costs of delivering that bandwidth to (usually) anybody globally and the costs of maintaining their network.
If a network gets large enough, they may often negotiate peering agreements with other networks. So Level3 and Cogent may have a peering agreement, or Level3 and Comcast may have a peering agreement. These are generally set up when two networks deliver roughly the same amount of traffic to each other, or it's considered beneficial to both parties to get the networks closer, and it simplifies the working relationship.
These peering agreements can often become contentious when they become imbalanced. Very early on, I remember a fight Cogent and France Telecom had over peering -- FT felt like they were getting the raw end of the deal, and de-peered with Cogent. Cogent refused to pay another transit provider for access to FT's network and so traffic from the Cogent network to FT would drop. They resolved about a week later, with lots of unhappy customers caught in the middle.
As you can see, it's simply not the case that (a) service providers never pay for bandwidth because the end-user already pays for bandwidth and (b) peering / free transit is guaranteed.
Maybe I'm not understanding the nuances here, but it seems to me like Netflix is taking an ideological stance to hide behind the fact that they are trying to negotiate a better business agreement for themselves.
The latter is something that more people ought to be thinking about, consumers using Netflix are not just consuming more peak bandwidth, they're competing with the video offerings of all of the big last mile providers.
Although. AT&T doesn't actually offer U-verse video service to very many of their subscribers; don't know about Verizon's FiOS vs. plain DSL offerings (except that they're shedding small population states and markets as fast as they can), but the others are cable companies so they obviously do.
Peering agreements should be about saving costs on both sides, but if Netflix is willing to pay Level3 for sufficient bandwidth then Verizon should have the ability to handle that for their users.
That's how it's always worked. These fights and skirmishes over peering agreements happen often. That's the reality of how things have always been up until today, it's not a change in the status quo.
I also think that there these peerage agreement create an untruthful dynamic with their customers, the point about transparency is valid. People think they are buying "internet" access at a certain speed, not different speeds for different sites. If Verizon thinks it's OK for the speed of their customers' connection to be dependant upon "interconnect agreements" or whatever, then letting their customers know about this isn't defamation. If they do have a problem with their customers knowing, then maybe they should be making sure their customers get what they expect (or expect what they get).
In any case, I don't see how this is foul play on Netflix' part. As long as the message they display is true, why should't they be able to display it? Even if the message openly advised their customers to switch away from Verizon, I don't see a foul.
I don't see why not. I think it might make more sense to think of it as streaming video, the category of goods, rather than Netflix, the company. So 35% of global internet traffic is people streaming videos. And who cares that it's video; from Comcast's or Verizon's perspective, it's just an awful lot of bytes. So if the issue is the amount of traffic, it's just Verizon complaining that we're using too much bandwidth (yes, us; we're the ones watching Netflix). I get that's expensive to deliver all those bytes to my home, but that's what precisely what I'm paying for. That a huge number of those bytes comes from the same originator shouldn't enter into it. It's just a site on the internet.
EDIT: Also, I wanted to add, that all of the traffic comes from the same source actually helps Verizon from a practical standpoint, since it can solve a lot of it with only a small number of interconnects. Compare to a more fragmented market, where Verizon might have to make specific peering provisions for a bunch of different 5%-market-share content providers to fulfill its customer obligations.
> Guilt tripping Verizon into adding more routers is a major net positive to Netflix's business.
All sorts of businesses benefit from the ecosystem of other businesses around them. If I run a store in rural Iowa and Acme Co opens a huge factory there, then I stand to make a bunch of money from all these newly employed customers. Should I subsidize Acme? [1] After all, hiring people is expensive. If I open a hardware store near a bunch of planned construction, should I pay those real estate developers? You can expand this to the whole economy being basically just a set of interdependent positive externalities. Maybe Verizon should pay Netflix for making their service more valuable?
Or we can use the simpler model, in which you get to charge your customers. Verizon charges us for delivering the content we request. Netflix charges us for the content itself. Cogent (or whomever) charges Netflix for its bandwidth. Everything works.
> They're not utilities and have a profit motive, right?
They could charge more, and they could even charge per unit bandwidth (or tiered bandwidth) if they wanted. "My business isn't working that well" is not a good excuse to hold your subscribers hostage. You could equally use "Wall Street demands growth!" to defend Comcast's new practice of robbing banks.
Imagine I bought a popular TV set in 1980 to watch broadcast content, but after I got it home, the TV maker remotely activated a previously undisclosed filter that would only show certain channels. Then they went to ABC, NBC, and FOX and said, "So guys, you want your content seen? My margins on these TVs keep going down, so somebody's gotta pay." Not cool, right?
EDIT: Another point, probably the most important. The "problem" here is ridiculous: it's that demand for Verizon's product (i.e. internet access for end-users) is increasing a lot because it is becoming more useful to its customers. That's what every business wants! "We sell internet access and everyone is now using so much more internet! Whatever shall we do?" The idea that Verizon is up against some sort of profitability wall because of how popular its product is becoming and thus needs to charge the companies responsible for that increased popularity...it's simply not credible. Not sure why I didn't think of this point earlier.
Edit: s/Comcast/Verizon, since that's specifically who we're talking about.
[1] This actually does happen in the form of tax incentives for companies opening factories. It's a bit messed up.
Then Netflix comes along, and suddenly 35% of Comcast's customer traffic is all going to a handful of Netflix DSNs? Suddenly Comcast has to redo the whole connection. They have to buy top-shelf hardware just for Netflix.
I don't know if this has happened, but it's an interesting thought experiment to see how "just one company" could place disproportionate burden/cost on the ISP, as compared to the case where traffic travels to & from many different entities.
The Tier 1 always wants to ask for more bits than a lower tier operator. If a lower tier operator asks for more bits and tilts the traffic balance, things get interesting. Cogent is not a tier 1 but slings 35% of the internet around at will.
It does cost a lot of money to route Netflix's bits and the narrative is hugely helpful to Netflix. I'm not saying this is correct, I'm simply trying to point out that Netflix has a ton to gain from carriers upgrading their network connections and routers.
Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source.
Edit: not sure why I'm getting downvotes since silverstorm clarified exactly what I wrote
I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.
You're missing a key element of the situation: Comcast made a deal with Netflix precisely to address this problem. Netflix now has direct connections to Comcast's network, so Netflix traffic does not have to go through a third party. In other words, Comcast now directly controls how they connect to Netflix, how many connection points there are, what the average network distance is between the closest Netflix connection point and their customers, etc. So they have a much cheaper way to even out Netflix traffic: open up more connection points.
(Which, btw, AFAIK they already did as part of the deal; they certainly did not come out of the deal having "just a few spots" where they are connected to Netflix. Netflix basically is their own CDN; they have servers with their content on them all over the Internet. So Comcast has all kinds of ways to shorten the average network distance between each of their customers and the closest Netflix endpoint.)
Sounds to me like just another way of rephrasing "ISP Customers are demanding huge network upgrades without any additional revenue source."
Netflix does not send packets to you unless you request them.
I think a lot of us would have more sympathy with the ISPs' situation if they hadn't consistently and for a very long time been promoting their services as ever more valuable by offering ever higher bandwidth, yet keeping quiet about little things like massive over-subscription and high contention ratios that meant if all their users actually tried to use the service that was purportedly being offered then no-one would get anywhere near the advertised speeds they were paying for.
With the rise of things like streaming video services and Internet telephony, some people are actually trying to pull significant volumes of data down the pipes relative to the advertised size of pipe available. Everyone's known this was coming for a long time, and no-one more so than the ISPs. The way I see it, if the current situation hurts ISPs that failed to invest or set a realistic pricing model, they'll just have to take their medicine and offer a more realistic deal to customers next time (or fail because they can't compete effectively).
Funny, I actually just added an edit to my post above that discusses this. Comcast adds interconnect capacity as needed. So in at least one way, it's actually easier this way, right? Just one peer to make sure they have capacity for.
But either way, it's Comcast's job to run a network that's optimized for its customers needs, not some abstract concept of fairness. And in this hypothetical, it's not on Netflix that streaming lots of content is not compatible with how Comcast happened to build its network. Comcast needs to fix its network so that its customers can get their data.
You can even ask them for a freeish box to run inside your network to reduce requests to their CDNs
But Netflix is not "just a small number of nodes". They're basically their own CDN; they have servers all over the Internet, precisely in order to not have the problem you describe.
What's more, Comcast has a deal with Netflix to directly connect Netflix nodes to Comcast's network; in other words, Comcast directly controls how many paths there are for Netflix content to get into its network. If there aren't enough such paths, they can just open up more.
I guess the core issue here is that, from a customer's point of view, the ISP is selling me inet access, regardless of where the data actually goes, so it is the ISP's responsibility to keep their end of the bargain (to provide inet access). As a customer I should not care how is their topology planned, what kind of equipment they use, etc.
So basically the ISP though that they could offer T amount of bandwidth, advertised it as U (with U > T) because most customer won't use even close to U, and now that more and more users are actually using U amount of bandwidth (or closer to it) the ISP's infrastructure is suffering because they simply where not prepared for it, i.e. they oversold their capacity.
OT: By the way, why downvote the parent? I really thought downvoting is for when the post is off-topic or offensive or something... but more and more I see HN modders downvote because they disagree... really? so you make the post go into an ureadable color (grey) just because you disagree? I don't think it's supposed to be that way, is it?
An airline for example; now everyone actually caches their flight because Google's self-driving cars always make it on time, and then they demand that Google stops bringing passengers on time?
Or a buffet restaurant, since now everyone goes to the gym, they can actually eat a lot more, thus restaurants complain about, either the gym or the pill maker?
I don't know, it would be a strange world to live in if this happened in other industries. Maybe prices would rise too much?
Fast forward 3 years later, you come out with your pill and throw a wrench in the works. Now gyms realize they have to go back to the old pricing model because the economics have changed. Consumers will no doubt become upset at the prospec of having to pay more. Some gyms take a wait and see approach to see what other gyms will do. Your pill started it but really, it's not your fault or your problem.
What should the gyms do?
Be Honest. They can invent a new business model based on transparency. Imagine if price is not fixed but you are charged per use based on the traffic at the moment. It could be similar to what Uber does, price based on demand.
In terms of ISPs, I think there should be a micro transaction exchange where I can buy X minutes of data transfer. If Verizon is clogged I switch my router to connect with ATT etc. All networks should create a single broker device thats given to each consumer.
And then there's this: http://arstechnica.com/information-technology/2014/04/bittor...
"[W]hen we ask [ISPs] if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download—thus filling their upstream networks and nearly doubling our total traffic—there is an uncomfortable silence," Netflix CEO Reed Hastings wrote last month. "That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs."
That's because around the time Comcast was building up their networks customer demand was wildly asymmetrical. People simply download far more than they upload and Comcast wisely dedicated more of it's bandwidth to downloads.
At the time all cable companies were were building up their data network, they were building that network over top of their existing video distribution network. That network was designed [ages prior to this point] to have tiny, tiny, upstream channels for use by system control and maintenance devices, and huge downstream channels to shove 100+ analog TV channels. Because it would have cost a metric shitload to replace all of that hardware, the cable companies were forced to give their data subscribers pretty large downstream links and relatively meager upstream links.[0]
Frankly, the nerds of the day (remember, this is the mid-to-late 1990's) would all have been running home servers, and would have been demanding a symmetric connection so that they would have a substantially cheaper alternative to ISDN.
(Note that nerds that remember those days consider asymmetric home Internet connections to have been more harmful than the Eternal September.)
[0] Note that this property is pretty meaningless in an HFC network where the cable company could run fiber directly to the home, but chooses not to. [See also ATT U-Verse.]
Consumer ISPs have never expected symmetric bandwidth. If they had, they would sell us 50/50 connections instead of 50/10 or 50/2.
The only way to incentivize a company to provide the best service possible is to tie the use of the service to its cost. I wouldn't mind paying, say $2/GB for service if it meant that Comcast's incentives were in line.
That would even have an external benefit on creating competition among streaming companies to have the best compression algorithms. Efficiency and speed become directly correlated to the service and its value.
That's a ridiculously high price. You'd pay $2 for every hour of streaming-quality HD video watched, quite a bit more for download-quality HD.
My main point was that this charging scheme incentivizes companies to provide better service, not set up monopolies or short change you service.
100 gb. / month (included) for 47,5 € monthly (65 $). Extra bandwidth @ 0,5 € (0,7 $) / GB, so it's an additional 1,5 € extra for your ISP, if you want to watch a netflix HD movie (2,3 GB).
Don't worry, i know i'm getting bend over :)
Personally, I'd really love to see Netflix start using encrypted peer-to-peer to distribute the bandwidth. See if Verizon would be willing to cripple all VPN and HTTPS users to try and shake down Netflix.
However, I don't think it follows that Netflix must be treated any differently. The customers are the ones requesting the data. To the extent that anyone (once at capacity) should bear a cost, it should be the requester -- either through a higher fee or a smaller fraction of the pipe.
In practice, that will mean "the more Netflix [or anything] you use, the slower your service". But you shouldn't "short-circuit" the process and just start punishing Netflix's packets directly just for being Netflix. Rather, it should be "this customer endures this cost [broadly defined] for using this much, regardless of what it was for".
People getting their video from Netflix threatens this business, and it's pretty obvious to me that's one of the reasons we're seeing all this now, especially with Netflix getting big enough in streaming to a) pose a greater threat b) potentially have a lot of voters on their side (particularly relevant because the last mile providers are all government enabled monopolies, cable companies at the municipality level, telcos at the state and Federal level).
I'd really like Netflix to offer their own speedtest site that looks to the network almost exactly like streaming video, so the ISPs couldn't game the measurements.
Because I, a Verizon customer, have paid to access Netflix.
I pay monthly for bandwidth to use as I see fit. It doesn't matter if all of the customers are using their bandwidth to go to one source. It doesn't matter: it's our bandwidth to use how we see fit.
It is Verizon's responsibility to provide us with the service that we are paying for. They advertise a service, they market this service, and they charge us for the service: but then fail to deliver on their promises because they're charging someone else too?
They sell us capped packages and tell us we have a set amount of data. We cannot exceed the data, because the infrastructure can't take it and we have to pay for our usage. We have to pay for our demand, that's the point of caps. Make the users pay for using.
That to me implies that we are paying for a set amount of data! That we are paying for it, not anyone else.
We pay for the upgrades that are necessary to deliver data to our homes. That is the point of a subscription.
This is like saying: all of the telephone customers are calling Pizza Hut, so Pizza Hut needs to pay the phone company to upgrade the switches at the Phone Company. What???
We've already paid for the infrastructure required to deliver the data we're paying for.
Content services should not be harmed for being successful. If the bandwidth customers are desiring certain content providers it is the bandwidth providers responsibility to make sure it is available and fast, per their marketing and products.
And that's in direct competition with these big last mile providers. Cablecos by definition, and the two big telcos have their own FiOS and U-verse "TV"/"cable" video offerings (even if AT&T's coverage is meager, and both stopped their build outs except for contractual requirements when the Great Recession started).
Which provides additional strong motivation to provide less than stellar service when handling Netflix packets.
Instead of fulfilling their obligations to provide advertised bandwidth to their paying customers (who only pay them in order to access the content provided by someone else), the last-mile providers are using their customers as hostages while they negotiate ransom from the content providers. While they also report record profits (sending their customer's payments back to stockholders instead of into the network), because the economics of their business are compelling on both the consumer side (low expectations, minimal support costs, essentially no competition) and on the technology side (network technologies increase capacity-per-$ as fast or faster than traffic grows).
Oh, also, 99% of the last-mile providers operate a sanctioned monopoly that has let them build and grow their networks with essentially guaranteed income from rate-payers and essentially no competition. Hence adding Internet is relatively low-cost and highly-profitable, but apparently not profitable enough.
Because that's Verizon's business: to provide internet. That's not up to Netflix, who is an internet-based content provider. If Verizon wants to charge the fuck out of their customers to beef up their network so they are the best Netflix streamer out there, then they're free to do that.
People want Netflix because Netflix is doing its job. People can't get Netflix because Verizon's not doing its job. Simple as that.
To put it another way, if Netflix was the bottleneck (and thus, the problem), Verizon would only have to show that their bandwidth from Netflix is bottlenecked by Netflix' lack of serving capacity. Of course they can't show that, because that's not the case, and thus not the problem.
It's not that Netflix is 35% of global internet traffic. 100% of my ISPs "global bandwidth" should be content of the end users choosing (usually requests). The user is paying for the right to request this content, and have it delivered. Wether its Netflix, gaming, news, poltiics or Zombo.com doesn't enter into it. At 35% though, it might make sense for Netflix to enter the ISP business.
This framing is a tool that ISPs and their supporters use to excuse charging twice for the same service, and more importantly, controlling users access to content through discrimination.
Thanks again for the writing prompt though.
But to answer your question: I am already paying my ISP to deliver all packets I request to me, up to certain limits that my ISP agreed to when they started my service. If my ISP is failing to live up to that bargain, I'd like to know.
Also, the "poor Verizon, helpless in the hands of Wall Street" thing verges on obscene. That they are in a position to shake down people for more money does not justify them doing so. If they don't want to be in the ISP business, they can get out of it. They are also welcome to find new ways to create value. But running a protection racket is not in any way morally defensible.
Verizon isn't some kind of customer owned co-op. It is a profitable for-profit business that has (over)sold a service. They need to deliver that service, not complain when people use it.
I would entertain your argument only if there was consumer choice. In such case we could perhaps experiment with different business models where content provider subsidizes last mile. Without it, consumers are essentially hostage to a monopoly, which is unacceptable.
Well, actually, none of my money goes to Verizon. My provider manages to avoid all warnings from Netflix and the graphs from Google show basically 100% of users getting the HD stream. They give me a legitimate 100mbps symmetric (well, almost: http://www.speedtest.net/my-result/3546047591) and I'm paying under $40/mo.
If my niche provider can offer that service, Verizon can offer that at scale. The fact that their service is a) worse and b) more expensive means either a) they're less competent in running the business or b) they're overcharging their customers. Either way, I have zero sympathy for them. They're a network provider and people are paying them to connect to the internet. People aren't paying them to connect to the subset of the internet that's willing to pay Verizon. It's already bad enough that TV viewers are seen as the product rather than the customer. Verizon's stance here is, once again, treating their subscribers as a product to be leveraged rather than a customer who's interests matter. That they can do it is due to their mono/duo-poly that keeps customers from having a choice of providers. But that doesn't make it any less offensive.
Basically, it boils down to this: if the service costs you more to provide, then charge more to your customers. You can even have a streaming plan that charges customers who stream more than customers who don't. But treating your subscribers as a resource that can be pimped out to all the services on the internet is offensive.
They have subscribers because those subscribers want to use stuff like NetFlix.
Take away all the good 'content' and why bother having an ISP? Mobile phone can check emails too.
Verizon is paying to provide its own customers with what they want (speed that enables access to Netflix and other entertainment), not to subsidize Netflix.
Netflix pays its for its own high-speed internet access already. Verizon offers high-speed access and then doesn't pay for it. Therein lies the conflict.
In other words, the ISPs are doing a bad job of running their businesses, but rather than do a better job, they want others to pay them more without offering any additional value.
Netflix isn't the driver, the Verizon customers who want to use netflix are the driver. They're paying Verizon to get them access to cool stuff like Netflix.
It sounds to me like those customers are not getting the service they expect.
However, I feel that this issue is one of misrepresentation. Netflix wants Verizon to be a common carrier, and have its revenue increase linearly with the amount of data it transfers. In that case, Verizon would happily increase their revenue by providing actually faster Internet, because customers would pay more per day if they downloaded more per day, since they'd then pay a fixed amount per GigaBit.
On the other hand, I don't understand what Verizon wants to be. I feel like the type of company it wants to be is only sustainable in the very short run, highly dependent on other companies such as Netflix, and incapable of good PR because what clients want, and what they subscribe for, is impossible to deliver (the amount of data clients want to transfer is always above whatever they can promise to offer). That market positioning is tolerable for instant Wall Street growth, but it is detrimental to customers, to website owners, to peering partners, and to the long-term future of the company, were a reasonable competitor to enter the market.
I also feel there could be a middle ground, such as Verizon asking customers from every city to tip in for a massive city-wide network upgrade. That would be good for PR and customers would happily donate. It seems fitting, because from what I understand, the real money eater is a one-off upgrade; it doesn't increase maintenance costs much, if at all.
You buy an Internet connection from an Internet service provider to consume the Internet. Until the word Internet is dropped from “ISP”, that's the service they sell.
They have no more rationale to charge Netflix to “access” Internet service purchasers than they have to charge your grandmother to deliver you her email.
And about that profit motive, cable already raises prices at 4x consumer price inflation:
“Since the FCC began compiling these reports 19 years ago, the cost of basic, expanded basic and the next-most-popular tier of service have risen at average annual clips of 4.3 percent, 6.1 percent and 5.1 percent, respectively. Inflation has grown at an average annual pace of just 2.4 percent over that time period.”
http://www.siliconbeat.com/2014/05/19/pay-tv-prices-continue...
Because Netflix has a higher brand affinity and customers of Verizon will be outraged due to what they perceive is shitty customer service buy a quasi monopoly.
The consequences is damage to Verizon's business because customers would switch if they could, but they can't.
Verizon doesn't HAVE to subsidize Netflix's business but they should if they don't want a legion of pissed off Netflix customers who feel trapped by their ISP.
This is supremely dangerous to Verizon.
Taking the US political area I know best, gun owning households are a majority in the US, but of course those who vote, vote first and foremost on this issue, let alone be politically active are the usual successively smaller fractions of that huge number.
And starting in the mid-80s we've been able to move mountains (details on request). The last thing the big incumbent landline last mile providers want is a large percentage of the population getting pissed off and doing the above, voting when they might not otherwise bother, especially in state and local elections, voting on this issue, and becoming politically active on it.
I think the incumbents' behavior is so outrageous that this is eventually inevitable, Netflix's PR attack, in part enabled by their being so big, has the potential to substantially move that day of reckoning forward, and is therefore probably the biggest thing to happen in the net neutrality fight since AT&T's Edward Whitacre opened his foolish mouth and really got the issue started 8 years ago.
ARPU is going up while Verizon's bandwidth costs are going down (peer with Netflix/Google/YouTube/Facebook and a huge chunk of your traffic is taken care of).
http://newscenter.verizon.com/corporate/news-articles/2013/0...
"FiOS revenues grew 14.7 percent, to $2.7 billion in second-quarter 2013, compared with $2.4 billion in second-quarter 2012."
vs.
"Verizon added 161,000 net new FiOS Internet connections and 140,000 net new FiOS Video connections in second-quarter 2013. Verizon had a total of 5.8 million FiOS Internet and 5.0 million FiOS Video connections at the end of the quarter, representing year-over-year increases of 12.2 percent and 12.6 percent, respectively."
(Revenue growing faster than new customer growth and a very high ARPU.)
Also, I find it interesting that netflix charges ~10-15 bucks a month, while the average internet only service plan in the 40-50$ range per month. This is a guess, but I would also say there are more Comcast/Verizon/Time Warner customers than netflix customers. The amount of money these large service providers rake in must be an order of magnitude higher than Netflix.
To me, this money that the big ISP's are trying to get from Netflix looks like some sort of fee applied for forcing them to actually provide the service they promise to their customers.
The message is "The Verizon Network is Crowded Right Now." not "Verizon is Deliberately Slowing This Right Now." (or some less loaded version of that)
It's bad PR for the ISPs, but combined with the fact that most people can hardly spell ISP, and that Comcast and their FCC homies keep describing the new toll booth as a fast lane, if I were a nontechnical consumer my first response would be, "What the hell are they waiting for? Put in the fast lane! I wanna watch my damn movies!"
Considering that customer satisfaction with the average telecom company is somewhere between that of the DMV and a crooked tax collector, it plays into the confirmation bias people have that they don't love their ISP, they tolerate their stupidity because its the only way to get online.
Telling a customer they're not getting what they paid for from their service provider is about as plain English as you can get. I daresay we need this sort of rhetoric, and I doubt their response will be levied toward Netflix unless they start getting letters from their ISPs saying "We're throttling you because you're watching Netflix and they won't pay up."
This situation could get ugly, though.
Verizon doesn't want more "cord cutters" of a sort to drop their FiOS video accounts in favor of a la carte offerings from Netflix et. al. AT&T's U-verse isn't as big a thing for them, but they use vicious caps to protect it, and of course all the cable companies are by definition video service providers.
If I pay for a 100Mbit/sec connection from my ISP why doesn't NetFlix send me 100Mbit/sec of data? They choose to limit my speeds (via streaming) and do not offer any options to download or pre-cache video content I plan to watch. Is this a violation of network neutrality? If I ask NetFlix for 4GB of data why are they allowed to limit the speeds?