Level3 is without peer, now what to do? | Hacker News Reader

Level3 is without peer, now what to do?

cringely.com

781 pointsmortimerwax382 comments

There is an interesting unbalance because Comcast has so much leverage by owning the last mile, they can push around Tier 1 providers. I'd like to fix that, mostly by creating a public policy around municipally owned Layer 1 infrastructure between customers in their cities and a city exchange building. Conceptually it would be no different than the city owning the sewers and outsourcing the water treatment plant to a contractor (or two). Creating a new "ISP" would involve installing equipment in the City Exchange(s), providing compatible customer premises equipment to subscribers, and then patching their 'port' at the City Exchange to the ISP's gear.

Its going to be a long conversation :-)

Sounds like the situation in the UK which largely works quite well.

This is how it should probably be done if we were starting from scratch. A system like the one you describe would cost somewhere in the neighborhood of $300-500 billion; which isn't completely insane as far as national infrastructure projects go.

The problem is that we're not starting from scratch. It's a lot harder for the government to justify that expense when there are a number of viable privately-owned alternatives. Or to put it another way: if you had $500 billion to spend and you had to choose between overhauling the nation's education system (which desperately needs it) and providing better Internet access to people, which would you choose?

Furthermore, do you trust your city to properly maintain said infrastructure? Because I sure as hell don't. All I have to do is look at the pothole-strewn roads outside my house to know that. Fiber optics are awesome now; but what happens in 10 years when we need a new type of quantum fiber made from carbon nanotubes to go faster? Will the cities be able to make the investments? I have my doubts given the overall poor state of municipal infrastructure around the country for things like water and sewage.

The system we have now is far from perfect, but we're going to need to figure out how to make something work within the confines of what we have today: a privately owned and operated physical infrastructure controlled by a small number of companies. How do we craft regulation such that those companies are more profitable when they do what we want them to do?

> if you had $500 billion to spend and you had to choose between overhauling the nation's education system (which desperately needs it) and providing better Internet access to people, which would you choose?

They'll choose D. Build a new sportsball arena

That would be enough money for 500-1000 large stadiums.

Citi Field in New York: $900 million

Allianz Arena in Munich: 340€ million

> The problem is that we're not starting from scratch

But actually we did have an opportunity to reboot last-mile networks from scratch, because FTTx is essentially all-new network.

And FTTx could be done at considerably lower cost, if we simply had the will to do it, and the patience to leverage existing work.

For example, last week, they were tearing up the street in front of my building to put in new sewer lines. I don't even want to speculate what that might have cost. But now that the work is done, to lay fibre, they'll have to do it again. If, instead, they'd said, "Well, while we're here, let's just drop some conduit for fibre", and continued to do that going forward, a national FTTx last-mile infrastructure would cost considerably less.

It wouldn't be free, of course. It's not merely "[dropping] some conduit", and there are plenty of places where they'd probably have to tear up the streets just for this purpose anyway. But, if every time we tore up the street for unrelated work, we leveraged that to put in place the FTTx-relevant things we'd also need to tear up the street for, we'd be so much closer, and at so much lower a cost...

You think people who tear up the street to put in sewers have never thought "hey, while we are at it, let's build some conduit down here and then sell that?"

I know they literally work on sewers, but they aren't dumb. They understand their business better than random people on the Internet do.

So please tell us, why is it not done? Contract provisions? Lack of profit?

Both.

Most common, SewerCo says "pay for half of our digging costs" and FibreCo says "You're getting paid by the city to dig it up already! Why should we pay again?"and round and round we go. Also, if either SewerCo or FibreCo's work has delays, then there's another whole drama about who pays/is responsible for additional work, penalties, and so on. And the respective Project Managers don't give a rat's ass about the other company since hitting their budget/timeline is the only thing that matters.

Seems like something that can (and should) be worked out, but unless local government steps in and mediates/forces their hand there doesn't seem to be a great urgency. Locally (in NZ) ElectricCo and FibreCo are starting to be the same company in a number of cases, in which case "stop arguing and just do it" is just a management edict.

This is why the municipalities--in the United States--that have city-operated utilities should be doing this. In Seattle, for example, Seattle City Light is the legal owner or half-owner (with the local telco) of every utility pole in the city, along with ground easements for electrical service. Since Seattle City Light and Seattle Public Utilities (the water and sewer provider) are two departments of the same city, they share the same easements and access rights. Tearing up a street to rebuild the street, reconstruct a sewer, or bury electrical wires is all done by the same entity, so adding in fiber conduits is easy as long as there is the budget.

Sadly, that last sentence is the kicker. "As long as there is the budget." There's no line-item in the city budget for building a municipal data network that covers the last mile (Seattle has a rather sizable not-quite-last-mile fiber network).

My brother used to do utility plumbing — literally, laying sewers. I know they think this. It's not them that's the issue; it's short-sighted municipal governments, it's contracts, it's "Well, who's paying for it?", it's liability issues, it's these and countless other little inanities.

That said, it kinda blows my mind that someone on HN would actually think I was talking about the people laying the sewer line when I said, "Well, while we're here..." Were you straw-manning, or did you really just not think that one through?

Sort of, but doesn't a lot of it reuse existing routes? Depending on who "owns" it (based on leases and contracts), this could limit the ability to make a clean break at this stage. If you had to sublet the routes, would it be cost effective? If you had to dig new trenches, how many and how much would it cost?

Good points, allow me to share with you a bit of my strategy.

There is a case to be made that much of the tearing up of the streets is a function of people laying down multiple communication infrastructures along side each other. We looked though the permits for the last 12 months and compared that activity to a on-going repaving and road maintenance work. There are several interesting variables at play in that, cost to maintain (tax based), property tax (revenue), new permits (revenue). If you consider switching the physical layer to an infrastructure basis you eliminate multiple fiber/copper paths under the street, multiple fiber/copper vaults in sidewalks and on street corners, and replace them with a common set of conduits. That helps a number of costs, site surveys for digs, delays for city work, and multiple trenching/paving jobs with different standards of integrity. My city, Sunnyvale, is dealing with ways to make maintaining the city more efficient. This sort of change helps that in ways that are not immediately obvious to folks because they happen with dozens of different actors across dozens of different departments.

My goal is to get the city thinking differently about communications (more like roads and sewers than 'optional thing some folks have') so that we can move past the current hodgepodge into a more efficient and maintainable system. If I can get my city to change, then other cities can see the change and see how it helps or hurts and decide independently to change. As a particularly tech savvy city we have a strong alignment with citizens who want better communications, and we've seen from places like Kansas with Google fiber that such communications upgrades increase property values, which increases property tax revenues. My hope is we can get everyone (communication providers, city, citizens) all aligned on this idea. If so it will get done.

This is all well and good if you're talking about relatively wealthy cities like Sunnyvale with a good tax base, but many cities like Detroit, Pittsburgh and Philadelphia are literally crumbling. They operate much more in a "try to keep the lights on" capacity because they just don't have the money to do anything else. The roads are constantly torn up from traffic and just never get fixed, water and sewer don't always function properly and the schools are an utter disaster.

Basically, many cities have much more pressing problems than telecom infrastructure. Not to mention that federally, the FCC holds jurisdiction and thus final say over what cities can and can't do. What may look like standardization to you in Sunnyvale starts to look like a hodgepodge of municipal implementations at a federal level, and the FCC may stop you from standardizing at the local level until they've developed standards at the federal level. It's bureaucracy at its finest; but it does serve a purpose.

You have to start somewhere. Sometimes an incremental approach is the easiest to get rolling. Framing it as a national issue likely makes it more complex, however.

A couple of points, first I talked with a lawyer who works with/for the FCC and established that if the city isn't involved in lighting up the pipes they don't care what they do. The regulations would apply to ISPs using those pipes but it would not apply to the folks maintaining them. He used the example that companies like the old AboveNet which ran dark fiber from place to place and kept it maintained were not liable to FCC regulation.

The second thing that is interesting is that having a city maintained connectivity infrastructure actually has some positive social justice aspects as it starts to chip away at the digital divide. Once existing contracts have expired and the city is in a position to change things (current single provider contracts don't allow for this) a charitable organization could be an "ISP" without the cost of maintaining the physical plant to/from the subscribers. This greatly expands the available market and gives at risk populations more accessibility. Currently kids study at the city library (which has publicly accessible Internet) rather than at home.

Keep them coming folks, these are exactly the sorts of things we're trying to surface and look at.

How do we know that wired broadband is even the answer? Wouldn't wireless connectivity be more attractive if the goal is social justice? Many lower-income homes don't own a PC, and at this point they probably never will. How attractive is a home PC to you if all you've ever known are smartphones?

Wired broadband, to me, is more of an entertainment consumption platform. Wireless serves the needs of low-bandwidth information consumption quite well these days, and at a significantly lower cost and better market dynamics. Wouldn't a city be better served by pushing Wi-Fi or 4G? Especially given the mass adoption of smartphones by all segments of the economy.

The laws of physics prohibit wireless from serving current needs in an urban density. It's not a solution for the future.

Are you able to expand? I tend to agree but dont really know why or what / where the physical limits are. Is it the 4g etc tech that can't perform with real load or WiFi that will fall over? Is a mesh network an answer?

The problem is that wireless is a shared medium. If you have a certain amount of spectrum, you can transfer a finite number of bits per second using it, and that capacity is shared between all devices in range. The bandwidth available in a certain chunk of spectrum is a physical limit. There is no way to expand capacity other than by allocating more spectrum, and radio spectrum is extremely scarce. So the more users you have using cellular for internet, the less bandwidth they each get.

The solution when you want to have "wireless" devices but there isn't enough wireless spectrum is to use wireless for the "last 50 meters" rather than the last mile. In other words, you have fiber to the premises and then WiFi within the building. By keeping the wireless power levels as low as possible you reduce the number of other users you're sharing the spectrum with and allow each user to have a satisfactory amount of bandwidth. Then reserve cellular wireless for users not in a building, e.g. accessing a map from your car, and have all manner of stupid bandwidth caps to discourage usage of cellular wireless whenever it isn't strictly necessary.

> Is a mesh network an answer?

No. Mesh networks would allow you to have more bandwidth per user by reducing wireless power levels and shrinking the geographic size of the collision domain, but that would require peers to act as repeaters between other peers and the tower. That's an epic fail. It adds latency, reduces battery life, impacts reliability if any peer's device is flaky, has security implications, etc. Wireless mesh networks ironically work much better for fixed-position devices than mobile devices, because they can use directional antennas, don't have the extreme power constraints mobile devices do, etc.

Thanks mostly confirming what I suspected. I did think we would be able to use the spectrum more efficiently over time with better, more precise measurements but my guess is that will scale to its own limits a lot faster than a hard line will.

Much like procesing power and moore's law, we're getting very close to shannon's law (http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem.

Wireless isn't this huge gamechanger that people may think. To beat shannon's law you would need so many base stations in the long term it looks awfully like what we have now with DSL/Fiber/DOCSIS and wifi APs.

It all depends how much bandwidth you need for a given use case. For most use cases other than video, existing wireless technologies are sufficient and will only continue to improve as things like beam shaping and micro cells become more prevalent. Wireless is easier to deploy and as its use grows, more spectrum will likely open up. Free markets have the benefit of deploying scarce resources where they are most in demand. Implementing that spectrum is often as easy as a configuration change that can be deployed remotely.

The biggest advantage of a wired home broadband connection over a wireless one that I see is the ability to stream video. A wired connection has a lot of disadvantages: it's useless outside the home and the infrastructure is significantly more expensive. Wireless has signal and coverage issues for sure, but wireless is a much more competitive industry than wired access in no small part because it's cheaper to deploy.

I don't think it would be going out on a limb to say the HN audience is much wealthier and more technology-literate than the average citizen, and thus would be more likely to want multiple Internet connections and have more use cases such as running file servers, etc. I think the HN crowd overestimates what people use their Internet connections for; the average person uses their home broadband connection primarily to watch video. It's not a bad use case by any means; but I'm not sure the government needs to be building infrastructure that will be primarily used by video streaming companies. Traditional cellular wireless (which can use any mix of wireless, wired or satellite backhaul depending on the application) and Wi-Fi are flexible enough to cover the "access" part of Internet access; especially in an increasingly post-PC world.

> For most use cases other than video, existing wireless technologies are sufficient and will only continue to improve as things like beam shaping and micro cells become more prevalent.

Excluding today's most demanding use case doesn't get you anywhere. If you need a wired connection for that then you need a wired connection. The installed base of smartphones and tablets is still growing significantly and that's going to eat any technological improvements you can make to cellular and then some. Meanwhile people still want to be able to watch Netflix on a tablet from the back of a minivan, and to be able to make that happen at scale you're going to have to remove as much other traffic from the cellular network as possible.

> Wireless is easier to deploy and as its use grows, more spectrum will likely open up. Free markets have the benefit of deploying scarce resources where they are most in demand.

Free markets can't change the laws of physics. Allocating more spectrum is a linear increase. 20% more spectrum costs billions of dollars and only makes it 20% faster. The market solution when there is a high demand scarce resource is to shift usage to a resource that isn't as scarce for any use case where the alternative resource is viable. There is a reason one of the first things you already do when you get to a friend's house for the first time is ask for the WiFi password, and it's not because there is no cellular coverage.

> The biggest advantage of a wired home broadband connection over a wireless one that I see is the ability to stream video.

The advantage is that it allows you to transfer more bits cheaply. Today's most prevalent use case for that is streaming video, sure. It's hardly the only one. Anything that requires uploading or downloading large amounts of data is a bad fit for cellular. And we can't predict what interesting applications will become commonplace once the typical connection is 1Gbps.

> A wired connection has a lot of disadvantages: it's useless outside the home and the infrastructure is significantly more expensive.

The infrastructure is significantly more expensive, once. The cost to maintain fiber is not materially different than the cost to maintain copper. And it's not useless outside the home when everybody else has one too. If you're at home you have WiFi. At work? WiFi. Friend's house? Coffee house? Hotel? WiFi. About the only fixed places there is no WiFi are the places there is also no cellular, like the middle of the woods.

Using cellular when you're in a building with WiFi doesn't make sense. Using cellular when you're on the move does -- and that's the use case we should be preserving the scarce wireless spectrum for, by using wired connections everywhere else.

> The cost to maintain fiber is not materially different than the cost to maintain copper.

You're right; and costs to maintain copper run about 5-10% of initial costs PER YEAR. Physical networks with lots of endpoints are very expensive to maintain.

> You're right; and costs to maintain copper run about 5-10% of initial costs PER YEAR. Physical networks with lots of endpoints are very expensive to maintain.

We already have physical networks. Maintaining them is a fixed cost.[1] You can't eliminate it because you can't reasonably expect to replace the entirety of Cable TV and U-verse with cellular, so the money might as well be going to maintain fiber rather than copper.

[1] http://en.wikipedia.org/wiki/Fixed_cost

$300-500 billion is a lot to install a national fiber network because "we might as well". If they're going to cost the same to operate, what do we get in exchange for this $500 billion investment? The ability to watch Netflix videos at a higher quality? A slightly lower cable bill? (since the cost to maintain won't be significantly lower, the bill won't be either)

I'm not trying to be sarcastic here -- this is how the government thinks about things. We have an existing infrastructure that is less than optimal, but it still functions and is in no danger of imminent collapse. Contrast with our healthcare system which barely functions and is on the verge of effective collapse. Or our public schools and police departments, which have collapsed in a number of cities.

In short, it's hard to drum up political support for large infrastructure projects even if you can show the need. There has to be a demonstrable catastrophic impact if we don't do the project, which just isn't the case with broadband.

The http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem limits the information that can be transmitted through a given channel. If you want to send more wired data, you add more channels by adding more wires. Wireless doesn't have that luxury.

Strictly speaking this is true, however you can add more base stations with lower transmit power so the shared media is shared by lower number of clients. This is what is already done in urban areas with smaller base stations.

Wired connectivity certainly isn't either. I find myself using my home internet connection almost exclusively for movies and games -- not exactly the great innovative services everyone talks about.

I use my phone for almost everything else: online shopping, communication, news consumption, even work since I tether my laptop to it when I'm on the go. It's a lot more convenient; and the only thing I really need high bandwidth for is media playback.

How do you think your wireless signals get to the rest of the world? Microwave backhaul is almost dead, those cell towers (and there are tons of them in urban environments, partly for want of spectrum) are wired up with fiber.

exactly. Plus even if I'm on my phone I prefer the speed and reliability of my own Wi-Fi.

Of course wired is used for backhaul between wireless nodes and the internet; but I'm speaking as a consumer technology. Wired access is definitely here to stay for infrastructure uses.

Wireless serves the needs of low-bandwidth information consumption quite well these days, and at a significantly lower cost

How do you figure? I pay $30/month each for wireless & wired services. My wired services are unlimited; wireless is capped.

Its a reasonable question, the challenge is spectrum management. As the city would provide layer 1 (connectivity) not any above layers, they would have to become the spectrum manager for wireless base stations. That is percieved to be too complex. Also this has to meet the needs of consumers and businesses alike. Currently fiber scales from 1mbit > 1TBit [1] so it has that flexibility. Maintenance of the fiber is independent of the special laser you would attach to it if you were say, the new LinkedIn headquarters, versus a consumer.

Not that fiber is particularly "easy", looking over street maps and figuring out what sort of bundles/routing you need to get full coverage is quite an exercise in itself.

[1] http://www.lightreading.com/optical/400g-terabit/euronews-bt...

If it's too complex for a municipality to manage the right technology, is it the right thing to deploy the only technology they can manage? There are market solutions in place for wireless; and spectrum management is already handled by the FCC (would a municipality even have jurisdiction?)

I guess I remain unconvinced how this is not effectively a subsidy to the media companies (Netflix et. al; including Comcast once they start offering a cable-over-internet product) given that the primary use case of wired high-speed internet access is overwhelmingly streaming video (since over 80% of Internet traffic is streaming video). I also don't imagine it would lower costs significantly over a company like Comcast in the long run; the margins aren't great on a business like that and line maintenance is much more expensive long-term than people realize.

I agree with this approach. This cost does not need to be born by the municipalities alone. Contracting companies will line up to build and maintain the infrastructure. The initial build out could be paid for by bonds which would be repaid by the ISPs who lease access. The ongoing maintenance would be paid out of the same fees. ISP competition would help insure that the whole endeavor would not be too costly.

Anything that I can do to help move this forward as a Sunnyvale resident and voter? Thanks!

Exactly, TBH I don't understand why we don't have the conduit equivalent of the shipping container. i.e. a completely standardized, royalty free design for laying cables in urban environments that are completely interchangeable and manufacturable by any company.

From there, the city can just lease the space on a basis of (meters * diameter * demand coefficient for a particular route)

Does such a standardized conduit design exist?

Sort of, there are lots of standardized pipes and conduits which are spelled out in building codes. I've seen installs where the company doing the install just digs a 20" deep trench and unrolls fiber into it, and installs where they put in pipe and then pull fiber through it. I would be really pleased if, by having a known quantity of installs, we could get some work done a sort of standardized civil engineering plan. Part of the permit process with conduits is that the city requires the conduit not crush under some multiple of the max load the street is rated for (assuming its under the street). That involves some number of billable hours by a civil engineer and some test specifications from companies. If there are more standard versions of these things that can be pre-approved for a wide variety of situations, it will reduce the cost of installation.

Yes, but there is still a very heavy cost in maintaining these conduits. They often collapse under a road (for example), which means closing the road off, digging it up, and then repaving the whole thing which is very expensive to serve a few houses or whatever.

In California, I feel like 'increases property values' is not that valuable to cities due to prop 13. Big increases in property values serve to lock people into their current homes because they lose their property tax subsidy if they move.

> The problem is that we're not starting from scratch. It's a lot harder for the government to justify that expense when there are a number of viable privately-owned alternatives.

This is what the condemnation process is for. The government is unlikely to have to spend $500B to rebuild it; probably much less for the depreciated capital.

> Furthermore, do you trust your city to properly maintain said infrastructure? Because I sure as hell don't. All I have to do is look at the pothole-strewn roads outside my house to know that.

I wouldn't judge that quality of municipal infrastructure maintenance by the number of potholes, because fuel taxes pay for road maintenance, and many jurisdictions haven't raised fuel taxes in decades (i.e. they're not keeping up with inflation).

Instead, you might look to your water, sewage, garbage and electricity services - there, public utilities score quite well, and it's because they're usually not operating at a significant loss.

"How do we craft regulation such that those companies are more profitable when they do what we want them to do?"

Sounds like the U.S. already has such regulation - anti-trust regulation. As ChuckMcM noted above, the problem is that Comcast has a monopoly over the last mile. Effective and enforced anti-trust regulations would break up this monopoly into several different competing companies.

BTW, here in Israel, the government apparently has taken the position that services requiring significant infrastructure investment (Internet or cellular networks) are best handled by handing out a limited number of licenses using a bid. We then get an oligopoly where the companies are wildly profitable, until the government has decided we got the infrastructure we wanted, and removes the barrier to entry. Specifically for Internet, we have a duopoly over the last mile "going until the city exchange" to put it in ChuckMcM's terms, then a lot of ISPs taking it from there. IMO, not a bad way to go - we don't pay a lot and get decent Internet everywhere in the country.

Anti-trust laws are not ideal, though. They effectively act as thresholds or hard lower limits on allowed competition, but they don't directly address the economies of scale and barriers to entry that push the market toward a monopoly state.

Agreed. I have been looking now for a while for a rigorous treatment of how to deal with monopoly-related problems (hopefully this would be a summary of the current consensus among economists), are you by chance familiar with one?

Haha, no, I'm not. I only have a 1-year undergrad intro econ course plus a mathematics degree and my own intuition. I don't think I've actually read any primary literature in the econ field, only my textbook (and not much of that, either!).

True, the original impetus for the market monoply exclusions that exist today was to insure that someone coming into the market had a reasonable chance of making their money back. This encouraged people like Comcast to bring cable to the city. What has changed is that the Internet which was a 'ride along' on the Telephone and Cable monopolies has become more of a lifeline service. Further it has gone to the point where it supercedes both, since a home with a 1 - 3mbps Internet connection can get both phone and television services over it. This is the message that I'm bringing to the city, which is that Internet pipes have become like roads, essential for the citizens and the proper functioning of the city itself.

And given that, lets move the infrastructure into the same place, city owned/maintained raw pipes, company services accessible by those pipes.

I don't think that is a hard choice at all. I've see what happens when the government tries to 'overhaul the education system', and it isn't pretty. I'd rather they stay out of that market and leave it to the local governments. Laying fiber on the other hand, that they may be able to accomplish.

> Fiber optics are awesome now; but what happens in 10 years when we need a new type of quantum fiber made from carbon nanotubes to go faster?

Fiber optics are awesome now. Fiber optics were awesome in the 2000s, and the 90s, and the 80s, and the 70s...

Quantum fiber isn't a thing. Nobody has even proposed anything to replace fiber with and copper has lasted more than a hundred years. You can get Tbps out of fiber so we should be good for a while. If somebody invents something to replace fiber with, we can deal with that when it happens (probably a hundred years from now).

Thanks for saying this!

It's worth pointing out the economic benefit to whoever invents the Next Big Thing -- if they make it compatible with existing conduit, it can just be pulled through replacing existing infrastructure.

It turns out the only irreplaceable part is the tubes.

Think datacenters, high density offices and apartments, Level 3's long haul links, etc. They will all be looking for a way to pull Next Big Thing like it was fiber.

> Or to put it another way: if you had $500 billion to spend and you had to choose between overhauling the nation's education system (which desperately needs it) and providing better Internet access to people, which would you choose?

Pretty sure that's a false dichotomy, there's no reason you can't have both (given a few years).

US military budget is ~$650 billion. Is that more important than investing in an educational system that desperately needs overhauling?

>Furthermore, do you trust your city to properly maintain said infrastructure? Because I sure as hell don't.

I hear this criticism quite a bit, but I really don't understand. Do we have any evidence at all that the government is actually less efficient, less trustworthy than private companies? The problem we're discussing is caused by private companies not living up to their responsibilities. There are endless examples of this other places as well.

Sure, there are dangers inherent in public services that are not present in private, but the inverse is true as well.

What we're talking about in this specific instance is infrastructure that is a societal benefit, and if that doesn't fall into the domain of public government, I don't know what does.

Let me just point out that $500 billion is larger than the entire yearly non-defense discretionary budget of the US, itself the largest discretionary budget of any government.

>But criley2, how do we spend trillions if $500B is larger than our budget

Discretionary budget, not mandatory spending like social security or welfare.

$500B is an enormous sum of money, it truly is.

$500B/10Y would still be an incredibly large sum subject to an incredible amount of corruption and graft...

So, like Singapore? http://en.wikipedia.org/wiki/Axia_NetMedia#Singapore_-_SingT...

IIUC that's similar to how the electricity grid works in Texas - the last mile is public, but then there are dozens of power companies that contract and subcontract to supply the actual juice. It is, reportedly, a disaster. You get small new companies popping up all the time and dying through mergers and acquisitions. Some own just a single power plant, some own nothing physical at all and are just a reseller to handle billing & arbitrage or broker cost differences. If there's a problem theres's nobody to call and nobody whose responsibility is fixing it. Consumers are left in the lurch whenever there's a merger or acquisition, and the company they pay their bills to usually bears little resemblance to who they initially signed up with.

(Come to think of it, before the wave of banking consolidation in the mid-2000s, this also described the banking situation in New England. My college bank account went from Bank of the West to BankNorth to TD BankNorth in the time it took me to graduate, and my credit card went from BankNorth to TD BankNorth to FIA Card Services to Bank of America in the 3 years I stayed in Massachusetts after graduation.)

IMHO the root problem is that consumers have no leverage because there are millions of them and none own any of the means of production, and so I'm more bullish on new technologies like mesh networking as a way of solving the problem.

Local utility companies and cable companies typically have terrible reputations across the board. I'd rather have competition and bad customer service/experience than no competition and bad customer service/experience.

A lot of this is not true. I live in Texas, and electricity is not a disaster. You can choose from a variety of rates and providers (you can visit powertochoose.org to see the site), but the local electric utility provider (the company that used to have the monopoly) is the guy you call when electricity goes out.

In the D/FW area, that's Oncor (and is for most of Texas).

Interesting. I'm reporting on my sister's complaints, for providers in the Houston area.

Edit: Hmm, interesting. Perhaps they just picked an unlucky initial provider and then got gobbled up successively.

I'm in the Houston area and haven't had these issues.

- You get small new companies popping up all the time and dying through mergers and acquisitions: to some extent this can be mitigated through your choice of provider. I haven't gone through one merger/acquisition, and I've used 3-5 different providers in ~10 years.

- If there's a problem theres's nobody to call and nobody whose responsibility is fixing it: I don't remember how it was at my apartments, but since I've moved into my house, I call Centerpoint to fix all power problems, regardless of whose service I'm actually using.

I think that is exactly what it was done in Viladecans, Barcelona: http://www.diba.cat/web/ri/newsletter_bcn_int_reg/-/butlleti...

But the telcos aren't going to like it: http://www.telecompaper.com/news/telefonica-urges-viladecans...

This is similar to what we have in the UK. The organisation responsible for last mile connections is Openreach, and for all its failings, we in the UK are served rather well by the setup. I have the option of six ISPs, 4 national and 2 local.

The issue with Openreach is the odd situation it has found itself in after several government reshuffles and countless legislation; leaving it under BT Group's umbrella but accountable to the government regulator Ofcom. As such the network has waned and become difficult to upgrade.

I'd hardly say it's difficult to upgrade, they've been rolling out network upgrades for the last decade, starting with the 21cn network, then fttc and persumably ftth as an eventual goal.

They are rolling out FTTC, however slowly, and there was a lot of fighting, bureaucracy and political bargaining to do so; in addition to ruling out widespread FTTH rollout (only available to new-builds and selected rural areas).

> The organisation responsible for last mile connections is Openreach

About 30% of UK homes are also served by ex-Telewest/NTL/Virgin Media cable. The last mile arrangements for that are almost identical to how AIUI things work in the US: the whole line from your door to the internet is owned and operated by one company, and there is no LLU-like competition.

Yes but the key difference is the word "also". Those 30% have an additional choice over the services using OpenReach final mile connections. In the US many people only have the one service that owns the cable and the service.

Municipal governments should threaten to regulate last-mile ISPs like power companies unless they agree to let competitors use their cables. Virginia requires that competing power companies be allowed to use power lines, why not fiber as well?

Absolutely. Isn't this what the FCC's Title II / common carrier provisions are for?

my understanding is that's what happened to make long distance affordable: the maintainers of the copper had to allow other forms to connect to customers.

Utah already has something like this called UTOPIA (http://www.utopianet.org/about-utopia/). The participating cities run fiber optic lines. The city pays for the fiber optic lines (lots and lots and lots of political controversy about this!) then subscribers can pick and choose what services and from which providers they want. I'm in a participating city and love it! Unfortunately politics, both the governmental and the organizational types, are constantly threatening the viability of UTOPIA (http://www.utopianet.org/press/).

>UTOPIA (Utah Telecommunication Open Infrastructure Agency) is not an Internet service provider. UTOPIA operates on an open access model, which means we own and manage the infrastructure, but lease the lines to private Internet Service Providers, who then deliver services to subscribers. This allows you to choose the provider that best meets your needs.

edit: added press link

That is pretty awesome, I've sent them email to see if we can use some of the work they have done in this space.

My work email archive from a decade ago is hard to search, but when I sent a similar email, I got back a reference to a study of broadband in Japan (?) about early adopters in fiber-to-the-curb communities...

They essentially found that when you installed a network like this in a neighborhood, the economic growth in that neighborhood almost immediately changed trajectory (upward) relative to similar communities without these networks.

When I bought my house, one of my absolute requirements was that the house be in a UTOPIA city. If I were a business owner, I would have the same requirement.

That sounds like a similar model to Australia's endangered National Broadband Network. The government owned NBNCo lays the fibres to customers' homes, then ISPs lease bandwidth to service their customers.

It seems to be facing the same issues that you describe, with some politicians being unable to stomach publicly owned infrastructure.

Interestingly, there is an Australian corporation, Macquarie Capitol Group, that may take over the administration of UTOPIA - http://www.sltrib.com/sltrib/money/57282051-79/macquarie-uto...

Macquarie Capital isn't little known here. They are famous for being able to extract maximum fees from public infrastructure. If they live up to past form [1,2], they will turn UTOPIA into a river of gold, and the source of that river will be the customers.

[1] http://www.smh.com.au/travel/blogs/travellers-check/the-true...

[2] http://www.smh.com.au/business/macquaries-sydney-airport-pla...

Good to know!

I think we can fully expect that NBN Co. will be sold to some organisation or another just like this. That is the sort of thing the Australian government does with public assets.

God no! That is not how Labor's NBN works, nor how it is priced. Simon Hackett, the founder and former owner of Internode did a very good job of explaining by NBN Co.'s pricing model is retarded and why the network build is dumb [1]. In brief, the wholesale pricing model precludes new ISPs entering the market, and Labor's NBN is not dark fibre to the premises, again see [1], along with Simon's other talks on the subject. Frankly, the less of Labor's NBN that gets built the better.

1. http://simonhackett.com/2013/07/17/nbn-fibre-on-a-copper-bud...

That linked preso was terribly salesy. He did a poor job of actually backing up some of his graphs with data.

He has a point when he talks about NBNCo's CPE device offering. But that's not really a big deal. Most of his griping seems like nitpicking. Te point of the NBN is getting fiber in the ground that no one company owns. Once that is done all of his other complaints can be dealt with. We can argue incessently over what kind of CPE device to deploy, and change it, without altering the fact that NBNCo now has the fiber in the ground.

He also has a point about the number of POIs that NBN will require the wholesalers to interface with. But again, this is missing the forest for the trees.

>Frankly, the less of Labor's NBN that gets built the better.

Simon doesn't make that argument. His argument is to do it differently, which is completely different than not do it. In the Q&A he makes suggestions to do some parts of the NBN differently, but it's clear he is generally in favor of it.

The NBN is about getting non-monopolized fiber in the ground. Once that's done everything else about it can be changed to suit his or your desires.

> That linked preso was terribly salesy.

Yeah, I get where you're coming from. I think by the time Simon gave that particular presentation he'd been saying the same stuff on a regular basis for a good two years, so it started to sound really rehearsed. Have a look at some of his older videos eg. [1] Although, that is, to a major degree, just how Simon sounds when he talks.

I don't think you fully understand the implications of what NBN Co is building.

> Te point of the NBN is getting fiber in the ground that no one company owns.

This is exactly what is not happening. Labor's NBN Co is (was?) building another Telstra-like monopoly. The one company that owns the fibre is NBN Co. which will, like almost every other government asset, be sold at a loss to a private company. And, like Telstra, it will become notorious for dragging its feet when it comes to appropriately pricing the wholesale service.

> He also has a point about the number of POIs that NBN will require the wholesalers to interface with. But again, this is missing the forest for the trees.

While you (I don't mean you personally, I mean 'you' in the general sense of the Australian voter) are focus on 'fibre regardless of costs' then yes, sure, you're correct.

> Simon doesn't make that argument.

You're right, he doesn't. That is my argument. And, like I always said from the very first first announcement of the NBN "well that will take Labor winning the next 6 elections".

> The NBN is about getting non-monopolized fiber in the ground. Once that's done everything else about it can be changed to suit his or your desires.

Change. Yes, just like Telstra. In fact, the scenario with Telstra is better than NBN Co. At least Telstra has been forced to resell ULL (Unbundled Local Loop, ie. vacant copper) to ISPs. NBN Co. is being built from the ground up to not be like that.

WHY!

I'll take Dark fibre to the premises, or no fibre thanks.

The creation of NBN Co. even resulted in the enactment of new legislation that prohibits competing fibre network builds. Hell, NBN Co. is taking TPG to court over TPGs pre-existing fibre network! [2] This is decided not the role NBN Co. could be playing.

(edit: clarity)

1. https://www.youtube.com/watch?v=IMeYr2_rJmo 2. http://www.theaustralian.com.au/business/nbn-turns-the-heat-...

I've been trying to follow NBN Co since I first heard about it 2-3 years ago, but it's tough not being in AU. Thanks for giving me more to think about, and some more links to read.

It sounds like the NBN that was planned is not the NBN that you guys are getting, and that sucks.

Interesting, I thought no government were competent enough to make this happen. Clearly I was wrong.

Once the last mile is owned by the government / building / land owner. They should be able to choose what ever ISP they have without being locked into any one of them.

Yay progress!

Don't let it die that's a pretty significant step forward.

Regarding Lever 3 and all other Tier 1 providers I suggest they build a database of all ISP's that let their connection saturate and if this keeps happening they cut connections and ban their ass.

All ISP's on the ban list should be forced to pay a large fee to be able to connect back with any Tier 1 providers.

At the end of the day regardless of how big you are you're not an ISP if you don't have access to the Internet.

If Concast and friends want to play the who is more important game i would wager Tier 1 providers are in a better position to win it.

I can't help but think that Spotify's much derided torrent style streaming of data between users on their upload channels, would be a pragmatic workaround for netflix and others:

https://en.wikipedia.org/wiki/Spotify#Technical_information

Netflix is reportedly looking into this already: http://arstechnica.com/information-technology/2014/04/netfli...

I am also a happy UTOPIA user. I have a full symmetric gigabit pipe to my house for less than $100 per month. There are huge benefits to open choice networks.

Come to Australia and revel in the NBN. Oh wait...

> by creating a public policy around municipally owned Layer 1 infrastructure between customers in their cities and a city exchange building. Conceptually it would be no different than the city owning the sewers and outsourcing the water treatment plant to a contractor (or two). Creating a new "ISP" would involve installing equipment in the City Exchange(s), providing compatible customer premises equipment to subscribers, and then patching their 'port' at the City Exchange to the ISP's gear.

This is precisely what we're attempting to do in Australia with the National Broadband Network—with the delays, technical changes and cost estimating issues you'd expect from a project that's survived across two federal governments.

NBNCo installs the fibre, the termination devices, the optical splitters, and brings it back to a POI where ISPs then bring in their own transit or purchase backhaul capacity from NBNCo (as needed).

It also, in my view, makes the critical process of engaging local councils, utilities, and ripping up roads, footpaths and driveways much more 'palatable' to the average householder, because it's seen as a public infrastructure project and not as a private organisation expanding their 'own' reach (even if it would achieve the same end goal).

"Survived" is not a good description for gutting its potential capacity by 90% by switching from FTTP to FTTN.

Looks like it will also cost more in the end once you factor in the lease costs for Testra's existing network. Guess that doesn't appear in this governments' books and they can kick the can (pun intended) down the road.

Sorry to be dense, but it is not quite clear to me how ownership of the last mile affords such leverage. It would seem that if you need to buy bandwidth for end users, you would be negotiating transit like anyone else - I'm clearly missing something.

Pay us $x to connect our network to Y network to your Z network, our our millions of Y users won't have access to customers on your Z network, and that will make the Z network look bad to your customers.

Of course, it goes the other way - Y looks bad because they can't get to Z's customers - but I suspect the commercial customers on Z might be more likely to ditch Z rather than the end users (who probably don't understand the situation) ditching Y.

> It would seem that if you need to buy bandwidth for end users, you would be negotiating transit like anyone else

It would sure seem that way, but the recent Comcast-Netflix deal shows that it isn't necessarily that way; big ISPs can exert enough pressure on content distributors to get the distributors to effectively pay the transit costs.

Internet peering has been a giant game of chicken ever since commercial connections were allowed.

This is close to what Britain has. The last mile infrastructure is owned by BT, who used to be a US-style monopoly, and now they have to give access to their infrastructure to the other ISPs. Unfortunately, quality is restricted by the fees BT charge being too high, and their general woeful underinvestment into much of their network. Having the last mile infrastructure be nationally owned would be a far better idea.

Exactly what happened in France circa 2000. France Telecom the national operator was forced to maintain the last mile and open to other ISP at the city exchange. This has allowed newcomers like Free (Free ISP), to disrupt the high speed internet market both in costs and speeds. It seems that telecom is one area where laissez-faire leads to less competition, and worse offers for potential customers.

This is the case in most EU countries, due to EU regulation of the telecoms sector.

Ironically, it is a result of deregulation of the sector, from a situation where most EU countries previously had a single government-owned operator. As part of the de-regulation, to ensure the former public telcos did not get a stranglehold, a long range of rules were put in place to guarantee competitors access to the existing subscriber base.

Madison, WI sort of has this. It's called the MUFN network.

http://www.mufn.org

The different ISPs etc in the city can sign contracts with the city and get access to the MUFN fiber network. The city owns the main fiber network and does repair/maintenance. The city can take ownership of any lateral you build where the conduit is 3" or more if they desire. Also, they can take it if they decide to use it to hook up traffic signals, etc. It's much more complicated than I've detailed but it's really a partnership with the city and ultimately they can take ownership of any fiber you build off it if it would benefit the city as a whole, while still allowing you access of course.

> Nobody paid anybody for the service because it was assumed to be symmetrical: as many bits were going in one direction as in the other so any transaction fees would be a wash.

The justification for peering is not equal traffic, it's equal value - my customer wants to communicate with your customer. Regardless of the direction of traffic, the traffic is equally valuable to both of us because the traffic is the primary thing our customers are paying us for.

Unless, of course, I can get you to pay me for it anyway because of some unrelated advantage such as the fact that your customers can leave you more easily than mine can leave me. Comcast and others are attempting to leverage exactly that - in many regions they have no viable competition whereas Netflix and L3 are much more replaceable in their respective markets. This is a prime example of abuse of a monopoly.

Would be very interesting to see what happened if the big CDN providers just depeered Comcast for 24 hours. Would certainly cost Comcast a fair amount in customer support calls, bad publicity, and properly bring the debate to the general public.

It would also give the CDNs a fair amount in customer support calls, bad publicity, and make their paying customers angry about lost business.

I don't know about bad publicity though. To each customer support call simply say, Comcast has broken their contract with our CDN. Please call comcast. Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin, people will easily believe that, especially since in this case, it would be true. Level3 et all are enabling this behavior.

> Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin

And Time Warner is the second: http://bgr.com/2014/05/20/comcast-twc-customer-satisfaction-...

Nobody likes the cable companies. It would be very easy for someone to capitalize on that hatred to win public support for net neutrality; it's just a matter of crafting the right campaign.

This is why I'm so disappointed to see the small-medium ISPs hop right in bed with Comcast and co-sign anti-net neutrality letters:

http://arstechnica.com/business/2014/05/top-isps-threaten-to...

With the national attention this issue is getting, this is a tremendous marketing opportunity to take customers away from the big dogs.

The smaller ISPs are just following Comcast's example: "You mean we can charge multiple times for the same data without doing any extra work? Sign us up!" All last-mile ISPs will eventually realize they can hold everyone hostage, so long as there is no significant competition.

Small ISPs also like this because they typically are very heavy on paid, private internet and don't do much peering due to having small, localized networks. They stand to gain, proportionally, the most since they can charge the people the were previously paying for access.

Local Television stations and sports teams have successfully used this as a way to get cable companies to fold over contracts for carriage.

Everyone despises cable companies, most of the angst would go their way. After all, 9/10 people don't know what a CDN is!

Support calls from whom? CDNs effectively sit behind the Comcast site.

From the CDN customers. They'll be annoyed if they find out the service they paid money for isn't doing what it's supposed to do.

Since they've paid the CDN, they don't care what Comcast has done or is doing - they paid the CDN, it's the problem of CDN.

Those customers aren't getting depeered, only Comcast.

I'm not sure of a more eloquent way to say this. Comcast's customers are the CDN's customer's customers.

To play devil's advocate, those CDN customers would also be unable to access content if their power went out. But nobody would think to say the CDN isn't doing their job if Texas had a blackout, because "I'm paying you to deliver my content to Texas, dammit!"

It's a bit of a stretch, but I think that's the analogy they're going for.

Right, but that relationship, especially as applies to support and subscription, is not transitive.

How do you depeer my customers' provider without depeering my customers? How do you cut off a router's network access without cutting off access to the networks behind it? How do you blacklist a /8 but not one of its /24s without an exemption list? How do I prevent Bob from forking my code, but allow people who fork from Bob to fork the modifications he would do to my code? How do you join tables on 2 databases when you can't authenticate to the first?

I'd be FURIOUS if my CDN cut off service to part of my customer base, so they could attack one of my customers' mutual other vendors. And you better believe our ticket system would overload with angry tickets about how our web platform is broken again.

Not nearly as much as Comcast though. If I have my home internet with Comcast and I suddenly can't access Netflix while my friends on other ISPs can access it just fine, then of course I'm going to call Comcast.

Too risky, no? If Comcast said the problem was not with their network, but with bit CDNs. Customers would shift their rage towards them instead. The whole thing would turn into a PR pissing match

I'm going to go with customers being mad with who they send a big check to every month (e.g. Comcast). 99% of people only know about their ISP, not their ISP's providers.

Not a big check, but they'd also be frustrated with Netflix/Hulu/Amazon Prime whom they also send money to each month (or year). The average customer wouldn't understand that it's the middleman causing problems, not one of the two endpoints (service provider or content provider).

I guess that depends on the number of participating Tier-1 providers that pull the plug, if enough did it would be very obvious it wasn't Netflix with the issue because everything would grind to a crawl.

If ABC has a blank channel, but you can still watch CBS and NBC and ESPN just fine, the average consumer will assume the problem is with ABC.

If all the channels are blank, the average consumer will blame the cable company.

Similarly, if "the Internet is down", from the customer's perspective, they're going to blame the ISP, not Netflix/Hulu/Amazon.

That's if everyone particpates on a Comcast/TWC/Cox/etc. boycott. If my parents can get to Amazon Prime but not Netflix, they'll assume the issue is Netflix. (ok, maybe not my parents, but the idealized tech-unaware parent)

But with such massive depeering as this, practically all of the internet would be affected, given the amount of traffic other peers would have to pick up.

Not true. I work for a cable/fiber ISP. Customers always blame the ISP.

Even when the problem is the feed (like with CW's networks last month)...or when some dipshit construction company digs where they aren't supposed to.

Luckily I don't work for one of the horrible ISPs that everyone hates, but we're definitely more in the blame-receiving business than is warranted.

Folks don't understand the complexity involved in making a utility service "just work" and if the provider doesn't have hundreds of millions to invest in infrastructure (like my small provider doesn't have) it won't.

The big ISPs have no excuse for not peering properly, but this is a fight they've been having with Level3 since the late 90s. Sadly they're winning.

When the customers call you, it is because they blame you. You never hear from the ones who don't blame you.

That's not entirely true, but the majority of them do.

Wait, did you just call broadband internet a "utility"? As in a Title II utility?

We provide phone service as well/primarily.

What about the CDN's customers, ie., companies with sites that their costumers couldn't access?

Ask your mom who her favorite CDN is.

Akamai. Moms love Akamai.

I don't think it would go that way, their customers would more often say, "what's a CDN?"

They don't have to say CDN explicitly. They can simply say Netflix or Hulu. For example, "Dear valued customer. The recent slow down of your Netflix service is the result of an issue with Netflix servers. ISP_X service is fully functional. Please contact Netflix customer support for further assistance"

I'm pretty sure Netflix just paid Comcast, and maybe I'm not keeping up with the issues as they arise and I'm really uninformed about the current state of things, but if Level3 is still having a dispute with Comcast, and Comcast blames Level3's actions on Netflix just because they need to blame someone who their customers will recognize...

Forget about the gigundous case for libel Netflix would have against Comcast on millions of counts... those Netflix payments that Comcast just earned are going to stop, and their first high-profile experiment with double-dipping payments will have failed abruptly.

I assure you Comcast does not want that.

> I'm pretty sure Netflix just paid Comcast

They most certainly did; and what's more, the transit provider Netflix was using prior to paying Comcast was, IIRC, Cogent, not Level3.

So, then this whole idea that Level3 is the only one having problems with these five last-mile ISPs is bunk? Level3's wikipedia page still says they deliver Netflix. Now you're saying they don't even host Netflix. What are Level3's other major clients? I have to imagine there are bigger fish than Apple/iTunes (maybe I'm all wet), they are the only other major client mentioned on the wiki page.

> this whole idea that Level3 is the only one having problems with these five last-mile ISPs is bunk?

I think so, but it's only fair to note that it's very difficult to get accurate information about this, since all of the parties have a strong interest in not making all the details public, at least not while they're jockeying for position.

> Level3's wikipedia page still says they deliver Netflix.

No, it says (if you go look at the reference that's footnoted) that Netflix was about to switch to Level3 (from Cogent, although that specific reference doesn't say so) as as transit provider, and Comcast used that to try to extract peering fees from Level3 based on an expected increased level of traffic. When Level3 refused, Comcast went to Netflix and got them to pay for direct connection to Comcast's network, bypassing any transit provider.

Um, that would be a trade libel.

Comcast's customers have no idea what a CDN is. All they know is that the internet is slow, and Comcast is the internet to them.

Can't do that. These are corporations with vast paper trails.

The concept of "Tortious interference" [1] comes to mind but I'm certain there are other legal theories that could be employed by Comcast.

[1] http://en.wikipedia.org/wiki/Tortious_interference

I think the point is that Comcast is already engaging in tortious interference, and then trying to charge the upstream provider for the privilege. Sure, they could file suit on those grounds, but the countersuit would follow very quickly, and it would be joined by everyone.

At this point this seems to be the ONLY solution. Knocking Comcast off the internet for 24 hours would result in a lot of attention towards net neutrality and their lackadaisical attitude towards peering.

They wouldn't be knocked off the Internet, they'd just have to route their packets through less efficient paths, which would slow everything down significantly.

The Internet would more or less shut down from traffic overload if all of Comcast's customers lost all the shortcuts that Comcast has created for them to content.

Well the CDN could alternatively blackhole these networks.

This would be an unmitigated disaster. Everyone would call Comcast, Comcast would tell the story their way, and things would be even worse than they are now. They'd probably get on national nightly news talking about how their "partners" "unfairly" "cut them off from the Internet".

It would be pretty interesting to watch the spin machine in action, though. 100% pure PR spin.

That's why you throttle Comcast customers. Can't it go both ways?

The point is that if Netflix throttles Comcast customers, the customers stream from Comcast On-Demand instead. They don't switch away from Comcast because they can't, which is what makes this an unbalanced situation.

Also, don't forget that Comcast now owns NBC. :-)

What's the purpose?

We're getting to the root cause of the problem here: lack of competition.

These ISPs already have the lowest customer ratings. If the customers could switch to another ISP they would.

This stunt would amount to kicking a man that's lying down. He's already used to the shitty quality of his Comcast connection, but now he's being punished by CDN providers without being able to do anything about it.

The reason there is no competition is because of regulations and physical constraints. We need ultra wideband mesh nets. Or some kind of single strand fiber with a glue that attaches it to the street and a new law that says that laying fiber strands is a natural right of every citizen.

I very much think broadband internet should be treated as a basic utility. however, I feel that "natural right" takes it too far. it's a very high bar to reach

> Would be very interesting to see what happened if the big CDN providers just depeered Comcast for 24 hours.

Those providers would very quickly get prosecuted via collusion and antitrust laws, and Comcast would gain a moral high ground to stand on, almost certainly making the situation worse.

Well then all of the customers of said transit providers would leave and switch to one that can reach the millions of Comcast customers. As Level 3 said if there is a problem with peering amongst transit providers someone can switch to another provider. If there is a problem reaching Comcast customers there is nothing the customer can do or the transit provider as there is only one route to that customer. That is the leverage Comcast has now.

If the end users had multiple options they would switch providers if connections were unbearably slow to YouTube or Netflix. But we know they do not so regulation is needed.

If a customer is paying for an internet connection, they are paying for access the full internet, to the best of their ISP's abilities. This is the net neutrality law we need: ISPs should be compelled to upgrade their backbone links as they become congested, to satisfy their customer's demand. Congestion can be easily monitored and often these peerings are "free". (Yes there is a non-zero cost to increase switch and router capacity and to have someone plug the cables in, but it's not like Level3 is charging for the bits exchanged.) But the point is, since most ISPs are de-facto monopolies in this country, we need rules telling them they have to upgrade their capacity to meet their customers demand, if they are promising broadband speeds.

This. ISP's of the like mentioned by the OP are being paid for INTERNET access. Not access to the local network resources on the ISP's network. If I pay for an X up/Y down connection to the internet then that is what I expect. I expect X up/Y down to any host/content on the internet (given that host can reply at that speed, etc.)

We probably need pricing more directly based on bandwidth consumed, maybe with peak hour pricing, as opposed to just based on "speed" with high bandwidth caps.

Don't buy into the cable PR. They have incredibly high margins and could easily make upgrades without further gouging customers.

"incredibly high"? Comcast and Time Warner have profit margins of 8-10%. For comparison, Target has a profit margin of 3%, and Google's is 21%.

Sounds more mid-range to me.

As for your second point, that they could easily make upgrade without additional charges.. have any numbers backing that up? The last mile is expensive.

It's not the last mile that needs to be upgraded.

Like the water or electric company, users should pay according to usage. As you say, this isn't on total bits pushed through, because the marginal costs of sending bits at 3am is nil.

So the "usage" is based on the continued need to build out more equipment. Usage at peak hours, as you say.

I'm concerned that the ISPs will simply charge "big users" and pocket the extra fees, with "big users" moving from 1-in-a-thousand to 1-in-a-dozen as more people want to use more bandwidth. I trust my ISPs as far as I can throw them. (And I believe this distrust is at the heart of a lot of people's calls for net neutrality; simply put, they don't trust their ISP to play fair. And I don't fault them for that.)

But I don't mind something that targets the single biggest users and makes them pay extra. The water and electric systems would fail without that, and those are much simpler to model and plan for.

Thinking out loud, we need something that both

1) keeps pressure on the ISPs to keep on growing their pipes, and

2) puts pressure on the biggest users, which, again, needs to be a very small subset of people putting the biggest strain on the system. I can't come up with an exact number, but 20% of people being considered "big users" is too much.

I'm willing to pay for usage, sure. And I'll pay for the physical plant to my house too. BUT (and this is a big but) I'm only willing to do so at reasonable prices.

You want me to pay for the physical plant? OK I can deal. The phone company can provide me with a pair of wires from my house to their POP for about $15. I might pay $20 since coax is different than UTP, but that's about it.

You want me to pay per bit? Great! Bill me at wholesale network rates. You don't want to do that? Sorry, fuck off. I'm already paying for the last-mile transport with the physical plant charges.

https://josephscott.org/archives/2009/01/how-much-does-one-t...

Here I can buy RETAIL 2TB of bandwidth for $120 WITH A SERVER INCLUDED. I imagine that the wholesale is slightly (or much, much) cheaper than that. But let's just go with it. At that kind of pricing I can get 1TB for $60, 500GB for $30 and 250GB (the caps Comcast is talking about) for $15. If I need another 50GB (their incremental unit of charge for "overage") I should be able to get that for $3, not the $10 that they're charging.

http://customer.comcast.com/help-and-support/internet/data-u...

Now I think that a lot of the cost of that server is the actual machine, the building, the air conditioning, etc. So maybe you should cut all those numbers in half again meaning that 50GB should cost me about $1.50 and 1TB costs more like $30. This is of course assuming you're not involved in a PEERING AGREEMENT which as Comcast you are.

What about Comcast's costs for termination equipment I hear you say. Okay let's talk about that. I bought my own cable modem so they're not paying for that. They have to have another one on their side, sure. Assuming that it's the same price as mine (it's almost certainly cheaper) that's $100 or less. Considering that DOCSIS standards are usually a few years apart $100 / 4 years = $2 per month. That's not too bad.

And what about the back-haul from their POP to the peering point? There's a cost associated with that right? Yup, that's what I'm paying for when I buy Internet access from an ISP since they're not paying for the peering agreement.

Adding all this up: $20 (for the plant) + $15 (for the wholesale bandwidth 500GB ) + $2 (for the termination equipment ) = $37 per month with each additional 50GB costing an extra $3

This seems reasonable to me and I would happily pay it, so long as the wholesale price continues track the market. But with Comcast right now I'm paying more like $65 rather than $37.

For reference 500GB/mo (if it's 100% evenly distributed) is about 5Mbps. If you figure it's a bit bursty, say 50% on and 50% off that's 10Mbps. If it's 75/25 then it's 20Mbps. 88/12 brings it up to 40Mbps. So these are reasonable numbers to expect given the service I've signed up for is advertised the way it is.

Metered ISPs would be fine if the prices were reasonable, something comparable to data center bandwidth pricing. The problem is that they want to charge unreasonable prices.

This cable-menu style image from the comments is scary: http://i.huffpost.com/gen/1567010/original.jpg

Fearmongering- the internet is not going to be that easy to kill.

If they stop supplying it to us without discrimination it will. The internet is made up of plastic and metal boxes that are under other people's control - not magic.

Things that cannot be simply cut off:

    - populist revolutions

Things that can:

    - electricity

    - the internet

But the clever thing about Comcast's behavior is that the customer does not see anything like that image.

This wouldn't happen if those ISPs didn't have local monopolies. Networks should be opened by selling traffic wholesale to other companies so that they can compete for subscribers on those networks. The network owners would have more than enough money for upgrades and if they don't, downlevel ISPs will sue them.

I'm not making a claim about the veracity of the counter argument to this, but it is easy to state. If network providers had to sell traffic wholesale to other companies, they would have less incentive to upgrade them as those networks then become pure commodities. Right now the local monopolies can use their networks to sell high margin services (cable) bundled with low margin services (internet traffic). If they had to compete with other companies, namely bargain ISP offerings, the cost of ownership of the network would be non-profitable.

I've heard some argue that this is precisely why DSL has lagged so far behind cable in upgrades.

> they would have less incentive to upgrade them as those networks then become pure commodities

Pit DSL against cable, as has happened in the UK. The POTS network is owned by one company (BT) and the cable network by others. Retail customers generally have a choice of connecting to the Internet by either. BT (POTS network owner) are required to sell traffic wholesale to competitor ISPs (who then buy their own transit). This seems to work very well here, and DSL upgrades continue.

That describes almost precisely how the US system works.

[text removed since it seem to have confused the person who replied]

http://news.cnet.com/FCC-changes-DSL-classification/2100-103... "The ruling puts phone companies on the same regulatory footing as cable companies, which are exempt from having to offer access on their infrastructure to competing Internet service providers."

I'm not an expert, but I'm reasonably certain that Cable providers in the UK are also not required to provide ISP access via their networks.

No, they are not. Otherwise I'd be using one.

Yes but he is claiming the US system works like the British system...when it doesn't.

British DSL is far worse than cable in my experience. Virgin's cable & fiber offering is massively faster and has a wider footprint than DSL products I've seen.

> British DSL is far worse than cable in my experience.

Define "worse". What ISP were you using?

From a peering point of view as described in the article, there are a large number of ISPs buying transit and then selling Internet connections through BT-provided last mile DSL lines. I never suffer from peering congestion, since my (DSL) ISP pride themselves on not having any.

For the link from my local exchange to my ISP, there are multiple options, too, and my ISP monitors congestion on my line closely. I currently have no congestion there, either, and if I did, there are multiple providers available (thanks to LLU): http://revk.www.me.uk/2014/02/bt-21cn-not-fit-for-purpose.ht...

The only issue with DSL in the UK is the (mostly analogue) quality of the DSL line itself, and the way that BT (the company with the monopoly on POTS lines) manages them. And perhaps the pace of upgrades (eg. to fibre to a street cabinet and copper from there, instead of copper all the way from the exchange), but upgrade rollouts are happening (and fibre to the cabinet is already available to me).

TV is also a huge factor in the UK. Both BT and Sky need high broadband speeds to be able to offer video on demand that can compete with virgin.

The article said that their cable business is way less profitable than their internet business. I haven't done any research on the topic, but given that they have to pay content providers for cable, and don't for internet, I'm inclined to believe Cringely -- even though in a normal market it should be a low-margin business, selling undifferentiated bits for $.

If it was "selling bits for $" we would already be done.

But today's ISP capacity is unsuitable for tomorrow's ISP demands. One might hand-wave the network as "sunk costs" but the ISP has to keep on sinking those costs.

In fact, that's what Crinkley is complaining about: the ISP isn't sinking costs into its peer with L3.

Sure, so you've got depreciating capital costs on a schedule coupled with ongoing operational costs, that gets you to some amount of $/yr to operate the network, and $/year/bit in terms of network capacity.

What should be happening is competition driving down the price per bit to some amount just marginally above their costs. These costs may be high but the margins should be low. That's Econ 101 and it's what everyone who ever advocated for the free market is making a case for.

What appears to be happening is they're collecting a big fat surplus on top of that figure, and that surplus is in addition to all the wasted money in their corporations that's never had to be cut in a down quarter.

And those people are completely 100% wrong.

DSL got screwed for around 10 years by having more expensive regulatory costs than Cable.

Why yes, if you tax X more than Y, Y is going to have the advantage.

We have essentially such a system in Germany, does not work well. Thing is, that the owner of the network has then to compete with the resellers, but the resellers need to play nice with the owner. The result is a total mess of regulation to force the network owner ( T-Systems) to play nice.

Back in the dial-up days I had my choice of ISPs. Some people chose on price, I chose on quality/service, and mine wasn't that much more than the cheapos.

I can't recall any issues with the underlying phone company operator of the wires, although that may be because the wires were primarily for phone and they have to work by law (I think).

I miss the days of being able to choose my ISP. Now I have Comcast, with CenturyLink DSL an inferior possible second choice.

We have the same system in Canada and the wholesaler ISP (Bell) famously imposed unpopular usage based billing on the re-seller (TekSavvy) to kill the only advantage it had over the ISP.

http://www.cbc.ca/news/technology/crtc-approves-usage-based-...

Somebody has to pay the money to upgrade the equipment and bandwidth available at these exchange points. The very reasonable argument in this article is that the ISPs should pay that cost, which seems reasonable given that their customers are demanding it. It sounds like the ISPs are playing a game of chicken, trying to see if their peers like Level3 will throw money in to pay for the ISP to upgrade its equipment and bandwidth. That's certainly something the ISPs can try to do; on the other hand, what are their customers going to do, not use Netflix and YouTube? If a pile of customers of one ISP start reporting that they're all having a poor experience with high-bandwidth video, and there are a pile of well-publicized press releases blaming the ISP, customers will start complaining to the ISP, and they'll have to upgrade their infrastructure eventually. (And in areas where they have competition, there's an incentive to upgrade before the competitors, to avoid losing customers; while there isn't such competition in every locale, there are enough locales with more than one ISP choice to make those customers painful to lose.)

But what does any of that have to do with mandated peering requirements at the NSFnet exchanges? Who would enforce that, and why, when any two major networks can set up peering at any number of meet-me rooms? Requiring that an ISP peer as much traffic as is available or not peer any at all seems ridiculous; some ISPs will suck more than others, but that's the problem of them and their customers, not a problem for the entire Internet.

Meanwhile, I'm surprised there aren't more startups and VCs looking to bet that "new ISP that doesn't suck" is a viable business model. People are chomping at the bit for Google Fiber, which seems unlikely to grow to a national level without developing competitors. This is a space with very few competitors, and there hasn't been serious competition in that space since DSL stopped being a viable option.

Google's ability to print money in conjunction with its generally high PR ratings give it the cache to take on a project like fiber. I can't imagine it's cheap or easy to lay fiber if you don't have substantial financial backing as well as a political goal you're trying to achieve.

That said, I've used both Sonic.net and MonkeyBrains here in San Francisco. Never had to rely on a major ISP in this town, and I like it that way.

I was unaware these problems were addressed with a large cache.

It's a good thing Google has sufficient cachet to persuade public officials to work with them, though!

Heh, here I was thinking he was making a pun between cash and cachet :-)

Please excuse the typo.

Well now I feel bad. It was meant lightheartedly, and your comments were valid!

The ISPs aren't dropping the packets, Level3 is. From the article linked to in the blog: "A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers." "We"in the quote is Level3. They're dropping the packets.

Why do the ISPs have to pay for Level3's dropping of packets? Shouldn't the ISPs replace Level3 if they keep dropping packets?

Also, dropped packets isn't the end of the world. In TCP at least, the packets just get resent.

The language that gets used in all of these talks ("deliberately harming", "American business", "It's insulting") just makes me want to tune out entirely. I don't have time for chicken littles.

Dropping packets makes load problems worse, because failing users start to send even more traffic.

The opposite, dropped packets are bad because it slows the Internet down. When you need to retransmit a packet because one is dropped, the network assumes congestion and radically throttles speed to prevent the network blowing up.

This is not true. TCP will retransmit, yes, but TCP congestion control will also reduce the transmit rate in response to lost packets. Overall, proper TCP implementations will back off to approximately their fair share of a contended bottleneck. The application layer sees reduced throughput, not message loss. New TCP connections start with very conservative transmit rates, so an application which is overly aggressive in abandoning slow connections and retrying will not add substantially to congestion.

UDP doesn't necessarily have these properties, but anyone who implements a high-bandwidth UDP protocol which isn't TCP congestion control friendly deserves public shaming.

Except between buffer bloat[1] and users spamming the reload button, TCP's congestion control is not going to solve the problem.

1. http://en.wikipedia.org/wiki/Bufferbloat

That is true. TCP congestion control throttles back the speed in the presence of dropped packets.

Thus, dropped packets adversely affect throughput at a magnitude greater than the simple overhead of packet retransmission.

AFAICT, the reason why packets are getting dropped is because the ISP's equipment doesn't have the capacity to handle the traffic.

i.e. internet is a series of pipes example. Level3 has a big fat pipe with 10Gbit/s of traffic going through to the peering point, where $ISP has a pipe that goes to their network rated at 1Gbit/s. Since the traffic can't fit into $ISP's pipe, the extra packets get dropped.

Level3 is dropping packets because $BIG_CABLE_ISP doesn't want to upgrade their cross-connects at the exchange points. Level3 sounds happy to increase this capacity, they are saying that the cable ISPs aren't negotiating with them to provide more bandwidth at these choke points. It's not Level3's job to pay for $BIG_CABLE_ISP's infrastructure, when it's their own customers who are requesting the traffic over these links.

But isn't that the point? Level3 is capable of handling more traffic but the ISPs are not. So what?

It sounds like Level3 wants to enter into bigger-money contracts with the ISPs but the ISPs don't want to, so Level3 is appealing to the masses to help them rake in extra cash (to justify the expenses they've incurred upgrading their equipment).

Why does Level3 get the benefit of being David here? IT just seems like they're trying to leverage the net neutrality debate to get themselves more favorable peering contracts with ISPs, which in a twist of irony, is exactly the kind of thing which goes against net neutrality in the first place (why are Level3's packets so special anyway?).

What? Level3 isn't appealing to the masses to help them rake in extra cash. They don't need to.

Do you think Level3 just throws data at Comcast nonstop, and Comcast is helpless to stop it? Customers of Comcast ask for data that Level3 is trying to provide; Comcast can't handle the bandwidth. But the only reason the data is flowing is because the customers of Comcast want it; it's not like Level3 is forcing Comcast to accept it.

Why is Level3 obligated to pay for upgrades to Comcast's infrastructure to satisfy demands placed on Comcast by Comcast's customers?

You've got to ask yourself why Level3 is writing blog posts about this in the first place. It'd be like a bread making company calling out the sandwich shop it sells bread to for not having enough stores.

BUT WE'VE MADE ALL THIS BREAD WHY WON'T YOU BUY MORE?! JUST OPEN UP 100 NEW STORES I KNOW YOU HAVE THE MONEY TO DO IT!

Level3 invested money in their infrastructure, but Comcast, et. al. have not (yet). Level3 is losing money by having all this additional capacity, are they not? So clearly it's in their best interest to light a fire under the ISP's collective ass to try to get them to upgrade quickly.

Oh hey look, a political issue has appeared that does precisely that. Why wouldn't Level3 latch onto that movement and attempt to use it for financial gain in the form of more favorable peering agreements?

>It'd be like a bread making company calling out the sandwich shop it sells bread to for not having enough stores.

No. It isn't. It's like a bread making company calling out the grocery stores for not having enough shelf space to stock all the bread that customers of the grocery store are asking to buy, saying "People keep on asking us for bread and you can't stock enough of it".

edit to add: You keep on not understanding that the only reason the demand exists is because the customers of the ISPs want it. This isn't a scenario where L3 is just trying to push data at Comcast; it just isn't. You keep on acting like it is, and it isn't, and I can't think of another way to phrase it so that you'll understand. Level3 is not just pushing data at comcast that nobody wants, and whining that comcast can't handle it. Customers of Comcast are asking for the data, Level3 is trying to provide it, and Comcast is blocking that transaction.

edit again: It's also worth noting that in this scenario, people are paying a monthly subscription fee to the grocery store that says they can come in and get certain amounts of bread (or UP TO certain amounts of bread whatever) and yet because there is not enough shelf space customers are going hungry. And yet despite the fact that people are going hungry, the grocery stores are insisting that the bakery provide, in addition to the bread, the shelves on which to store the bread. And you're arguing that since the bakery has already spent money upgrading their ovens, their proofing rooms, and their mixers, they should also upgrade the grocery store's shelves.

The demand is not for L3, the demand is for Netflix. Level3 is pushing itself on "Comcast" (which is really a stand-in for the 5 ISPs who aren't playing ball with L3, and may not actually be Comcast at all).

Why must these ISPs play ball with L3, and not try to find competitors to do what L3 does? Isn't competition good?

Edit: People aren't going hungry (I know it's an analogy, but come on), people are watching Netflix at 480p instead of 720p/1080p. Let's try to keep things in perspective here.

L3 is not pushing. I cannot stress that enough. L3 is not pushing. L3 does not push. Pushing is not something L3 does. In the Game of Pushing, L3 is not a player. If you navigate to wikipedia.org/things_which_push, L3 is not on that list. If all the entities of the universe were put into two groups, one labeled "pushers" and one labeled "not pushers", L3 would be in the second group.

The ISPs must play ball with L3 because their customers are asking for content that L3 is trying to provide. If the customers ask for different content from somebody else, then somebody else would be providing the data.

And excuse me? Don't blame me for the crappy analogy; you're the one who brought it up. I thought it was a stupid analogy too, but I thought that speaking in your own terms might help you understand the situation.

Customers are not asking for L3. Customers are asking for content, of which L3 is one of the providers. L3 is pushing L3, even if you say they're not.

Without customers asking for content, L3 wouldn't provide anything to Comcast. It's at the specific request of the customers that L3 relays the replies. The customers initiate the request, and L3 provides the response.

Without the service L3 provides, Comcast wouldn't be able to (as efficiently) provide the service to their customers. In a sense, Comcast should pay L3 for their access to the content.

But they're not asking for L3, they're asking for Netflix. So maybe Netflix should hire a CDN that can play ball with Comcast, instead of L3, who is apparently being shut out.

> So maybe Netflix should hire a CDN that can play ball with Comcast, instead of L3

Um, didn't someone upthread point out that Level3 doesn't actually carry any Netflix content?

So what you are saying is that Netflix should pay extortion money to Comcast - either directly, or via a CDN - because Comcast have a monopoly situation.

You can try to package it any other way you like, but that's what you are suggesting: The only reason Comcast is in position to try this gambit is because there are alternatives (albeit very few) to backbone providers, and major content providers can be bullied directly, but most Comcast customers do not have alternatives to Comcast.

If anything, that Comcast even dares ask for this should be evidence that they are abusing a monopoly situation, and ought to be a trigger for massive reform - in a free market, they'd get told to take a hike.

L3 is not the provider, Netflix is the provider. Netflix has chosen to contract with L3 so that their customers can receive the content. L3 is also an ISP and so has peering agreements with other ISPs. Netflix has customers who aren't using L3 as an ISP so L3 and these ISPs make use of their peering setup to transmit Netflix's content to the ISPs' customers.

L3 is not pushing, the customers are pulling. The ISPs serving the customers are not able to keep up.

You want pushing? That'd be L3 using (as an example) Comcast to reach customers of TWC. Then L3 would be pushing through Comcast. That's not the (apparent) situation here. The end user is on the other side of this peering divide, not multiple peers away. Customers are pulling data from Netflix which happens to be on L3. Blame the customers and up their rates if that's what's needed to cover the costs. But at $15+ billion a year in profit Comcast can probably afford to use the customers' money to provide them with the promised experience.

L3 is pushing to maintain its contract with Netflix. L3 is pushing, because L3 isn't the only CDN responsible for Netflix traffic. "Comcast" (it may not actually be Comcast and in fact I doubt it is given Netflix's recent agreement with Comcast) can just use Netflix's other CDNs, and traffic will continue to flow normally.

L3 is not necessary here, and everyone who's actually involved knows it, including L3.

If Comcast can't handle the amount of data, how would using a different CDN help? Does the other CDN somehow send less data given exactly the same requests?

Or are you arguing that an ISP should be allowed to restrict which services people are allowed to access depending on how much those services pay the ISP?

> It sounds like Level3 wants to enter into bigger-money contracts with the ISPs

As the article points out, peering rarely involves money (and shouldn't ever; as the word peer implies, its a largely symmetric relationship).

The ISPs' customers are paying the ISPs for bandwidth and then requesting traffic. Level3 is providing said traffic and then the ISP drops it because they don't have enough bandwidth. How can a reasonable person possibly interpret this as being Level3's fault?

If an ISP isn't willing to supply its customers with the bandwidth that is being paid for, they need to either fix that or stop promising things they aren't willing to deliver on.

> stop promising things they aren't willing to deliver on.

Very few ISPs offer residential Internet access with SLAs that guarantee performance or even availability. They mostly promise only a 'best effort' class of service.

That's a good point. I meant promise in the sense that that's how customers often interpret a bandwidth claim - legally, they're making no such promise (otherwise they'd be in a lot of trouble).

"Best effort" doesn't mean "Here's your dinner, take it or leave it."

ISPs have been successfully managing aggregate network bandwidth growth for over twenty years.

I work at a place that has transformed it's utilization of the internet via cloud services -- utilization going from 80-120MB/s to multi-gigabit. Adapting to that change was relatively easy and cheap. With the economies of scale that a large Telco has, it's a drop in the bucket.

Right, and the problem is that their version of best effort doesn't involve trying very hard at all, since they have minimal competition in most local markets.

I've been speed testing my shitty (Time Warner, yay) internet lately and am getting fully 50% of my 15m/s that I pay a whopping $80+/mo (hard to disambiguate between the 'packaged' cost with TV). I once got a Comcast rep to admit that they basically were unconcerned with providing anything which is at least %60 of the level that they advertise. Unless the entire company is both mentally and physically challenged I seriously doubt that this represents a "best effort" class of service. This is go-fuck-yourself service that a dive bar would be embarrassed by.

If Level3 can't give Netflix what it promised Netflix it would deliver, maybe Netflix should use a different CDN.

Other CDNs aren't having this problem, because they're willing to work with the ISPs.

> Other CDNs aren't having this problem, because they're willing to work with the ISPs.

Do we know this?

You asked this question in like 3 places, but yes, if you read the blog post written by Level3, you'll see them mention that other backbone providers are making deals with the ISPs, but that they're refusing to.

> if you read the blog post written by Level3, you'll see them mention that other backbone providers are making deals with the ISPs, but that they're refusing to

Where in Level3's blog post does it say this? I'm not seeing it. Certainly at least one other transit provider, Cogent, doesn't seem to be making ISP deals; if it did, the Netflix-Comcast deal wouldn't have happened

(Also, in the post I was responding to in this subthread, you said CDN's, not transit providers. They're not the same.)

Yes, let's encourage people to pay ransoms. That's going to end in a rational system.

Netflix switching would, in the short term, give them more bandwidth. In the long term, encouraging ransoms will cause them to go up and Netflix will be forced to keep raising their prices. This is exactly what Netflix does not want to happen, and they are wisely refusing to play ball.

I'm not very familiar with networks at this level, but it seems logical: Level3 wants to push more traffic through the ISP's pipe than the ISP's pipe can handle (because the ISP's customers want those packets).

The only solution for Level3 is dropping or queueing packets, and I'm assuming queueing is not feasible.

The ISP is pulling the packets from Level 3, Level 3 is not pushing it. Also Level 3 claims they can get the packets all the way to the cross connect between their network and the ISP's and they have extra capacity at the cross connect, but the ISP's refuse to upgrade their side of the cross connect (IE the part that turns them from a network provider to an Internet provider.)

>Also, dropped packets isn't the end of the world. In TCP at least, the packets just get resent.

Depends on what the application is, and how many are being dropped.

True, standard HTTP can in theory even still work with 99% drop rate (if you're willing to wait long enough) but Netflix, multiplayer games, and any other application you can think of that demands either high-bandwidth or low-latency aren't going to work. As far as those applications are concerned, it is the end of the world.

Streaming services utilize a buffer, so no, it isn't the end of the world if packets are dropped. The buffer gets used while the packets get retransmitted. As for multiplayer games, most use UDP, and the ones that use TCP tend to use some tricks that allow for a certain amount of latency.

This is why the Internet isn't currently down, even though this is currently a problem.

No widely used Internet-based technology can't handle dropped packets. Zero. None.

Besides, Netflix is a unique case, as it constitutes 30% or so of all Internet traffic. You can't list Netflix as one of many, because it stands alone.

Twitch.tv, justin.tv, youtube. That's off the top of my head and I'm confident there are more than I'm missing. There are plenty of video streaming services; not all of them are as large as netflix, but to say "Netflix is a unique case" is pretty weird.

And dropping packets isn't the end of the world, but it's certainly not good. Should we just ignore a problem just because the world isn't ending?

Netflix is a unique case. No one even comes close to network usage. Not even close. It's a very unique case, given its volume.

And I don't know who here said to ignore this problem, but they're wrong too. It's just that this isn't the "It’s insulting." problem the submission tries to paint it as. His solution is to "throw the bums out". Are you kidding me? Why are we wasting our time reading and talking about this tripe?

Can you clarify for me why I shouldn't feel insulted that I'm paying for a certain level of service from my ISP, and yet not receiving that service while they insist that it's not their fault?

Because you actually are receiving the level of service you're paying for, you just didn't know what you were paying for when you signed up.

In what sense? Are you referring to the "well actually the contract just says you pay for UP TO certain speeds" bullshit?

I'm referring more to the fact that ISPs probably reserve the right to negotiate with peers on their terms in their contracts, not on your terms.

You having a month of 720p instead of 1080p Netflix is worth it to the ISPs instead of raising your service prices by $5 bucks every single month.

Oh; instead of referring to some bullshit clause in my contract, you're referring to something that's actually irrelevant. That's weird, but okay. I can roll with it.

ISPs probably do reserve the right to negotiate with peers on their terms, but I don't care about that because I'm not paying them to have a certain agreement with peers, I'm paying them for a certain quality of service. If they are not providing that service (an example being giving me 720p Netflix instead of 1080p like I have paid for) then that seems like a pretty straightforward contractual violation. I'm paying for service and not receiving it. Netflix is providing the service to the ISP. I am providing my money to the ISP. Why is the ISP excused from holding up their end of the bargain?

This is turning into a flame war. If you can't see how negotiating peering agreements is relevant to a post whose first sentence begins, "There’s a peering crisis apparently happening right now among American Internet Service Providers (ISPs) and backbone providers..." then I don't know how we can continue...

The crisis, if you've read the article, is that Level3 is getting shunned by 5 of its client ISPs. The ISPs are NOT shunning other backbone providers, just Level3. So yes, this is entirely a contract negotiation between Level3 and 5 ISPs.

This isn't an existential crisis of the Internet, like Level3 is trying to portray it, it's Level3 being shut out of additional business by 5 ISPs because they won't play ball like the other backbone providers.

Why should L3 play ball? Just because everybody else is doing it doesn't make it a good reason.

They spent money on upgrading their network, and now that money is going to waste if the ISPs don't make use of the additional bandwidth (for additional money).

ISP customers are not suffering, because L3 is just one of many providers. What content does L3 provide that no one else can? And if that's the case, how is L3 not a monopoly in that respect? Aren't we all calling for more competition in this space anyway?

I guess I'm confused; in one post you're pointing out that I'm getting a month of 720p video instead of 1080p, but now you're saying that I'm not suffering. If my quality of service is being degraded, is that not suffering in the context of this issue? I'm paying for service and yet not getting it. So clearly there IS an issue, because connections are suffering. Are you telling me that ports between L3 and various ISPs are totally saturated and dropping packets while there are other nicely upgraded connections with plenty of bandwidth to spare? Because I'd like to see a source for that claim if so.

L3 doesn't provide unique content; they provide content that ISPs don't have. People ask for that content, L3 tries to provide it, and the ISPs can't accept it. Are you telling me that there other providers and L3 is just shouldering them out of the way? Again, I'd like to see a source for that.

First of all, L3 doesn't actually provide Netflix content, I don't think, so your video stream is unaffected entirely by L3's agreements with any ISP.

Second of all, you have no SLA with your ISP, so your quality of service is, "best effort". If you want an SLA, you have to pay, otherwise you're going to get the best that the ISP can give you, which means some degraded performance. That's what you've paid for. If you want better, pay more money.

Third of all, you really should read the submitted article. It's very clear you haven't. When you do, you'll find that L3 is the only backbone provider not playing ball, and the other backbone providers are paying, which means that there actually are nicely upgraded connections (I don't know about "plenty of bandwidth to spare"), paid for by the network backbone providers.

When you read the article, you'll be much better informed on this topic. I do recommend also reading L3's blog post. There's some good info there, and it provides all the "sourcing" you've asked for.

I've read both the blog post and the linked article. Can you provide specific quotes for the claims? Because the only thing that came close which I saw in either the post or the link was a graph of a different connection which isn't fully saturated, but has very little head room, certainly not enough to handle the spillage that L3 is dealing with. There's a mention of capacity upgrades in the works for that connection, but no specific numbers; it's not clear that the upgraded connection will be large enough to take on the lost packets from L3. And I didn't see anything in there about L3 pushing other providers out of the way.

So like I said; can you provide sources for those claims? Because you also thought the bakery analogy was a good idea, so I'd like to read your primary sources instead of trying to explain myself to you further.

This isn't the forum. Email me (provided in my profile) if you'd like to continue this discussion.

Your profile, at time of viewing. Congratulations; I'm done here.

user: diminoten created: 1831 days ago karma: 1799 avg: 1.51 about: QA/Wannabe Dev submissions comments

Oh sorry, I thought the email field was available!

diminoten@gmail.com

> When you do, you'll find that L3 is the only backbone provider not playing ball

Where does the article say that? It says Level3 is having peering problems, but I don't see anywhere that it says Level3 is the only transit provider having peering problems.

> The ISPs are NOT shunning other backbone providers, just Level3.

Is this true? Or is Level3 just the only one we know about? As I remember it, the transit provider that was linking Netflix to Comcast before the Comcast-Netflix deal was Cogent, not Level3.

> If they are not providing that service (an example being giving me 720p Netflix instead of 1080p like I have paid for) then that seems like a pretty straightforward contractual violation

But your contract with your ISP most probably does not guarantee you 1080p Netflix instead of 720p. It basically says something equivalent to "we'll give you enough bandwidth for 1080p Netflix whenever we can, but we don't guarantee it, and if you don't get it, there's nothing you can do about it".

Please note that I am not saying the ISPs are the good guys here. I'm just saying that trying to nail them on contractual violations doesn't seem like a fruitful approach to me.

It's a shared network. It's not bullshit that you don't have SLA guarantees. Business customers get those, and they pay a lot more for them.

There's an old saying in the car world – people buy horsepower and drive torque. Car companies always talk up the HP (even when they're kind of sad for the weight of the car, but I digress)…

ISPs sell peak speeds and customers live the reality of the fine print that talks all about it being a shared line and that real speeds will often be lower than the advertised peak…

Yes, it's their fault, but no, it's not – expecting average consumers to fully investigate every spec and to understand things like SLA guarantees for every product and service they buy in a world that is growing ever more complex has to end somewhere right? At some point, that total burden becomes unreasonable from a user-centered perspective, right?

Even the power company -- which, for all its complexity, is a vastly simpler system to model and predict -- turns off my AC from time to time to keep up with load.

Availability of a shared network cannot be guaranteed. It's the irresistible force. Even if consumers don't understand, it still cannot be guaranteed.

Or, they could each have explicitly allocated pipes. They will pay a lot more for the privilege, though, just like they would pay a lot more for a private road instead of one they have to share with their neighbors.

I have problems using myfreecams.com for live adult video chat because of the high rate of packets being dropped.

I don't really think your comment makes sense, if you and I are juggling balls and suddenly you throw up your hands and shout "I can't take any more balls" do you think I should pick up more balls and throw them at your face or should I leave them on the ground (which in this analogy would be level 3 dropping the packets)

What do you mean when you say, "Throwing their hands up"? Are you saying the ISPs are refusing to route any traffic after a certain threshold? Does Level3 get cut off entirely if it exceeds its bandwidth limit?

They effectively are refusing to route after a certain threshold. Their connections are saturated and they aren't wanting to play ball the way other peers have (at least per Level 3's side of the story). So once their X Gigabit link is saturated, quality goes down. They don't want to use the same arrangement that Level 3 has made with others (with respect to cost distribution) to upgrade the connection so the situation remains.

You should try browsing the Web with a 10% packet loss sometime.

> Shouldn't the ISPs replace Level3 if they keep dropping packets?

How would that help when the reason Level3 is dropping packets is because the ISPs don't have enough capacity to receive them?

Level3 can't magically make a 10Gbps port take 20Gbps of traffic, nor could any hypothetical replacement the ISPs might go to (there aren't many realistic alternatives).

The problem is that it's easier to compete with Level3 than with Comcast, because while it's incredibly hard to build a backbone provider, it's nearly impossible to build an ISP. This asymmetry is making Comcast bold. They think that eventually competitors to Level3, Netflix, etc., will take advantage of their poor connectivity, pay them, and reap the rewards. They might be right.

It could just as easily be the case that nobody's going to put however many dollars into building out a nationwide infrastructure that's completely dependent on future goodwill from Comcast et al, in light of their current behavior.

Maybe I'm just an optimist, but I think Level3 wins here just because most people have no idea who they are or what the 'internet backbone' is. "My netflix is slow!" => Netflix's problem. "My internet is slow!" => Cable company's problem.

You would think, okay, if Comcast is terrible at maintaining sufficient peering for it's customers needs -- and if the OP proposal to throw Comcast out of peering exchange points happened, that would certainly lead to increased terribleness for it's customers -- then eventually it's customers would choose a different ISP. The market would solve it.

The problem is that in many many markets, Comcast (or another ISP) are pretty much the only choice. Customers don't have another option, no matter how much Comcast underfunds it's peering infrastructure or gets thrown out of peering exchange points.

So what is the consequence to Comcat for underfunding? What is the consequence to Comcast for even such a disastrous outcome as getting kicked out of the peering exchange point? Not a lot.

I'm not sure what the solution is, but 'regulate them as a common carrier' is certainly part of it, since they are a monopoly, and the common carrier regulatory regime was invented for exactly such a monopoly.

> Comcast (or another ISP) are pretty much the only choice. Customers don't have another option

If an entire town is angry, their city councilors will very quickly utilize eminent domain on the fiber in the ground and sell it to a competitor.

But not before the ISP lobbies the state legislature to decree that cities can't annex fiber.

I've previously interned at one of the mentioned Cable Companies, and I see both sides.

The solution is to make it 'capitalistic'. Change all of our internet contracts from Unlimited (up to 'x' GB/month), to a simple $/gb cost.

It would be in the ISPs best interest to provide their customers the fastest internet connection as possible. E.g, if a customer can stream a 4k video vs SD then the ISP would make more money per unit time.

Think of it this way, if comcast charges $0.25/GB, and a netflix SD show is say 1GB and HD is 4GB, then comcast grosses $1 for HD and $.25 for SD for the same customer streaming request.

Over time its likely the price per GB would decrease, just like it has for cellular.

On a more evil side, this would also stop chord cutters. Pirating content is no longer 'free', and Netflix would cost significantly more than $10/month ($10/month + 'x'GB * $/GB).

As for what rates to expect, if comcast charges in ATL $30-55 for 300GB, that'd be about $.10/GB to $.20/GB. As for speed tiers in a $/gb system, your guess is as good as mine.

How does changing plans from unlimited to metered data make them more 'capitalistic'?

An example of making things more capitalistic would be to make illegal the municipal contracts granting monopolies on last mile service.

In an unlimited system, a customer which uses 10GB/month grosses the company the same amount as a customer who uses 1TB/month.

Going to a metered system, people who use more bandwidth are charged more.

Technically ISPs don't pay for bandwidth, but for a 'x' GBps link. But our electric companies do the same, they pay for 'x' Kw/h (a power plant can only output so much electricity per hour).

>...make illegal the municipal contracts granting monopolies on last mile service.

Thats a very hard peice of legislation to pass. I'm not even sure if Google Fiber would agree to this. Even if this legislation passed, would that fix our net neutrality rules?

Moving to a metered system gets rid of most of the ISPs arguments against net neutrality....all of them except having a company pay for your bandwidth.

> Thats a very hard peice of legislation to pass.

That may be so. But all of Europe manage to go from monopoly telcos to a system with guaranteed access for ISPs to the last mile.

> Moving to a metered system gets rid of most of the ISPs arguments against net neutrality..

In my experience, from having run an ISP: It also massively curtails usage, to the extent that it is a net negative for the ISPs because users become less dependent on you, yet don't pay you as much.

There's a reason why ISPs went to "unlimited" contracts: If users have to meter their usage, they tend to turn out to be willing to pay less, because their internet connection is suddenly something where they have to think about the money they spend from day to day, rather than something that's just a fixed "thing" in the background they don't think about. People are willing to pay extra for "unlimited" to not have to think about it, and when they don't think about it, they don't cancel.

>It also massively curtails usage, to the extent that it is a net negative for the ISPs

I think that's the reason why Comcast wants to do the 300GB 'unlimited' with a $10/50GB overage charge. As they get the best of both worlds.

The issue with having a 'hybrid' plan is that to Comcast's finance department they still view it as a fixed gross per customer. Where-as with metered usage, the finance department would hopefully realize that the bigger the pipes the greater the potential for $$$.

In regards to peering, Comcast currently has no incentive to upgrade there peering exchange points. But in a metered system, they have a lot of money to gain for upgrading the exchange points. Giving level 3 and Comcast incentives to upgrade peering exchanges. (Today its only Level 3 who benefits).

Interesting suggestion.

I've often thought about this. What I hear (I'm not very knowledgeable in this field) is that somewhere up the chain, someone has to pay for bandwidth (per GB) (can anyone confirm this?). But consumers prefer to know how much they will pay each month, and so they get that -- a flat rate service.

So now we have a situation where a company's expenses are measured in GB of data transferred, and their customers pay proportional to their maximum transfer rate. This means the incentives of this company and its customers are not aligned.

It's a bit like if you rent a car and the price you pay is proportional to the top speed of the car, but the expenses of the car hire company is proportional to how many miles you drive.

Where I live, we're currently in the process of wiring up the apartment complex to FTTP (fiber to the buildings and copper from the basement up to the apartments). I'm in charge of the process, and I've thought about asking whether I can get an unlimited connection speed, but pay per GB. I haven't asked yet, but I think I might prefer this over a flat rate price per month. When I use the connection I pay, and it's blazingly fast, and when I don't use it it's free. Contrasted with being slow when I use it, and also paying when I don't use it.

somewhere up the chain, someone has to pay for bandwidth (per GB) (can anyone confirm this?)

No, wholesale transit is sold by bandwidth; e.g. a 10 Gbps pipe costs around $5,000/month regardless of usage.

More like the ISP's expenses are proportional to aggregate peak demand, but yeah, it's not aligned.

> No, wholesale transit is sold by bandwidth; e.g. a 10 Gbps pipe costs around $5,000/month regardless of usage.

When I went around asking for a price for a shared connection to out apartment complex, I was asked how many tenants we had. And the reason -- I was told -- that they asked for this, was that the more tenants the more traffic consumed.

So it seems to me that somewhere, someone is paying for traffic. Or else I don't understand.

If you want to buy x Gbps no questions asked, that is definitely a service that exists. It sounds like you were dealing with retail-level sales people and you weren't clear about what you want to buy so they're trying to help you figure it out.

End user ISPs often charge by traffic. But buying transit from larger ISP for colocation or to operate your own ISP for example, is typically done based on some combination of port speed, committed information rate (CIR) and peak usage within a defined interval (burst).

For example, at work we pay abour $6 per Mbps 95th percentile use averaged per 5 minutes, with a 10Mbps CIR and 100Mbps port speed at one of our data centres. This means we always pay for at least 10Mbps. Our ISP then measures our traffic and averages it over 5 minute intervals. They then throw away the top 5% of samples (so we can have short traffic spikes up to our port speed without paying extra), and we pay for the higher of 10Mbps and the next sample.

Any bandwidth above the CIR is entirely dependent on whether or not our ISP has spare capacity. The higher your requested port speed, the less likely you are of being able to burst much more than your CIR - any halfway decent provider will have 90Mbps excess capacity much of the time, so paying for 10Mbps CIR and bursting to 100Mbps is not unlikely, but paying for 1Gbps and being able to burst to 10Gbps gets far more dicy.

At some higher speeds some providers will only offer links where CIR == port speed - effectively a "unlimited" connection. If you know your bandwidth exceeds the CIR a lot of the time, you will typically want to pay for a higher CIR for the reason that you otherwise risk getting throttled even if your port speed is higher and your connection in theory is burstable.

Effectively the sum of the CIR of their customers + some margin is what the ISP will turn around and contract with their transit providers and peering partners for. If other customers don't use their CIR, and your CIR is lower than your port speed, your ISP is likely to let you burst (at a cost) that moment, but traffic within the CIR (should) always have priority.

Virtual Servers come with some TeraBytes per month. AWS and Google Compute Engine counts traffic by the GigaByte.

Most upstream providers at colocation facilities use 95th percentile pricing, that is you have a gigabit link, or a 10g link, and every 5 minutes they take a sample of current speed, and then at the end of the month they sort those samples and take the sample at the "95th percentile", which can be understood as link utilization discarding bursts which, gives a speed, which you can multiply by the length of the month, and you get transferred bytes (or octets as network/telco people like to call it).

A 100Mbps link is ~30 TB per month. If you do business on the internet, the moment you get big enough, you can negotiate these prices.

Just buying a gigabit ethernet port is not even that expensive ( https://ams-ix.net/services-pricing/pricing ), but this is without peering, and you have to either deal with every non-open partner there ( https://ams-ix.net/connected_parties ) or get transit from someone who has a full coverage of the internet. (That is all the prefixes assigned by local internet registries are reachable via them.)

Just a follow up to your 95 percentile rule, there is a famous story of Google exploiting this (in the early days before they were flush with money) in order to transfer their data to a new data center on the opposite coast. http://www.dodgycoder.net/2013/02/googles-fiber-leeching-cap...

I don't think the cable companies have much of a side. I think a large part of the problem is complacency about improving service on the cable companies' side rather than active malfeasance: they simply don't have to work very hard to retain customers since the local markets are monopolies or small, stable, oligopolies.

There just needs to be some more competition to hold their feet to the fire.

> "On a more evil side, this would also stop chord [sic] cutters. Pirating content is no longer 'free', and Netflix would cost significantly more than $10/month ($10/month + 'x'GB * $/GB)."

Maybe you've been living under a rock, but ISPs who are also cable television providers are very interested in caps, arguably to protect their cable television business. [1] http://www.pcworld.com/article/2155286/comcast-exec-says-ban...

This whole debate about who's supposed to pay for peering is smoke and mirrors. The truth is that cable companies have a conflict of interest between their two businesses, and they're attacking the internet side to protect their cable tv business. End of story.

While I agree with the general thrust of the article, there is one fallacious argument here.

Cringely argues that cable breaks even and money is made on the net, but that's an artificial distinction. What if cable disappeared? Would they still make money if they had to pay for the upkeep of the network with only Internet fees? The desperation and risk of this game of chicken convinces me that the answer might be "not much." The loss of cable might very well be apocalyptic for these companies, at least from a shareholder value and quarterly growth point of view.

What's happening is very clear to me: the ISPs are trying to either harm the Internet to defend cable or collect tolls on streaming to attempt to replace cable revenue. That's because cable is dying a slow death. This is all about saving cable.

The fundamental problem is that cable ISPs have an economic conflict of interest. They are horse equipment vendors that got into the gas station business, but now the car is driving out the horse and their bread and butter is at stake.

> Would they still make money if they had to pay for the upkeep of the network with only Internet fees?

In parts of the country, slower-speed copper, fast-download cable, and a few fiber networks are already built out. The cable distribution giants like Time Warner Cable and Comcast are already making a 97 percent margin on their “almost comically profitable” Internet services, according to Craig Moffet, an analyst at the Wall Street firm Bernstein Research. As Levin points out, “If you are making that kind of margin, it’s hard to improve it.” And most Americans have no choice but to deal with their local cable company.

-- http://www.technologyreview.com/news/510176/when-will-the-re...

If that's accurate, I suspect that they could live without cable as we know it today and still make plenty of money. Working on finding out more numbers to see what the revenue breakdown is between cable and internet, though I wouldn't mind if someone else chimed in.

EDIT: Found this [1] for Comcast. For 2012 and 2013 they've lost video customers, increased internet customers. Had growth in revenue for both categories, but a greater increase from internet (percentage and absolute).

So the way things are heading, cable leaving would cost them a lot. However, they can probably make up for it elsewhere, and, ultimately, should be planning on cable TV going the way of the dodo regardless of the nearterm revenues and profits. It's not a viable longterm strategy, like SMS (while immensely profitable) for cell carriers, a good short term strategy but as content shifts to the generic data channels the specialized channels will fall into disuse.

[1] http://files.shareholder.com/downloads/CMCSA/3182774232x0xS1...

EDIT 2: Check out page 53 of the linked filing.

Video revenue for 2013 was $20.5 billion. Programming costs were $9.1 billion. So the net, ignoring any other costs, right there is $10.4 billion. If the ISP side of the house is actually making 97% margins like the claim in the Technology Review article, then it is making almost as much as the video side of the house. At the current growth rates it'll surpass the video profit, but not revenue, sometime in the next few years. Their NBCUniversal acquisition may reduce their programming costs, I'm not sure of the legalities involved in that (their growth and acquisitions could put them afoul of anti-trust/monopoly laws and regulations).

"97% profit margin" is a self-doubting figure. As soon as you hear it you should know that someone is trying to sell you something.

It's like deciding Microsoft's profit margin based on the cost to physically acquire CD-ROMs.

Alright, perhaps I shouldn't have quoted that, but let's continue with the 2 edits of my post where I found real figures on revenue and costs. Video revenue is (in absolute terms) higher than internet revenue. Video revenue growth is slower than internet revenue growth. Video customer growth is actually shrinkage, they're losing customers (hundreds of thousands each year). Internet customer growth is actually growth (over one million each year).

The primary video cost (programming) is approaching 50% of video revenue, and it's growing at a pace that has literally equaled or exceeded video revenue growth for each annual period in the linked document. So, making an assumption here that's likely untrue, if we consider that the other costs are distributed equally between the two (marketing, tech support, physical support, etc) then the internet service is approaching parity with the video service, and will exceed it in several more years as the higher profit business component.

I suppose if someone else has more time to dig through this document or other insight from being in the industry they could chime in with which has the greater costs once programming is removed from the equation.

The problem is that you can't model the ISP portion as something with fixed costs to build and then small marginal costs of moving bits around. Building it out is the primary cost to the ISP portion and we, the customers, rightly demand they keep on doing it. They have to always keep on building more capacity. You can't say "okay, now they have stopped building capacity, it's all rent-seeking from here."

The actual electrical cost to move the bits back and forth is nil. It's making sure that half the people in the neighborhood can use Netflix at 7pm two years from tomorrow that requires them to spend money today.

The problem with your later analogy is that there was minimal gating to opening a gas station if the local tack vendor started trying to extort the local populace to maintain a dying business model.

That is why, usually, capitalism works in these situations - if the status quo is exploiting its customers, you can create a competitor because there is profit between the extortion and the break even.

With ISPs, or any general infrastructure, it is a hugely inefficient usage of resources to duplicate the work - good analogies would be how dumb it would be to have multiple sewer systems, with only one attached to the house at a time, or multiple voltages of electric lines where only your choice electric company is hooked up.

Fundamentally, it is that all and any future goods that require transport to the home over infrastructure need to be public services, and the maintainers be common carriers. Because creating the pipes (the water, the subway, the electric, the networking) are all prohibitively expensive to try to compete in, require deep intervention of states (which rarely doesn't result in market dilution) and are naturally a common good because unused bandwidth - in roads, in power line voltage, in pipe flow - are wasted potential, and duplicating the effort and having all the lost potential as a result can be, and is devastating to economies.

You're referring to the concept of a natural monopoly, which is problematic for anarchist and libertarian versions of capitalism. There are certain markets that inherently favor monopoly for physical constraint reasons. Utilities are the classic example.

Having multiple water pipes would be silly because of their massive size and the need to tear up the street. But I don't see the same problem with stringing multiple wires to the same house.

Especially if you want to keep on increasing capacity, which we do.

What makes you think of desperation and "a game of chicken"? What's really at stake here? Looks to me like only something ISPs have already learned to live not having, customer approval.

Far as I can tell, there's not much more they can lose here until the business landscape changes, and it's a toss up as to whether this sort of brinksmanship is going to drive changes in that landscape or not.

Internet service is a much higher margin business than television. I don’t see why Big Cable should want to stay in it, at this rate.

81% of the money TWC takes in video subscription fees they spend on video programming from Viacom, EPSN, HBO, etc. [1]

3% of the money TWC takes in broadband subscription fees they spend on high-speed data. [2]

These figures do not include infrastructure investments ($3.2 billion in 2013). [3]

[1] (4,782m+772m)/6,825m = .81. (http://ir.timewarnercable.com/files/doc_financials/Annual%20...) [2] 175m/5,822m = .03. (Ibid) [3] 3,198m. (Ibid)

And Verizon, with it's pretty big FiOS buildout, and AT&T with their more limited U-verse buildout. Both are now halted except for contractual requirements, but they also offer "cable" service, and AT&T is rather nasty in protecting it, caps of 150 GiB per month if you have plain ADSL---even if AT&T is not providing U-verse in your city---and 250 GiB for U-verse customers, $10/each additional 50 GiB. Verizon occasionally hints they might do caps, but haven't yet to my knowledge (I'm an AT&T customer so I don't follow them closely).

The proposed solution is at the bottom of the article (which is why everyone seems to have ignored it):

> The solution to this problem is simple: peering at the original NSFnet exchange points should be forever free and if one participant starts to consistently clip data and doesn’t do anything about it, they should be thrown out of the exchange point.

I do have a couple questions though - who is in charge of the original NSFnet exchange points, and do they have this authority?

I don't know why everyone is up in arm over this. Here is reverse thinking and a perfect oppoturnity for everyone.

The current situation is that Comcast doesn't have the equipment/resources to handle extra internet traffic at its peers. Most people want Comcast to buy more stuff to handle it, why don't we think the opposite way- get Comcast to decrease its amount of traffic?

If we can get Comcast to consume less traffic, they wouldn't have to complain to other peers about load asymmetry.

The best way to decrease traffic? Make Comcast has less customers.

Why does Comcast has so many customers, even though their resources cannot handle it? Because they have a government mandated monopoly in the last mile, so they are forced to have more customers than what they can handle.

We can come to a conclusion that last-mile monopoly -> network congestion -> forcing L3 to pay for peer.

If Comcast has to compete with other ISPs for last miles, the traffic load would shift from 1 single entity (Comcast) to 10+ smaller ISPs. In such case the traffic load problem would not exist.

Another solution is to breakup Comcast.

See? This is a perfect opportunity. Comcast can has its multi-tier network, but at the price of the last mile monoploy. After all, if they want to have the right to choose peers, we customers should also have the right to choose ISPs.

The UK experience is, broadly speaking, that any ISP who is sufficiently tall to have the appropriate interconnects can offer a service to a customer via the incumbent's last-mile infrastructure. (This is for telephone-based ADSL broadband - the UK's only cable operator is not bound by this.) But, furthermore, competitor ISPs are enabled to install their own equipment and backhaul directly into local exchanges, known as LLU - local loop unbundling. LLU allows competitor companies to provide just your broadband, or your voice telephone service, or both.

There is one further step, whereby the prices of the incumbent monopoly are regulated in areas where no competition exists. Ironically this works in the opposite way to how you'd think, as it forces the incumbent NOT to offer their lowest prices in that market - the intention being to make monopoly areas prime targets for competition and to ensure that potential competitors aren't scared out of the area by predatory pricing.

It's an odd system with good and bad on both sides, but it seems a lot better than being stuck with a single source of Internet access.

People keep claiming that there should be a "free market" for bandwidth...but then they say that the ISPs should have to absorb the costs of peering (which can be significant -- the hardware isn't free) without passing the costs on to anyone. The backbone providers complain when the cost is passed to them; the consumers scream bloody murder when the costs are passed to them in the form of a bill.

Obviously, there's no free market in the status quo: we (consumers) basically expect to pay a low, ever-declining price for bandwidth, while someone else eats the costs of a growing network infrastructure. There's an economic disconnect, and legislating that it shouldn't exist seems worse than futile.

I say: pass the costs on to the consumer, and break down the monopolies on last-mile cable service. If the cable companies had to compete for subscribers, they could still pass on the costs of improving their infrastructure, but they'd have to compete with everyone else to do it.

In other words, the problem here isn't "net neutrality" -- it's that we've got a monopoly at the last mile that we need to destroy.

>which can be significant -- the hardware isn't free) without passing the costs on to anyone.

I'm not trying to be rube but have you seen how much comcast/verizion/TWC customers pay? It's more than enough to string another switch and fiber line at a route point.

>basically expect to pay a low, ever-declining price for bandwidth, while someone else eats the costs of a growing network infrastructure

Are we though? Our bills have only been increasing. The amount of money they put into infrastructure has decreased[1].

The Telecoms did have to put a lot of initial investment into infrastructure, but that was largely from laying the wires. Most of them are already starting to see positive returns on those investments [2]. They already have the infrastructure to support traffic to consumers in the last mile. The peering infrastructure being upgraded shouldn't cost them even 1/10th as much.

So why should they get to pass their peering infrastructure costs on to the content providers? They should pass it on to the consumer if they must, after all it's us trying to get the content. But really they are going to be making such a large profit off us over the coming decades it's ridiculous for them to cry poor.

Net neutrality is fairly meaningless when it comes to the last mile. I doubt hardly anyone is saturating their last mile worth of bandwidth. I regularly have three streams going in my household and it doesn't saturate my bandwidth. That part isn't what needs upgrading.

[1]http://www.vox.com/2014/5/12/5711082/big-cable-says-broadban...

[2]http://dailycaller.com/2013/02/15/does-cable-really-have-a-9...

> Are we though? Our bills have only been increasing.

I would venture the guess that the price you pay per megabit has decreased. The problem is that the number of megabits you demand has increased faster than the price per megabit has decreased.

>I would venture the guess that the price you pay per megabit has decreased.

For me personally or on average? I'm a heavy usage user, but most my neighbors pay the same as me for using a lot less. My grandparents have basically the same service as me and all they use it for is email, some web games, and the occasional VOIP call and video.

Considering that nowadays almost everyone has high-speed unlimited plans for the same price I might question whether the net per megabit price for ISPs has really gone up. At the very least I'd gamble to say that it hasn't been anywhere close to exponential or unsustainable for them to keep up with.

I can't speak for you, of course, but 10 years ago my parents paid more for a 256kbit/256kbit DSL connection than I pay now for a 35mbit/5mbit connection now.

My (parents') connection back in 2003 (256/256 Kbit/s for 395 DKK per month): http://web.archive.org/web/20040604100538/http://www.cyberci...

My connection now (35/5 Mbit for 279 DKK per month): http://www.fullrate.dk/privat/bredbaand/priser

That's 7.97 DKK per downstream megabit in 2014 versus 1580 DKK per downstream megabit in 2004.

I very much doubt it has become 200 times cheaper per megabit/s in Denmark and actually more expensive in the US (or wherever you live).

That's all bandwidth, not total data usage. I'm talking about price per actual megabit transmitted by the ISPs. It's not like you actually do 35/5 mbps every second of the day for the entire month.

I'm in an east coast city in the US. I remember having had maybe 5/1 mbps 10 years ago. I wasn't paying the bills but I'm pretty sure it was like $50 per month from the TV commercials. Now I've got 25/10 for closer to $70 per month. Bandwidth wise it's risen a bit, but not really that much. Course I'm pulling down a lot more megabits per month in streaming content (though I was big into Napster/Torrents back then).

But my point was that most people have upgraded to broadband from dial-up even if they don't really use it that much. Only the ISPs would really have the data, but I'd be surprised if the cost per megabit transmitted for them has gone up dramatically.

Your guess is false. Comcast raises prices twice a year. My plan has gone up in price by about $20/mo since I activated it a couple years ago with no increases in speed or reliability.

ISPs charge users for the bandwidth they consume (or you pay extra for an unlimited connection, which is often not really unlimited).

So ISPs are already charging consumers for amount of data transferred, and consumers for the size of the pipe (maximum bandwidth).

Godaddy, Rackspace, Google, Amazon, etc. have skin in this game. With multiple redundant network connections they could, for a day or a week, defend neutrality by shaping their traffic to the lowest common denominator or routing their traffic to avoid the peer's punitive bottlenecks. Today its Level 3 and Netflix, but tomorrow it could easily be them.

"and make their profit on the Internet because it costs so little to provide once the basic cable plant is built."

That's some big hand waving, because laying the cable costs a fortune, and takes many years to recoup the cost which is why there is so few are competing for this "super profitable" business.

It's not even a one-time cost.

Without knowing the specific numbers, I'm pretty comfortable saying that the costs to an ISP to get the bits all the way until they reach the peering point is much less than the costs to an ISP to continually upgrade their network.

If the peering ports are congested that means that either the ISP needs to add more ports, or they are oversubscribing their capacity. Just make it illegal to sell more capacity than you have and the problem is solved.

> Just make it illegal to sell more capacity than you have and the problem is solved.

No. The result would be that you would pay 100 times more for your Internet connection. Or, alternatively, that you cannot get a guaranteed bandwidth, but you pay per GB.

This article: http://blog.level3.com/global-connectivity/observations-inte... states that Level3 has 13,600 Gbps of capacity with its peers.

Your solution would mean that Level3's peers can only sell 13.6 million 1 Mbps connections without overselling their bandwidth. Even if Level3's peers have several other Internet wholesale agreements -- let's say they actually have 5 times that capacity (most likely an overstatement) -- all Level3's peers would only be able to sell 68 million 1 Mbps connections. That would be some expensive 1 Mbps connections!

I like the article overall but I don't understand the author's proposed solution. The issue as it stands is apparently a lack of peering, in that big ISPs are using transit to reach large content providers rather than directly peering to those networks. So how would "kicking them out" for a maxed out connection work? If I buy transit from Level3 and my connection maxes out, I'm no longer allowed to be a customer of Level3?

I think the proposal is to simply sever the link when the peering connection saturates. The traffic out through the link from LazyISP is generating more traffic in through the link than the pipe can handle. LazyISP pretends that this is not their problem, that the Internet is somehow trying to shove its traffic through poor ol' LazyISP's pipes.

The claim is analogous to traffic exiting a Autobahn-style limited-access highway and attempting to reach its destination faster by taking the narrow country backroads to get to the on-ramp of the next highway. The only real reason for any traffic to route to LazyISP is if one of the endpoints is in there, and there is seldom a reason for that traffic to go anywhere but the shortest route between the end point and the nearest backbone highway access point.

The typical argument from LazyISP is so ridiculous on its face that it makes me furious every time I see it. All of those guys present asymmetric upload and download speed to residential customers. Every last one of them. And they pretend that when all of their customers' asymmetric bandwidths are summed up, the total is somehow magically supposed to be symmetric.

Every business on the planet whose cash flow depends on pushing bits out to the Internet goes directly to the backbone with content delivery networks, or at least to an ISP that is serious about peering. They are not going to an ISP with crappy interconnects. There is no way in hell that any cul-de-sac on the network map like a last-mile ISP is ever going to push as many bits to the backbone network as it pulls from it, especially when they do nasty tricks like throttling, port blocking, and hidden metering.

The point is that a steaming pile like Time Warner does not have any internal content providers that any of its customers want to connect to, precisely because of its business practices. If Time Warner cannot connect to the Internet, it has nothing to offer the customer through that precious last mile connection except television channels. So they stop paying the $25-150 a month and try to make do with standing on a ridge waving semaphore flags at each other. LazyISP needs Level 3 more than Level 3 needs them.

Therefore, when LazyISP tries to extort them, the correct response should be, "Fuck you," as they flip the off switch.

I think he is saying that companies like Level3 should kill all peering with Comcast if they refuse to upgrade to more peering. At least temporally to get their attention and that of their customers that is.

Not all peering, just peering at certain key sites (12 specific sites).

The extreme download vs upload traffic asymmetry between Comcast and L3/Netflix has been mentioned several times as a straw man argument for why Comcast is justified in charging Netflix directly.

Maybe Netflix could find some creative uses for all that idle viewer upload capacity to reduce the deficit ;)

- Have every Netflix client cache and serve chunks of the most popular streams P2P-style. You could have a DHT algorithm for discovering chunks or have Netflix's own servers orchestrate peer discovery in a clever way, for example by only connecting Comcast customers to peers physically outside of Comcast's own network. This would reduce Netflix's downstream traffic and increase viewer uploads.

- Introduce the Netflix-Feline-Image-KeepAlive-Protocol, whereby every Netflix client on detecting a Comcast network uploads a 5MB PNG of a cat to Netflix's servers over and over again while you're watching a video. Strictly for connection quality control purposes of course.

The problem is not to peer or not to peer. The problem is WHERE to peer.

I work for a european ISP and the problem we have is the location of the peering. Big content providers will happilly peer with you in, say, Palo Alto or Miami; but they will refuse to add a peering connection in Europe. Why? because today the problem is about WHO pays the Intercontinental route (which limited and is expensive bandwidth).

Level3 is known in the industry as a pioneer for bit-mile-peering agreements. This means you have to sample the origin and destination of the IP packets and make some calculations to know how many miles the packet has traveled and pay / get paid if someone dumps long haul traffic to a peer. Getting to this is complicated with current tecnhology and many companies are refusing to peer with Level3 because they don't know what will happen with their business with bit-mile-peering agreements.

Along these lines, can someone ask whether net neutrality ever existed at all? Akamai and F5 have been helping big corporations like Disney circumvent internet bottlenecks for over a decade now. Those who have had the money have managed to purchase faster delivery schemes for over a decade. Could it be, then, that telecommunications companies are consolidating so that they can extort money not from the small guys, but from the big guys? Are Hulu, Netflix and others willingly submitting to the extortion because they see no other way out? To be sure, the telecommunications industry is in desperate need of regulation because providing good service at a reasonable price for a reasonable profit is not good enough for them.

Since everyone is pitching their own solutions, how about I post mine. Let's take the example of Netflix and Comcast. Instead of no deal with Comcast, and thus giving Comcast Netflix users really slow or no service, Netflix should make the deal for now, and tell subscribers that if you use Comcast the Netflix rental is higher. By passing off the higher costs to the users, Comcast customers are given the incentive to switch ISPs.

Everyone shows loss aversion, and so will be determined to find out why being on Comcast gets them penalised. They will learn about its dick moves, and complain to Comcast to make them remove these fees so they can access Netflix, which they have already paid for access to.

Netflix would lose, they have competitors and Comcast (and potentially, in the future, ComcastTWC) has too many customers. Increasing the costs to all of them would result in the loss of too many customers before the dust settled.

Regulate ISPs as utilities.

Common carriers are the words of art for these sorts of utilities. And Comcast and company are in steadily increasing danger of this as they piss off more and more voters (at a certain point, all the money in the world is of no interest to a politician who perceives himself in danger of having to spend more time with his family after the next election; gun owners have demonstrated this many times).

> It’s about money and American business, because this is a peculiarly American problem.

Hardly. We've experienced the whole interconnect brinkmanship locally too (South Africa). Its actually quite the opposite - the interconnect things are a lot nastier in other countries because it tends to be paid for (powerful co vs underdog) whilst the bigger US setups seem to run mostly open peering.

I can't help but wonder if the RICO Act applies to this sort of extortion. My first thought was FCPA, but none of the ISPs involved can likely be construed as "foreign officials." The behavior can be described as demanding kickbacks, however.

Just for scale, backbone links these days are not 10 gigabits/sec, more in the order of 40-100 gigabits/sec.

The Amsterdam Internet Exchange is the largest and most important exchange in Europe, and it's peak traffic each day reaches 3 terabits/sec.

Maybe I'm naïve beyond any recognition, but shouldn't the ISP's or whoever is peering charge based on the bandwidth amounts? It sounds like they have a flat-rate contract with each other and now they're charging more?

From the article, it seems like the "flat-rate contract" the ISPs have for peering with L3 is $0.

The issue that L3 is raising, is that the equipment/appliances that the ISP has installed in these peering arrangements are insufficient to keep up with the amount of traffic which the ISP's customers are requesting, and L3 is attempting to provide.

If the ISPs were to upgrade their peering equipment to handle their customer's requests, there would be no issue.

So by Netflix making a deal with Comcast, did L3 effectively get cut out of the picture?

Also, isn't increasing equipment/bandwidth part of the business? It seems so ridiculously absurd that Comcast is balking to increasing its bandwidth...

Inexperienced in the field? This sort of thing has been going on since at least the '90s; this search ought to bring up some relevant hits: https://www.google.com/search?q=uunet+peering+ma-east

One thing to focus on is that there are 2 customers, and two middlemen. E.g. Netflix <-> Level 3 <-> Comcast <-> a Netflix subscriber.

Netflix and the Netflix subscriber are each paying their ISPs hefty sums for access to "the Internet". In the ideal world, Level 3 and Comcast would peer at various points for maximum efficiency on both their networks.

Comcast enjoys a local municipality government monopoly in their markets, with iffy competition from state and Federal monopolies granted to the RBOCs Verizon, AT&T, and in theory CenturyLink (which owns the US West states, but has never been very serious about Internet service and especially their own "TV"/cable service), and all the small ILECs who provide telephone service where the big boys had no interest.

One big problem here is that "TV"/cable service deployment for the first two RBOCs, who cover the vast majority of the population, has been halted outside of contractual requirements with municipalities and the like. (If you're in a really bad spot, like I am, on unincorporated land adjacent to two cities, the local cable company is not required to provide service and doesn't, AT&T provides slow and expensive (but reliable!) ADSL with very small caps, 150 GiB per month ... and now they're buying the DirectTV my parents use to get around that cable company problem!)

So as noted in the article, the bandwidth flow might be asymmetrical, and for these local "last mile" ISPs, there are technical and economic reasons for only providing asymmetrical service in the first place, e.g. head end noise, but all the Netflix bits coming in through Level 3 to Comcast Netflix subscribers are paid for, by the hefty payments the subscribers pay to Comcast for access to "the Internet". (And which are soon going to get caps as well from what I've read.)

But Comcast would of course prefer you get your "TV"/cable bits from them, and have no particular interest in enabling Netflix to supplant them. And they're now committing fraud on their subscribers, by telling them they're providing access to "the Internet", but deliberately degrading significant portions of it.

Level3 should drop the same percentage of outbound packets from Comcast, that Comcast drops on the inbound. If every tier 1 did, Comcast's internet service wouldn't look all that good any more, would it?

It will be interesting to see how gaming download services such as Xbox Live, PSN, Steam, etc would be affected as there as the file size of video games gets larger due to advances in the video game industry.

The problem would disappear if ISPs used metered pricing. Why do we have unlimited commercial broadband?

Lack of competition all around - just break up the big boys and everything will be fine...

why there are no last mile providers like comcast and ATT?

Local governments grant monopolies to Comcast or AT&T.

And some state governments force it on the local governments (see NC and TWC versus Wilson, NC).

So on the one side is the fat-cat ISP who doesn't want to make expensive capital investments ih their transport.

And on the other side is the fat-cat vc funded video content providers, who don't want to pay for the their mp4-based saturation of all the pipes.

This is a negotiation. There are two active media campaigns that are trying to gin up our anger against The Other Guy (tm) as part of their negotiations. I just can't get invested in this nonsense.

My read from the post was different; the peering agreements are in place, but one side (the isp's) aren't matching the investment level3 is making into it's infrastructure to handle all that streaming. They are instead asking for $$ in addition to the $$ they collect from their customers, essentially double dipping.

Unless I'm mistaken, by paying for an internet connection, the "fat-cat vc funded video content providers" are paying for "their mp4-based saturation".

I find it a bit offensive that this can be accurately viewed as a negotiation.

This is like demanding the supermarket buy your grocery delivery business a larger van when the toilet paper manufacturer starts selling larger packs.

It's arrogant, and violates the proper division of responsibilities. It only might work because of local monopolies. Netflix is already paying for traffic to Level3. ISPs need to upgrade their infrastructure and increase prices if it's too expensive.

L3 isn't the only content provider, and the 5 UNNAMED ISPs (Comcast isn't necessarily the culprit here) can still provide content to their customers. Netflix is free to hire one of the many CDNs which do play ball with these ISPs.

I'm paying my ISP to maintain a good enough connection to the backbone infrastructure such that I can download what I want.

The content providers are paying their ISP to maintain a good enough connection to the backbone infrastructure such that they can upload what they want.

The content providers ISP is maintaining a good connection, and shunting their packets into the backbone happily.

The problem, is my ISP is being a dick and refusing to upgrade their connection. The backbone is dropping my packets because my ISP isn't holding up it's side of the bargain and giving the backbone the connection it needs to offload the traffic I'm requesting. Instead, they want the businesses I request packets from to pay them a toll to deliver those packets.

This isn't how the system is supposed to work, but they don't mind breaking everything and pissing everyone off if it means they can put a surcharge on the traffic I request.

1. Peering is based on equal traffic both ways. At the moment we tend to download gigabytes with a few bytes of request. As video-communications really takes off (yes chicken and egg - see below) this will get lost in the noise

2. rise of ad-hoc local networks This might come out of mobiles, this might be me dreaming, and it might come with sensible home router designs, but ultimately most of the traffic I care about probably originates within 2 miles of my house - my kids school, traffic news, friends etc

A local network based on video comms - that will never happen. just like mobile phones.

3. electricity and investment In the end this is down to government investment. Let's not kid ourselves, gas, water, electricity, railroads, once they passed some threshold of nice to have into competitive disadvantage not to have, governments step in with either the cash or the big sticks.

Fibre to the home, massive investment in software engineering as a form of literacy, these are the keys to the infrastructure of the 21C and it's a must have for the big economies, and it's a force multiplier for the small.

> As video-communications really takes off (yes chicken and egg - see below) this will get lost in the noise

Aside from the chicken and egg problem, I don't think you ever get to a point where that really is the case. Not because video comms never take off, but because even when it does its a small share of content delivery and the net is still wildly asymmetrical.

The problem is your point #3 has never been true, though. In each case, in American history (though I don't know about water), the utility was privatized for _years_, became local monopolies, and used that monopolistic advantage to become one of the most major Washington lobbying industries of the time, often to the massive detriment to specific sections of the economy. In each case it was only _after_ an alternative industry rose in lobbying prominence in washington (often, the next one on your list) that the previous utility lost their position as #1 lobbying firm in Washington, that their competitors were able to then force anti-trust lawsuits, and or get fair government regulation.

The railroads were monopolized until the late 1800s, the original American idea of a fat cat comes from railroad owners who made profits at the expense of midwestern farmers. That was eventually regulated when oil became a major lobbying industry, whereupon oil and gas subsidies became the new norm and we regulated the railroads. Electricity is a general complex area, but outside of domestic coal-based electricity production, my understanding is that the majority(?) of American electricity comes from nuclear power, which in turn is a regulated monopoly. That industry is so locked that Northern America imports much of its electricity from nuclear plants in Canada, and Texas is actually on Mexico's electricity grid.

I was somewhat hopeful that as Wallstreet rose to replace other industries as one of the top lobbying firms in America that they would end up pushing for net neutrality so as to commoditize their network costs, but instead they simply pushed for B2B fiber to be regulated completely differently (and more sanely) than consumer networks.

I don't want to wait for something better than the internet to come around, before we get net neutrality back.

interesting. Is there a constant contraction in the time between monopoly and regulation (ie decades for rail, years for electricity?)

Except it’s actually right (not wrong) because those bits are only coming because customers of the ISPs — you and me, the folks who have already paid for every one of those bits — are the ones who want them.

What is the source of the notion that, because you paid for your consumer broadband, all bits are paid for and the charge for carrying them cannot be split with the other side of the connection? Why is it so bizarre that both sides of the connection have to pay for it? Because you're used to your phone working differently?

As an analogy, you know how you used to pay for a subscription to a magazine and there were ads inside which advertisers (the other side of the connection via the magazine in this case) also paid for? The magazine split its fee in two: you paid part of it, and the advertisers paid the other part. It's the same here.

There is nothing fundamentally wrong with charging both sides. You may prefer a different fee structure but a better argument than "I already paid for it!" is necessary.

In this case, the difference is that broadband subscribers are not being sold a magazine which they know will contain ads--they are being sold a full connection to the internet! That's the precedent, that's what we think we're getting for our money.

What does full connection to the Internet mean? No ISP can guarantee it because any other independent network can drop traffic at will. And the Internet is just a bunch of independent networks. The service you believe you paid for doesn't and cannot exist.

I can filter out all Comcast IPs on my firewall tomorrow if I feel like it. Many people actually do it to third world providers.

There's a difference between bits not being delivered because of the originating server failing to deliver and my ISP actively preventing delivery.

"Unlimited Internet" has already been established in sales pitches for years, the language defined once people understood the concept of bandwidth caps. "Unlimited" means you have unlimited "access", not unlimited "bandwidth". This has been the industry's very own sales pitch for years.

The other side of the connection has paid. ISPs are now wanting the backbone, i.e. the provider that makes the internet go long distances, to pay them.

No, it's not, because the other side also paid for it to their ISP, CDN, or the build-out and operation of their own CDN.

What Comcast and friends are asking for is a _third_ payment, just because. And don't be mistaken, if they win this battle they will start looking for the fourth payment, which is from their customers for faster service to "premium" sites like Netflix, HBO, iTunes, etc.

because the other side also paid for it to their ISP, CDN, or the build-out and operation of their own CDN.

That's like saying that you should be able to park for free because you paid for your car, all the taxes on it, and the fee for issuing a driving license. Paying your ISP doesn't necessarily grant you unlimited access to every other network on the Internet.

There is no God-ordained fee structure here. It can be split between you, Comcast, Netflix, your landlord (many will pay for laying fibre and then for servicing it), local government... Some may be better than others but there's definitely no moral highground.

I'm sorry, your comparison doesn't work for me. I don't see how those even remotely compare.

What I was actually sold from my ISP and I'm currently paying for was indeed unlimited access to everything on the Internet. It's called an Internet connection and that's how I interpret what I'm paying for.

If the bill can be split between me, my ISP, and Netflix then I want my bill lowered if my ISP is going to charge Netflix to deliver bits I requested. But I won't be holding my breath over that negotiation happening.

What I was actually sold from my ISP and I'm currently paying for was indeed unlimited access to everything on the Internet.

There is no such service. Not only does it not exist, it cannot exist. No ISP can provide that. Any independent network on the Internet can start dropping your packets tomorrow for any reason. Including a lazy admin filtering out your entire country. Or due to a copyright agreement. Happens all the time.

If they really promised "unlimited access to everything on the Internet", then you were misled but I doubt their lawyer would allow for it.

If the bill can be split between me, my ISP, and Netflix then I want my bill lowered if my ISP is going to charge Netflix to deliver bits I requested

Why? It may not be increased. Or it may be increased less than it otherwise would. Or it may be jacked to whatever amount they can get out of you.

That's the whole point. There is no pre-ordained price for carrying bits. There is no pre-ordained split between carriers and end points. There is no pre-ordained service level. It's all just a matter of agreement between parties. So now they're negotiating.

I don't understand however why some believe they have moral highground because "they already paid." Everyone already paid something.

> There is no such service. Not only does it not exist, it cannot exist. No ISP can provide that. Any independent network on the Internet can start dropping your packets tomorrow for any reason. Including a lazy admin filtering out your entire country. Or due to a copyright agreement. Happens all the time.

The point of net neutrality is that, while ISPs should not be held liable for what other entities in the network do, they shouldn't engage in those practices themselves. In logical terms, that's a perfectly reasonable goal.

That's a nice goal but it doesn't answer the crucial question: who pays for the expansion to accommodate traffic generated by Netflix?

Netflix? Comcast's subscribers? All of them or only the ones using Netflix? Level3? Comcast's investors? Tax payers? Some combination of the above? And if so, in what proportion?

This is completely impervious to "I already paid and I demand..." argument because everyone listed there already paid and wants something. Preferably cheaper or paid by someone else.

>who pays for the expansion to accommodate traffic generated by Netflix?

They are already being paid - that is the problem. They are trying to double dip.

A visual aid (not the real arrangement, but good enough for discussion:

    Netflix <-> Level3 <-> Comcast <-> Me

Netflix pays money to Level3 for an internet connection.

I pay money to Comcast for an internet connection.

Leve3 and Comcast have an agreement to connect to each other for the purpose of serving their respective customers.

The concept of "an internet connection", this thing that we're both paying for, implies the interconnects working together to deliver, in good faith, what their respective customers have paid for - i.e. access to arbitrary services at best effort speeds.

The players have already been paid everything they deserve at this point. If they feel they are not being paid enough, they should raise their prices.

Comcast, they decide they want to double dip and charge Netflix for something I am already paying for. They are no longer playing in good faith, and I argue not delivering what I pay them to do. Why? Comcast's argument carries the sneaky assertion that they deserve money from Netflix since Netflix consumes such a huge portion of their capacity.

Except that doesn't wash - it is their customers requesting those resources and consuming capacity.

They could throttle their customers instead of Netflix, and this would even be defensible, but it would get them pilloried in the marketplace. But that assumes it's a capacity issue in the first place..

But in the end, it isn't. That is a damn lie perpetrated by the ISPs. It is a politics problem. Note how, when Netflix agreed to the demand for protection money, their capacity problem with Comcast went away literally overnight? Comcast either has an army of the best and fastest network engineers in the entire world (and I invite you to speak to a Comcast user if you think that's true), or already have the gear and configurations in place and just won't switch it on.

There is no cost for expansion here. It's money grubbing, pure and simple. ISPs have always been expected to steadily grow their capacity and speeds over time, and just now Comcast decides they have an issue with it? It's BS. It's a business decision - some suit decided they could charge twice for the same thing and do no extra work.

They're not being paid already because the explosion of streaming video traffic is a relatively new and ongoing process. They expect to be paid more for moving more bits.

Now, you might expect the capacity of the network to simply increase as a matter of technological progress. But it's not that crazy of a notion to charge more for doing more. They could make their own customers pay but they're trying to shift it to Netflix, and indirectly to Netflix users. There's nothing fundamentally unfair about it.

The answer that they shouldn't charge anyone more amounts to financing the expansion by their investors (in the form of lower returns). It's not a political problem, it's an economic one. All of those companies are after money, whether in additional fees or lower costs.

The fair and forthright thing to do, if that's really the case, is to charge me more. I am their customer, Netflix is not. Comcast's job is to deliver what content I ask for.

Except, they won't do that, because an ISP who will play in good faith will come along and eat their lunch, and their plans are already absurdly expensive.

It's not as if Netflix is blasting unsolicited traffic into Comcast's network that they should somehow compensate them for the inconvenience. "Oh, So sorry! That neighborhood scamp Netflix, blasting their packets all over the place".

The fact that it's Netflix is utterly irrelevant in truth. In the end, Comcast's customers are the ones requesting the data. It's just the simple fact that it's all coming from one source starts the wheels turning, where if it was more spread out, they couldn't come after any one person in particular to seek rent.

>Now, you might expect the capacity of the network to simply increase as a matter of technological progress.

As it has been for the past couple of decades? Yes, that is exactly what I expect.

I'm not saying they shouldn't charge more. I'm saying they shouldn't piss on the internet's leg and tell them it's raining. Don't come to us with that 'But but but capacity!' argument when their behavior with Netflix clearly indicates the opposite, and less so when you're a monopolist with the second worst customer satisfaction score in the entire country, and even more less so when it's their damn problem in the first place!

On top of that, streaming video is a time-sensitive medium and others are not. The sane thing to do (again, assuming this is really a capacity issue, which I absolutely believe is 100% grade A horse manure) would be to throttle down the other, non-time-sensitive packets like torrent, http, mail traffic. Basic QoS.

If you believe this company's stated reasons for anything, you are being played for a fool. They can not be trusted.

Who pays for the expansion to accommodate traffic generated by Comcast customers requesting data from Netflix? Comcast? ...

In a world with ISP competition, an ISP that chose to purposefully neglect its peering connections, or to throttle one of the most popular services on the internet, would lose customers. Unfortunately, we don't live in that world.

I don't understand your point. If my computer hooked up to the internet can request traffic from another computer it sees on the internet then from my perspective I have that connection. You are trying to somehow say that an ISP offering unlimited access to the internet has to be held responsible for a server that refuses to be seen by my computer. I'm not making that claim. You are trying to refute a claim that isn't being made.

I see "unlimited access" to the internet as being my ISP is not preventing me from accessing what can be found on the internet. You are twisting definitions around to make it mean how you wish it to be defined.

It's not like I'm claiming "unlimited access" to public roads includes private driveways.

If there were no public servers at all on the internet, an ISP can still offer unlimited access to the internet despite there being nothing to see.

Also, there's a difference between "everyone paid something" and "everyone paid something twice". Your own argument supports the notion, it's a negotiation and our request in the negotiations is that bits are not paid for twice in the same transaction.

> Paying your ISP doesn't necessarily grant you unlimited access to every other network on the Internet.

That's actually the whole thrust of the argument, as I understand it. Net Neutrality says "If I buy internet I should get all internet at the flow rate I bought internet."

Content providers would buy a flow rate, consumers would buy a flow rate, and the people providing that flow rate (e.g. Level 3 for the CDNs, Comcast for the consumers) would deliver that flow rate, regardless of what or where it is going.

Yes, by all means, let us have the Internet enjoy the same level of success as magazines!

This idea that the senders haven't paid at all is absurd; even more absurd is the idea that we want to move to a model where whoever has the deepest pockets should have the loudest voice on the 'net.

Don't people see that the value of this whole thing is that anyone can be heard by anyone else who tunes in? Adding these other barriers to entry, just to make a quick buck, is bullshit.

> There is nothing fundamentally wrong with charging both sides. You may prefer a different fee structure but a better argument than "I already paid for it!" is necessary.

Fair enough. Eminent domain can also be used to seize the cable plant and put it under ownership of a non-profit/municipal broadband provider, who won't gouge those who request/send bits.

The knife cuts both ways.

That's been attempted. Many municipals have tried to roll out their own broadband to be sued by the industry to prevent it.

Although eminent domain is an interesting argument, not that I agree with it necessarily.

> Although eminent domain is an interesting argument, not that I agree with it necessarily.

Why? Public funding and a government-granted monopoly were the reasons local ILECs have the power they do; they'd be given fair value for their gear in the ground.

Well, my feeling on the matter stems from the fact that eminent domain is an incredibly powerful tool that can potentially disrupt a large number of people's lives. Like all such powerful government tools it is easily abused. It is especially terrible when it is abused because it essentially, and legally, removes people's property rights. Many think property rights are a cornerstone to personal freedom.

Essentially, eminent domain should be used as a last resort in all cases. No matter how trivial or non-invasive to the public the seizure may seem. If a local municipal moves in to seize something like that, any other company may think twice of implementing a similar service in the future. If they perform the seizure and it works out, what else could they seize for the greater good?

I would rather they remove the government-granted monopoly to open competition and perform an audit to make sure the company fulfilled their part of the bargain in receiving public funds. If they fail the audit, request the money back.

> I would rather they remove the government-granted monopoly to open competition and perform an audit to make sure the company fulfilled their part of the bargain in receiving public funds. If they fail the audit, request the money back.

We are in agreement then.

I don't think your argument holds water. Netflix and Hulu don't get their internet connections for free. They pay for the hookup just like we do, except they buy their bits in bulk.

Incidentally, this is also how phones work. Both sides pay their own phone companies for the connection. You would be appalled if Verizon decided to start charging T-Mobile extra so that T-Mobile users could call Verizon subscribers.

The argument that "I already paid for it" is compelling and final. I paid for a service, if I don't receive that service I've been robbed. It's as simple as that.

The "other side" of those bits is paying their internet service provider. They pay their ISP, you pay your ISP, why should they also have to pay your ISP?

Err... because the contract you make with your ISP is not "I'll pay for outbound packets, inbound are free because somebody else is paying for that." The contract is "I pay for this much maximum bandwidth in each of two directions, and you make a good effort to deliver my packets and return what gets sent back."

I would think there are few comparisons because in the long run the idea is not logical and few would agree to such an arrangement unless they were forced to. There is nothing fundamentally wrong with it, but it's not always a sound business model. I'm sure it's great when you can make it happen but it would be easy to disrupt by a competitor.

It would be the equivalent of a third-party delivery service charging the buyer a fee to deliver and the seller a fee to deliver the same purchased item with their delivery trucks.

Now, some may do that; but it would be easy for a competitor to see the consumer unfriendliness of such a practice to step up and say "we only charge once per delivery!". That would likely be the end of the Double-Dipping Delivery Service because both the buyer and seller would be better off with this new service.

But that's the key, many of these ISPs don't have competition so they can try to enforce such a consumer unfriendly practice.

The question is whether am I paying to have bits delivered that I request or am I paying to have access to the Internet so that bits can be delivered to me if I request them? Based on the sales pitch and pricing from my ISP, it is clear to me I'm paying to have bits delivered.

If the ISPs will lower my pricing to be a connection and then charge companies to deliver bits to me, then that's something different. There would have to be some financial agreement between me and that company as to how to cover that cost. That could either be a direct payment transaction or I have to watch ads to cover their costs.

Netflix could charge me $20 a month if my high-speed connection was $5 a month, or even free, since they charge Netflix a nominal fee to deliver bits to me.

As far as I know, no ISP has ever sold Internet connections that way. So for them to claim that both parties need to be charged for bits delivered is attempting to double charge on a bill of goods already sold. The argument is more like, "Somebody already paid for it!" and is quite valid.

Your magazine example fails because the ads conceivably subsidize the cost of delivering the magazine to me. Without the ads, my pricing for the magazine would be higher. To be accurate to the argument, the magazine would need to charge full price for ads and not subsidize the cost of getting the magazine to me at all, charging me full price as well.

What if we look at it from the other side? Here's an argument that's very similar to yours:

What is the source of the notion that, because Netflix paid for bandwidth, all bits are paid for and the charge for carrying them cannot be split with the other side of the connection? Why is it so bizarre that both sides of the connection have to pay for it? Because you're used to your phone working differently?

As an analogy, you know how you used to pay for cable and the cable company paid money to stations they showed you? The TV station split its fee in two: the cable company paid part of it, and the advertisers paid the other part. It's the same here.

If you're going to say that one side gets to charge at both ends what makes the receiving side special? Why shouldn't the side with the content get to double dip instead?

But it is split, or let me assure you, my AT&T ADSL connection is very much not free. TV is not apropos because in the US subscribers don't pay for it (compare to the U.K. "television license" and advertisement free BBC: https://en.wikipedia.org/wiki/Television_licensing_in_the_Un... ); cable is paid for, and there are or at least have been commercial free channels on cable. That plus a lack of censoring was part of HBO's unique selling proposition after all.

What exactly is the point of your post? So far everyone has agreed that both the company hosting the server and the consumer are paying for their connections.

You're presenting a valid argument, but I don't think that you can equate postal distribution system with the internet.

The nature of the internet is such that it's a public network assembled out of a number of private networks. It works because these private networks meet at exchange points and transport each other's traffic at no cost. Everyone pays for access.

The difference between the Internet and the USPS is that the post office is a closed loop where the sender pays to admit its payload into the network. The post office delivers mail to me at any volume -- I don't have to pay to receive mail.

The whole point of the internet was to escape the rate tariff structure imposed by AT&T back in ye olden days.

Reply on news.ycombinator.com