Massive Comcast Outage Hits the US | Hacker News Reader

Massive Comcast Outage Hits the US

newsweek.com

280 pointscrazysane218 comments

Comcast's confirmed the fiber cut: https://twitter.com/comcastcares/status/1012767042158510080

> One of Comcast’s large backbone network partners had a fiber cut that we believe is also impacting other providers. It is currently affecting our business and residential internet, video and voice customers. We apologize & are working to get services restored as soon as possible

They seem to be at least starting to route around it now; I was having intermittent connectivity issues to various sites here all morning, but as of ~45 minutes ago, I'm not having any issues reaching the sites I couldn't reach earlier.

Best current info is as follows:

There is a fiber cut somewhere between NYC and Chicago.

At the same time there is also a fiber cut between Ashburn, VA, and locations in South Carolina. The cut is somewhere in NC.

Copy and paste from Level3 notification system:

"* CASCADED EXTERNAL NOTES 29-Jun-2018 15:13:37 GMT From CASE: 14815828 - Event Field Operations dispatched to the estimated failure location and upon arrival was advised by local law enforcement there was a truck that struck a utility pole, which then fell across the street and struck another pole, which then tore down the aerial fiber lines. In addition to the fiber lines being down, live power lines are down as well. OSP will be dispatching a fiber repair and construction crew to the site; however, until the local utility company has completed their repairs and deemed the area safe, fiber repairs cannot begin.

* CASCADED EXTERNAL NOTES 29-Jun-2018 19:21:05 GMT From CASE: 14815828 - Event The OSP has contacted the personnel transporting the replacement fiber, and they advised they remain en route, and on schedule for their ETA of 20:15 GMT. The next update will be provided upon their arrival, or as new information becomes available.

There are massively important fiber lines being hung from a utility pole?

It's more common than you think, a lot of long distance links between cities are a varied combination of aerial, direct buried trenched armored fiber, underground fiber in ducts.

There are also a lot of long-distance links carrying DWDM traffic that are >95% aerial, not on poles that can get hit by trucks, with singlemode fiber inside high voltage AC transmission lines (https://3ee2108sdg.files.wordpress.com/2014/07/cropped-20140...).

Yeah I understand aerial lines carried by those large towers shown in that image, but utility poles that can get hit by trucks seems problematic. Partly for accidental incidents like a truck crash, but also partly because of how easily it can become an attack vector.

I guess the plus side is that utility pole lines are probably the easiest to fix when cut.

Aerial fiber (ADSS or lashed to strand) is like 1/15th of the cost to build per km compared to underground, 40 inch buried trenched.

A lot of the fiber built during the dotcom 1.0 boom with VC money was trenched along railways or pipeline right of ways.

I do fiber runs and whenever possible it is strung up on poles. It is far faster and easier than doing underground cabling because there are no utilities or gas lines or what have you to possibly hit. More of the stuff put down 15 years ago was put underground but a lot of it now is just lashed around current telephone lines.

Yeah and for last mile lines it makes complete sense, no problems there. It just seems crazy that a utility pole holds a massively important link between two cities. But I guess that's just how it is.

Yeah seems crazy but not crazy since traffic can always be rerouted and once the line is cut you immediately know its out. I think the bigger issue is that people actually experienced an interruption of service.

Much easier and faster to repair. The key to keep in mind is not if a break happens but average downtime over the year.

For the infrastructure-level network admins out there, is there different lingo for an intentional and accidental fibre break, or is "fibre cut" used for both? I'm just thinking 'cut' is a pretty intentional word to use here.

“cut” is used for both. Common causes are digging/construction work, maintenance by other carriers sharing the same right-of-way, weather (mostly for aboveground runs and equipment), and occasionally stuff like animals (“squirrel chew”), hunters shooting at things, and equipment failure that doesn’t fail over to redundant gear properly.

Intentional, targeted damage is really rare, even when multiple paths experience concurrent outages. The fact that 2 paths were affected isn’t enough to assume it was intentional.

I have seen a lot of people digging up the fibre believing it's copper wires they can steal.

Not one of those admins, but generally the term "fibre cut" always seems to be related to "backhoe operator accidentally digging a hole through the buried fiber line" when the final reason for the outage comes out.

Fiber cuts happen all, the, time. The reason users don't notice is because networks use redundant links. There's really no way to know without doing some serious investigation if it's intentional or not. But my guess would be accidental. A large network like Comcast/L3 is likely dealing with multiple cuts to fiber on a daily basis.

A fiber cut can happen from innocent enough things, such as digging without getting the proper surveying done first.

Sounds like it was 2 cuts, which likely means it was intentional IMHO:

NY| Manhattan| Communication Failure| 30 Rockefeller Plaza (Comast)| Comcast is experiencing a National outage due to 2 fiber line cuts. One between NYC & Chicago, the second between Ashburn & South Carolina. customers urged NOT TO CALL 911, no ETA on a fix, police monitor

Edit: source below. Thank you dfee.

I work for an ISP and have several times seen totally coincidental dual fiber cuts, in totally different places 150+ miles apart, within the same 8-12 hour period of time, not due to any malicious activity. Just bad luck and assholes crashing trucks into power lines, railroad bridges getting washed out in floods, etc.

Why is it not more trivial to auto-route around those kinds of events?

In many cases BGP does its job just fine at layer 3. But budgetary constraints and financial reality for the cost of really huge interconnection mean that a cut may result in a 100 Gbps PNI peering session between two ISPs filling up and flat-topping its traffic chart (very, very bad) when the traffic re-routes. One example is that as a result of today's events there was apparently major packet loss and congestion between HE and Comcast in the NYC area.

Or when traffic re-routes, a major ISP will temporarily congest its own intra-AS city-to-city links on a few key paths, for their own router-to-router interfaces.

Ah, makes sense, you still need the ability to handle the rerouted traffic.

Well precisely. And not like Comcast doesn't have money aka "But budgetary constraints and financial reality...". The reality is that they do annual surveys and find out spots on map where cutting happens very rarely and in those places they won't have any redundancy running - something, you know - the whole internet is based off of.

So no, two cuts like this are not something out of ordinary. The fact Comcast is cheap and don't have redundant pipes/contracts to operate secondary backup lines is simply being cheap and saving money on their part.

given the scale of our fiber networks, i'd expect coincidences like this.

the more cynical side of me says that an intentionally crashed truck looks the same to outsiders as one that was crashed to cut the fiber line.

but i really don't think it's the case that there is foul play here today. this is where calculating p values comes in handy though.

> customers urged NOT TO CALL 911

Do people really do this? I can't even comprehend this.

Yes. http://komonews.com/news/offbeat/bothell-police-dont-call-91...

I'm wondering if their 911 service is having problems as a result. If that's the case then it's more of a "try not to need 911, but if you do try to use another network"

I had to read that a couple times to make sure I wasn't missing something.

Wow. It's sad that has to be said. Do people not get fined for abusing 911 like that?

People call search and rescue for blisters too.

I _reeeally_ want to hear those 911 calls. Please, somebody.

Lol, customers call 911 when Comcast goes down?

And they vote, too.

Non Bitly link: https://twitter.com/joangralla/status/1012761031922409473?s=...

Thank you. Sorry I messed up the link.

Intentional ... hundreds of miles apart? .... Sounds far fetched unless you like conspiracies.

Is there any other source to suggest 2 cuts rather than one?

Can confirm 2 cuts, L3 to blame apparently.

Maybe Comcast is seeking some of that AT&T money after they were exposed for housing NSA spy tech.

Our users are beginning to recover from the issue as well.

> They seem to be at least starting to route around it now

Isn't IP supposed to do this automatically?

IP itself doesn’t, that’s BGP. I’m not sure what it’s limitations around large links going down are though.

On Comcast here in the bay area. Some sites are fine. Some are just gone. Luckily, my corp VPN endpoint is accessible, and if I VPN out, I can get to everything again.

It was the same with Denver. I couldn't hit a handful of sites including github for almost an hour.

Same thing on Siesta Key in Florida. Couldn’t reach home or github, wife couldn’t reach Hulu. Google worked fine.

"Fast lanes" got accidentally turned on a bit too early, I'd guess.

I was watching packets get dropped in Level3's network in trace routes, I was getting out of Comcast just fine.

Maybe they were overloaded from all of the additional traffic they got sent?

This leads me to believe they are testing to see what they can get away with for fastlane access.

Or it could be, ya know, a run-in-the-mill network routing issue.

I'll be that guy: run-of-the-mill

If what you're saying turns out to be true, it would be very disturbing.

Disrupting your paying customer's access with unannounced 'tests' is a sure-fire way to torpedo your brand value and rating.

Comcast's brand image and reputation is largely irrelevant - they enjoy captive markets, little competition and are mostly defensive in their brand posturing. There is little choice in the markets they are in and they know it...

“Comcast” and “brand value” do not go together very well as it is.

This is true.

When your company name is so hated by the public that you have to re-brand your flagship product "Xfinity" to distance it from yourself, you know you're running a crappy company.

They have been consistently rated as the most-hated consumer brand in the US. There's nowhere to go but up.

https://www.pcmag.com/news/350979/comcast-is-americas-most-h...

Doesn't really matter considering Comcast's customer's only option is to move.

Yeah and then most Customers still have no alternative.

As a non-American, I find to hard to grasp how a country that places super high priority on access to choices and options can get stuck with just two major parties and just ONE ISP (in most places, that is).

>As a non-American, I find to hard to grasp how a country that places super high priority on access to choices and options can get stuck with just two major parties and just ONE ISP (in most places, that is).

Distance.

America has states and even counties larger than many countries. And even where there are lots of people, population density is typically less than most other nations.

Distance is a killer in last-mile applications like internet.

See also: Australia.

In Germany the owners of physical lines must rent to ISPs. There aren't usually multiple physical last mile lines, but you still have multiple ISPs. That's at least how it was ~10 years ago. But that would be super unamerican. Free market first, especially in areas where the free market doesn't really work and benefits large companies that make massive campaign contributions. (Disclaimer: I'm pro-free market, but generally anti big Corp)

> But that would be super unamerican.

I know what you mean when you say this, but there are strong differences between American ideals and American realities. When there's a conflict of interest between a large company and consumers, I think it's a safe bet to assume the company in America comes out ahead.

Sweden is about the same size as California but with about one third of its population (and about 2/3 of the population density of US), still it is in the top 10 internet access list in about every metric you can think of (together with for example Norway and Iceland, neither especially densely populated) so I don't think the density argument holds.

Stockholm and LA are excellent examples. In Stockholm, there is a municipally owned company that builds fiber. It received no government funding, and built the fiber over more than 15 years in a demand driven way (building first to businesses, then to places with highst demand). It leases access to ISPs, and there are no mechanisms to force it to subsidize lower income or disadvantaged people.

Contrast Los Angeles. There, Stokab’s business model would be illegal. Builiding out based on demand might mean that wealthier neighborhoods would get fiber a decade before poorer neighborhoods. Disproportionately, white residents would get fiber before hispanic residents. That would be politically untenable (and would be completely impossible to put the government’s imprimpteur behind such an effoet as with Stokab). For that reason, most US cities make Stokab’s business model illegal.

Los Angeles had a fiber proposal: https://www.wired.com/2013/11/la-fiber. It tried to get companies to build a fiber network. By contrast with Stockholm’s approach, neighborhood income and population density could not be a “factor” in the rollout. That means that any ISP would have to build to new neighborhoods that were not economically justifiable. The ISP moreover would have to provide a minimum level of free access to all residents. It would thus have to recover the cost of that from other residents, driving the price higher and decreasing the competitiveness of the product. Unsurprisingly, nobody took up Los Angeles on that proposal. Nobody would build it, it made no business sense.

In Stockholm, fiber was a simple business proposition, built with private capital. In California, it was a social justice initiative, unattractive to private capital. Which is not necessarily itself a problem, but if you want to do that you need to be willing to build it with public money.

Also, New Jersey is about as dense as many European countries that have great broadband and competition--yet New Jersey has crappy broadband. So, it's not due to USA's low population density.

If New Jersey were a country, its average internet speeds would be in the top 5, right after Sweden. And far ahead of comparably dense UK. https://www.akamai.com/fr/fr/multimedia/documents/state-of-t...

6 years ago in Jersey City I had Fios, while across the river in Manhattan, it was very difficult to get. In my part of New Jersey in 2012, I had far better internet than I did in 2018 France — and much cheaper.

Think per-unit density, not per-geographic division.

On average, houses in Sweden and closer together. Apartments are smaller than the U.S.

Perhaps instead of "last mile," the term should be "last meter."

Sure, Swedish cities are a little less sprawly than American cities of the same size (I have no data on this but my general experience is that both apartments and houses a slightly larger in Sweden but a bigger part of the housing stock is in apartments), but I still don't think the geographical explanation holds especially in densely populated regions like the Bay Area. I think it has more to do a lack of interest from US politicians.

The Bay Area isn’t densely populated. San Jose to San Francisco is mostly single family homes in suburban neighborhoods. Even the “dense” parts don’t hold a candle to East Coast cities.

We place super high importance on access to choices for yourself, not for others. rich people and businesses are on a higher tier as well.

Not really. We place super high priority on freedom from government influence. It's OK for a company to fuck you over but not the government for <reasons>.

Edit: I know the <reasons>, but the debate about that is not particularly relevant here.

We cannot escape the government. We can choose to ignore any or all companies. If I want to, I can lock myself in my house, disconnect from society, and subsist on my backyard garden. But there is no escape from the state. Even if I go completely off-grid and never leave my property, the state still comes banging on my door, demanding a rent payment for property I (supposedly) own -- i.e., property tax.

> We can choose to ignore any or all companies.

Except you can't actually do that. There's a huge list of companies you simply can not choose to ignore. That's sort of why monopoly/oligopoly is so evil, and why strong independent antitrust action is so critical for preserving freedom.

Yes, you can escape all companies if you live like a hermit in a prepper dungeon on your property (that you somehow purchased while 'ignoring all companies' with money you got ahold of by 'ignoring all companies'), That is technically one form of freedom. But most people define Freedom a bit more broadly than that.

Thanks for commenting. I understand your position, but the extreme "hermit" case is exactly what I'm talking about. I didn't say anything about purchasing a shelter. I can hang leaves over sticks like humans have done for millennia. I am owed nothing by the world and I have no right to demand that companies provide me with anything.

I agree that I probably understand "freedom" differently than most, but my definition demands nothing from my fellow citizens (which is what "companies" are made of). :)

As much as I despise these companies, I wouldn't jump to that conclusion just yet. This would be an arrogant and very stupid move, even for Comcast. Doing something like this intentionally is just asking for legal ramifications. I think if they were to test fast-lane features they wouldn't do so by blacking out a huge percentage of their customer base.

No this is just run of the mill shotty comcast service.

Shoddy

I chose to read that as "shoddy"+"shitty"="shotty". And a new word is coined! Although not to be confused with shotguns and (apparently) some sort of pipe.

Curtesy urban dictionary:

A variation of the bong native to SouthWest England, where a straight tube is used instead of a conventional "gauze type" bong tube.

The end of the shottie tube (also known as a downie or downpipe/tube) is plugged with a small amount of rolling tobacco, onto which a thin layer of crushed or chopped cannabis is applied. The user then lights the cannabis/tobacco plug until such time as all of the cannabis glows red and the plug can be seen to slide down the tube. At this point, the user gives a sharp tug to pull the cannabis/tobacco into the bong water, the smoke is then inhaled, providing the user with a more intense hit than that provided by conventional smoking paraphenalia, this means shotties provide a both economical and enjoyable smoking experience

Yeah I really don't care about spelling corrections from strangers on the internet. most people don't either, you should stop wasting your time making pointless posts like that.

> Customers were also taking to Twitter to reach out to the company about the possible outage. The company recommended that some users try checking the status center online, which would require an internet connection of some kind.

Got to love you need internet to check the status of your internet :)

To be fair those customers were reaching out online via Twitter.

People have phones.

Phones have browsers.

Yeah, because nobody has a second internet connection through their phone these days, right?

deleted

Except we're not talking about real work, we're talking about checking a status page.

Good point. Retracted.

This may just be my personal preference, but please don't delete comments like you did above just because someone made a "good point" that convinced you you were wrong. If you want to retract it, edit it so there's a line at the top saying that you retract it. Even wrong opinions (I'm assuming it was "wrong" for the sake of discussion) can contribute to a discussion.

No, you can use a cellphone data plan to check after disabling your phones local wifi connection.

Comcast's oauth server was down for me, so I couldn't log in to my account like they recommended.

What's that NANOG .sig about 'always carry a piece of fiber, that way if you get lost just bury it and when the backhoe comes by to dig it up you can get directions back to civilization.'

"I always bring a length of fiber with me while backpacking. If I find myself lost, I just bury the fiber and a backhoe should be along shortly to dig it up." --Sean Lally

Similar, not the one I'm thinking of though.

ah-HA! We'll string it up on poles, no way the backhoes can get it up there!

This is why it would be better to pay by bit consumed, like most do for electricity. This would encourage carriers to build more infrastructure. It would also create an economic incentive for the ever increasing number of Internet connected devices to be more conscientious of data usage. Of course, we can't have that in the oligopoly situation in the United States.

I'm not sure that makes sense - bit consumed neither reflects the value of the service to the customer, nor the unit economics of the provider.

You want a service level guarantee (99.9% available, with some extra wiggle room for peak hours and degradation. With electricity, you're paying for energy generation, plus grid maintenance. With internet, you're only paying for grid maintenance.

It reflects value to me. I want my connection to be as fast as possible. If I pay a fixed monthly fee, the incentive to the ISP is to throttle my connection at some point. If they have no unit economics, why do some ISPs throttle and cap after hitting a maximum?

The OP didn't say they have no unit economics--just that bits consumed doesn't reflect their unit economics.

A wired ISP's costs are primarily determined by capital improvements to their network, and increasing the reach of their network. The marginal cost of providing service to someone in a neighborhood that already has cables built is effectively zero - unless that households additional usage at peak hours causes congestion at the trunk.

The econ 101 solution to this is variable pricing based on hours of the day, to shift consumption to off peak hours. However, that's not attractive to ISPs for a few reasons:

1. Customers don't have a good understanding of how much data various activities use.

2. Just like airline tickets, you may be paying substantially more or less than your neighbor for the same service. ISPs do not want to add additional pricing transparency or control - they would much rather adjust the rate as high as they can regardless of usage, then lower it for individuals newly signing up for service or when existing customers threaten to cancel.

3. Many large ISPs also own media and cable TV businesses that are in direct competition with high bandwidth services like Netflix. Putting caps on that apply regardless of time of day directs users to their own offerings, which do not count towards the cap. It's also a hedge to gain revenue as cable dies, and extract rents from it's replacement.

4. If you run a gym, your best customers are the ones who show up least often - the same is true for the current ISP model. A cap is a good way to shed heavy users and provides a chilling effect on existing users who don't have a good grasp of computer networking.

How about bid on a variable price per bit? Just like is done with bitcoin transactions. If you are sending an important file, in an important conference call, or are receiving an important download that you don't want to wait long for, then you choose to pay more to get faster and more reliable service, but if you are streaming a youtube or doing something in the background, then you select a lower price for moderate speed & reliability.

You're describing the exact thing net neutrality seeks to prevent--pricing based on the type of data being sent.

I think it's dangerous to let those with deep pockets prioritize their traffic over mine.

> "it's dangerous to let those with deep pockets prioritize their traffic over mine"

If only a finite amount of bandwidth is available to the world, then yes, the rich will be able to spend more to take bandwidth from us.

However more cable can always be added, so what would actually happen is that the pie would grow.

> However more cable can always be added, so what would actually happen is that the pie would grow.

This assumption is the same one behind energy grids, and assumes an ideal world with no bad actors.

Instead, what happens is that incumbent players with the deepest pockets buy up all the (relevant) competition and then corner the market on capacity.

That sounds like a user-hostile, free market dystopia.

Technically this means limited-data mobile connections should be the best in the world, because the carrier has an incentive to build good infrastructure?

Let me check. IPv6? Nope. Public IPv4? Nope. HTTP interception proxy? Check! Peak bandwidth? Only a handful of mbps over LTE. And this is in markets like the UK and France where there is supposedly “competition”.

This model doesn’t work, the carriers will still do the bare minimum that they can get away with.

No, it just means that carriers who charge by consumption would have aligned incentives. If there is no effective competition, the market will still fail.

No it wouldn't.

Please elaborate.

We currently have this issue with mobile data and it doesn't do a damn thing. Engineers don't optimize for mobile data consumption beyond "download for offline usage," and ISPs don't build additional infra based on consumption: it doesn't affect the industry on any level whatsoever.

OP would need to elaborate on why it would in the first place. Bald assertions need only be met with bald assertions, especially when those assertions have real world examples negating them.

Allegedly the outage due to a utility truck in North Carolina knocking down a few power lines. When I heard this, I realized it could've been what happened in my hometown!

There was a video going around Facebook of a truck taking down several utility lines in Wilmington, NC.

https://www.facebook.com/110795228128/posts/1015572616317812...

In case you don't want to go to Facebook:

http://www.wect.com/story/38541106/video-accident-with-crane...

Thanks! Wasn't sure where to find the non-facebook link!

Streamable mirror: https://streamable.com/8qb3i

Been ongoing for us for almost an hour - was surprised to see it not posted here. We cannot reach any of our US-East AWS resources at the moment. Getting about a 50% success rate on other sites - Facebook (& Co.) pages load but no media displays.

You have to pay extra for fastlane access if you want to go from Comcast -> AWS.

Thanks Ajit!

Serious question, anybody try VPN, and then connect to AWS?

I didn't notice for a while because my primary Amazon Workspace wasn't affected, but when various requests weren't working on wifi but were working from my cell phone, that's when I figured out something was up.

Posted ~30 minutes ago, here: https://news.ycombinator.com/item?id=17426058

I work for an ISP, seems the issue was localized to Comcast's AS. Probably a network engineer made a change that went bad :)

A backbone fiber was cut, lots of ISPs are going to be making network changes!

Seems to be caused by third party fiber cut

Good thing that the internet was built for redundancy so single cuts don't affect the connectivity.

It actually looks like two simultaneous cuts in this instance.

We’re talking Comcast here so they are obviously too poor to be able to afford redundant links.

possibly... https://www.reddit.com/r/sysadmin/comments/8uukck/comcast_le... read 1st post

I am pretty confident because of my sources

I use them at work and home (no choice). From a technical perspective, Comcast is usually rock-solid for me. Sure it's expensive, and an absolute joke that I have to call every year to keep the discounts and a TV plan that I don't use, but I'll give credit where they have actually done a good job. I'd much rather a stable service that's a bit of a pain on the billing side than a terrible service that has a nice website.

I also use Comcast at home & work. In my experience, my home service has been more reliable than work. My office is in the northern Chicago Suburbs, my home is in the western Chicago Suburbs. I've literally had days when I've had to leave the office to work from home, because, at least from home, I could still VPN into our data centers in NY/NJ from home, but no access from our office.

Yes, everyone loves to complain about the cable company but in my experience Comcast's network, technicians, and support people are the best in the business. You could not pay me to use CenturyLink, for instance.

> in my experience Comcast's network, technicians, and support people are the best in the business.

in the context of US based ISP 'businesses', that's like saying 'the smartest kid on the short bus'

CenturyLink seems to have far fewer phone outages nowadays than they did e.g. three years ago. One time by some miracle I got someone on the phone who wasn't a "basic" CSR. He admitted that they knew why their outages lasted for hours. Whenever they had a fault that required a system-wide restart (!), they just rebooted everything at once. So there were millions of modems trying to talk to the RADIUS/DNS/etc. servers all at the same time. Which didn't work at all. I asked if anyone had considered a rolling reboot. "Yeah, that's what we've been telling them..."

Heroku is currently very, very slow loading. This is the traceroute:

  traceroute dashboard.heroku.com
  traceroute to dashboard.heroku.com (151.101.2.49), 30 hops max, 60 byte packets
   1  router.home (192.168.xxx.xxx)  0.355 ms  1.103 ms  1.074 ms
   2  96.120.4.xxx (96.120.4.xxx)  9.575 ms  14.987 ms  15.859 ms
   3  ae113-rur01.d9chamblee.ga.atlanta.comcast.net (96.108.191.229)  18.802 ms  18.846 ms  18.882 ms
   4  ae-29-ar01.b0atlanta.ga.atlanta.comcast.net (162.151.88.217)  15.913 ms  15.978 ms  15.733 ms
   5  be-7725-cr02.56marietta.ga.ibone.comcast.net (68.86.93.125)  19.719 ms  19.659 ms  19.528 ms
   6  be-11486-pe03.56marietta.ga.ibone.comcast.net (68.86.85.254)  16.524 ms  17.280 ms  16.821 ms
   7  * * *
   8  * * *
   9  * * *
  10  * * *
  11  * * *
  12  * * *
  13  * * *
  14  * * *
  15  * * *
  16  * * *
  17  * * *
  18  * * *

I think this is consistent with backbone / fiber cuts. Local networks in the Atlanta area are pretty consistently okay. Other stuff is just down or slow, very slow.

We're also having issues routing through Telia in ATL.

At least this outage is not local. It took me months to get them to replace the coaxial aerial run from the pole to my house that had been compromised and had liquid water inside the cable and enough pressure to force said water around the cable loops, inside my house, and into the splitter. Water-caused signal attenuation was making for some bad service quality.

I need to consider my plans for obtaining backup Internet. The only real choice is 4G and the only way to pull that off is to somehow run a hoist up into one of the very old, mature oak trees on my property to get an antenna way up above the house (thinking like we do with ham radio antenna runs). I have no usable cell signal in the house except through the personal cell that uses the comcast connect to get back to the carrier.

Before letting one's walls fill with water, one might have made a small hole in the jacket just before the cable entered the house, to let it run out where it couldn't do any damage...

Good to know its just not me. I finally had to call it quits and head to Starbucks with the laptop. My home office is in a no-cell signal black hole so when this outage hit, my cell phone goes out too until I drive up the street at least a few houses.

I just realized the huge downside to having moved my cell phone from Verizon to xfinity mobile. Although neither my wired Comcast or my cell service is out right now, I would be screwed if these both went out.

Xfinity Mobile is just an MVNO (Mobile Virtual Network Operator) of Verizon. So it is Verizon's towers/networks that would be providing your connectivity, not Comcast. Comcast basically just handles your billing and customer service.

Good point about using the Verizon network. They also use their own access points for internet service, which could have an impact.

I’d be more worried if they take Verizon traffic and route it across their own network, which most MVNOs don’t — but Comcast might for data collection purposes. Hopefully not!

I sincerely hope this is a technical malfunction and not a co-ordinated to attempt to take down Comcast's infrastructure by malicious actors, domestic or foreign.

Non-localized outages (especially a coast-to-coast one) are signs of deep faults with the network architecture -- not surprising given the way Comcast is run.[0]

[0] https://www.pcmag.com/news/350979/comcast-is-americas-most-h...

I'm on Comcast and cannot get to Twitter and a few other places... But Netflix and YouTube work just fine. I wonder why I can get to some stuff but not others...

Netflix and YouTube have caches inside Comcast's network. That'll probably be why they are still serving?

https://openconnect.netflix.com/en_gb/

https://peering.google.com/#/options/google-global-cache

I'm on HN...seeing your update...I an connect to my employer remotely... Feels like Comcast is not the whole story...

HackerNews is served from Cloudflare, which employs caches close to you as well. Seems like Comcast has a problem routing over distance.

Seeing talks about 2 separate fiber cuts... Lots of issues all over the US. Chicago to NY and also another cut that goes south to SC. r/sysadmin has good info

Discussion here reminds me of Rule #3 in Richard Cook's classic "How Complex Systems Fail": Catastrophe requires multiple failures – single point failures are not enough. http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...

My Comcast internet has been flakey for a good week.

Sadly the alternate provider offers "up to 100Mbps" ....and in my area that means 5Mbps... maybe.

I hate a lack of competition.

Here in Atlanta I had a comcast outage on this Sunday and Monday evenings last for 4-5 hours each. I wonder if the current outage is related.

About two months ago, they took the Internet down for the whole neighborhood 3 days in a row like clockwork to "split the node" (because I went and talked with the crews out on the road to see what was up). No notification, not even for business class customers. Completely bonkers!

Not related. This time it's a backbone issue. We're seeing corporate users affected.

Switching my DNS to 1.1.1.1 alleviated some of the problems (access to Google/YouTube)

I'm picturing plenty of bundled optical fiber cut in two. How big are these bundles ? How can they repair the optical fiber (i.e. how to reconnect each single fiber to its previous half ?) Do they have to reinstall a section of cable ?

They can be sizable depending on the type or how many fibers are run. Marine vs terrestrial are different as well.

Repair depends on the type of cut, too. Aerial run? Buried? Did the fiber get damaged as a result of an underground gas explosion in a nearby vault that caused several meters to fuse together? :)

Often, a splicing is necessary - that involves using special equipment to re-align and fuse the damaged pairs. Slack in the run can be used to bring the damaged ends flush, or a segment can be added to couple them.

My dad used to do this for AT&T (mainly for train derailments if I remember right). As I understood it, he would cut/polish the ends of the glass and glue together. The process may have changed since then, it’s been about a decade.

What I don't get is how they match each individual fiber on the left with its right one, as I imagine there are probably thousands and individual fibers inside a bundle. When they installed the fiber at my home they had some laser to check the fiber correspondence. Is it just that, pick left illuminated by laser fiber, pick right illuminated by laser fiber, put in a glueing/fusing machine, and doing it n times (where n is huge)? Where do they get some slack to access the fiber bundled inside ? (Do they need to add a patch section and do the above work two times, or is there some neat trick ?)

Fibre buffer is standard colours, you just match and splice.

Service loops in the run provides slack to take up, else you just cut and splice in a short segment.

That is no longer done.

Coworker in Colorado can connect to slack but certain websites fail. Weird.

Confirming from S.Florida. I can get to some/most of my AWS workspaces and GCE instances, but not to wikipedia.org... unless I turn on my VPN in which cases they block some traffic or make me go through JS verification, but then at least wikipedia works. :-(

Is your VPN IPSec or SSL or other?

I'm using PIA

Well, good thing we had our scheduled maintenance last night.

I'm not having any issues with Comcast in San Jose. Been using every normal website for approximately the past hour and a half. Odd.

try changing your DNS. I got mine working after switching to the Level3 DNS server IPs : 209.244.0.3 & 209.244.0.4

Listed here : https://www.lifewire.com/free-and-public-dns-servers-2626062

Seems like the list is legit but do verify.

downdetector.com isn't even working for me: https://twitter.com/davywtf/status/1012778410714812416

Most stuff has worked today. GitHub hasn't. IPv6 has been hit-or-miss as well.

First thing I noticed failing was Github. Then nothing for a while. I'm not home, but was able to VPN back into my home and run traceroute and it seems to have healed with the route to Github at least.

  traceroute github.com
  traceroute to github.com (192.30.253.112), 30 hops max, 60 byte packets
   1  router.home (192.168.xxx.xxx)  0.455 ms  1.454 ms  1.431 ms
   2  96.120.4.xxx (96.120.4.xxx)  10.247 ms  16.604 ms  15.646 ms
   3  ae113-rur01.d9chamblee.ga.atlanta.comcast.net (96.108.191.229)  17.851 ms  17.839 ms  17.810 ms
   4  ae-29-ar01.b0atlanta.ga.atlanta.comcast.net (162.151.88.217)  16.559 ms  16.673 ms  16.660 ms
   5  lag-5.ear2.b0atlanta2.Level3.net (4.68.71.45)  17.321 ms  17.411 ms  17.353 ms
   6  * * *
   7  GITHUB-INC.bear2.Washington111.Level3.net (4.14.98.158)  128.508 ms  122.411 ms  120.312 ms
   8  * * *
   9  * * *
  10  * * *
  11  * * *
  12  * * *
  13  * * *

> 5 lag-5.ear2.b0atlanta2.Level3.net (4.68.71.45) 17.321 ms 17.411 ms 17.353 ms

> 6 * * *

Obviously Comcast is trying out a nefarious scheme to give their own routers Level3 hostnames, to throw us off their trail, and not, say, a fiber cut. /s

Same for me. All seems fine except GitHub.

It started last night. Luckily I had two other modems and kept switching them out. Each modem would work for roughly 30 mins and then the service would go down. I gave up and went to sleep.

Some of you conspiracy theorists are insane. No Comcast didn't purposely do this. I can guarantee it was just a backhoe digging around. Happens more often than anyone would care to admit.

The fiber-seeking backhoe strikes again

Backhoes. The internet's natural predator.

Well, we all know they’ll learn to do better because of this: they’ll take all that money from monthly fees and make sure to improve quality of service! Right?

Nope. They’ll probably raise prices if anything, and receive no ill effects as a company for basically failing to provide what they happily accept money for. (Buried somewhere in an “update” to Terms of Service, they probably pardon themselves from exactly this type of outage.)

There ought to be strict laws around long service outages, resulting in automatic multiple months of free service, etc. as a deterrent for allowing things to just collapse for hours at a time. This isn’t just Internet anymore; entire businesses can be affected for example.

Of course nothing will happen. Comcast is too good at lobbying and Internet still doesn't get treated with the same legal importance as other infrastructure like road, running water or even snail mail.

For what it's worth, we let bridges crumble, communities sit without clean drinking water, and are intentionally bankrupting our postal service in the hopes of being able to privatize it. So I would say the Internet is being treated about the same.

If you're thinking of Flint, you may wish to update your reading material. It's possible your talking points could be more up to date.

It took them the better part of 4 years to bring clean water back to Flint - they only ended their bottled water program in April of this year. Those who were there during the crisis will be living with the consequences of lead poisoning for the rest of their lives. Odd that you view something this recent, and of this magnitude, as an outdated talking point.

Anyway, Flint isn't the only place in the US with water problems. https://impact.vice.com/en_us/article/wj4qvx/these-us-cities...

You're absolutely right! It took years to replace a huge number of pipes while keeping a water system working, and those affected will suffer lead poisoning so long as they live.

I've seen a lot of "Flint still doesn't have clean water!" talking points in the past month or so. Without passing moral judgment, the talking point is not reflective of current reality.

But the government shouldn't be able to run postal services! Socialism something, something, Venezuela, uga buga!

This is entirely disingenuous, the United States Constitution explicitly allows the federal government to run a post office (Article 1, Section 8).

Gp is being sarcastic (hence the “socialism uga buga”)

I meant that it's disingenuous to portray small-government proponents like he did by using a strawman of someone upset at a clear government power. It adds nothing to the conversation.

For many decades (certainly at least through the 1990s, I think) there was a popular narrative that made poor postal service synonymous with inept government. Almost every citizen regularly visited a post office, and everybody had at least one poor experience. Standing in line at the post office was the equivalent of standing in bread lines in communist countries. This shared experience made the postal service a key talking point by politicians when selling small government, pro-capitalist policies.

For various reasons this narrative has subsided. It probably wouldn't even occur to younger people to perceive in the post office reflections of a wider debate about government. Indeed, for some the post office is seen as a pragmatic answer to providing banking services to poor communities, only distantly related to or even entirely divorced from the big government vs small government debate. Actually, I think in general the debate on both the right and left has shifted away from bickering over the size of government, per se, and instead emphasizes individual rights and injuries.[1]

It's why you'll hear the echos of demands on the right to completely privatize the USPS, almost entirely from the older generation because newer conservative generations couldn't care less unless you're talking about Jeff Bezos.

[1] IMO I think that conservatives won the small government debate in the 1990s, for better or worse. There was a confluence of events in both America and Europe that effectively resolved the debate. Rather, policy proposals these days either rely on some public-private model, or simply attempt to exact concessions from private industry. The government directly doing something themselves is completely off the table. See, e.g., rayiner's comparison of Stockholm's and LA's fiber initiatives.

Snail mail is handled by one designated quasi-private firm, the USPS. Roads are often handled by contractors on a per-job basis. How would handling it like roads and USPS help?

Internet is more like water, and electric utilities than USPS. Should not have middlemen who incease fees every two years when there is no service improvement. The data these companies charge 50-100 dollars to pass on is way less than 1 cents per average customer. A utility company could easily handle the infrastructure logistics, hiring of workers totalling a bill of 10-30 dollars?

https://broadbandnow.com/report/much-data-really-cost-isps/

There's a significant difference, and that is that water and electricity can be meaningfully metered by volume, but Internet can't; that is to say, there is little to no meaningful marginal cost to your ISP for delivering 1GB of data vs 10GB. Bandwidth does have a marginal cost, but by and large data doesn't (yes, peering arrangements, it's more complicated, etc).

The USPS (at least in my neighborhood) does a far better job than Amazon+OnTrac?

I prefer USPS now. I order a lot from California, and it's to me in Ohio in two business days generally. That's with Priority Mail, which is similar to UPS and FedEx Ground. Both of those are 4-5 business days.

Beyond that it's way more economical to use, in my experience.

I thought OnTrac was bad, but then Amazon somehow figured out how to provide even worse service.

It was easy to pick on the USPS when people could only compare them to UPS or FedEx. But these days complaints about the USPS fall flat as its more obvious how much value they provide. I'll happily keep shoveling junkmail into the recycle bin if that's the cost of Priority Mail.

I was referring to the fact that you might get arrested for taking photos of "critical infrastructure" like bridges and we have federal agents and laws to make sure the mail is safe and available to people in the most remote places. Internet? Mess it up all you want!

What do you suggest happen? We levy criminal charges against them because somebody/something severed important fiber connections?

When a road is closed does the government pay us a fee? Do we have SLAs for roads? When a letter is lost in the mail, does the postal service refund the stamp? Your comparison is illogical.

Business internet accounts can and often do have SLAs. But, should consumers pay more for SLAs like businesses do? It’s a question of cost.

In France, I had a four hour outage on a business connection once; no refund, not lawsuits from the government. I have also had at least five or six outages that lasted 15 minutes or so. Expecting 5 nines uptime is just unrealistic unless prices rise. My gigabit Comcast connection costs me $101 per month, including taxes. In France, I paid €79 for a “business” connection that had 10mbs down and 0.8mbs up — with an 8 hour SLAS. I am getting vastly better service from a consumer Comcast connection than I did from a business Orange France connection. And France is heavily regulated. My point is to be careful what you wish for.

Why months of free service? It’s easy to just calculate the lost hours during the outage and compensate for that with maybe a multiple of 2x, or 5x or even 10x on the monthly bill. Multiple months of free service for an outage of a few hours just sounds like consumer greed.

This is one of those cases where I might agree if these companies weren’t actively trying to monopolize, failing to invest in their services, or trying to charge hand over fist in every possible way. Since ISPs are generally terrible though, they absolutely deserve to pay through the nose for every failure. It has to be punitively bad for them to fail, or they have proven that they just don’t care.

And to take a parallel into the business world, usually you'd pay extra for lines with higher SLAs, and I bet most consumers wouldn't bother.

It's not really possible for a single ISP to get to the level of quality that has no hours at a time outages. For minutes per year outages level quality you need at least two ISPs independent from each other and not sharing infrastructure. Which means they won't even be available to consumers in the US and other countries, where internet access market is monopolized. I mean I can't imagine a law mandating ISPs to effectively split into two companies with each providing an independent link to the premises and all that. But when there is a competition a consumer at least has an option to just pay for multiple links to multiple ISPs.

They are in the business of Internet service, whatever that means. If there are things legitimately beyond their control, then pricing and/or claims of service must reflect that.

Instead, we see many issues of their own making (monopolies and refusal to invest, mainly) that make it hard to sympathize.

They actually do give a prorated rebate when outages like this happen. At least they did for me.

A fair amount of the money is wasted on adverting. Mailers, TV adverts, etc. Sure it's bought in bulk but it eventually adds up. I can't imagine what they pay to acquire a new customer, or worse, get a disgruntled customer to come back.

Verizon? Same shit different name.

Even where there's competition, they don't try to compete. I presume it's mainly a ruse to avoid any further regulations, etc.

I don’t understand this trope that ISPs don’t invest in network upgrades. A decade ago you 25 mbps was the fastest Comcast offered. Today, Comcast is rolling out gigabit around the country. What else has gotten 40x faster/better over the last decade? Even your iPhone CPU has gotten maybe 10x faster in that time. Maybe the GPU has improved that much.

Don't forget all those folks without home phone service and unable to make emergency calls.

The same thing happens when telephone service goes out because of a down power line. Nothing new here.

Landline phones generally work during a power outage in the United States.

As carriers change to fiber for all or part of their network, this is increasingly not true.

It does bring back fond memories of trying to explain to WoW guildmates that we needed to hurry up and finish the fight, because my power just went out. (Still online thanks to telco central-office based DSL and a good UPS.)

Are they using Comcast internet for their phone or do they likely have a cell phone?

> They'll probably raise prices of anything

> There ought to be strict laws around long service outages, resulting in automatic multiple months of free service, etc. as a deterrent for allowing things to just collapse for hours at a time.

FYI, that means even higher prices...

What’s that saying? Poor Pornhub planning on your part does not constitute an emergency on my part. Actually they do give rebates, unlike the road system. When’s the last time you got money back or a tax rebate from a massive road closure?

When's the last time you had literally only one route available to drive somewhere and it was closed?

When your ISP goes down, you can't just switch to another one on the fly. Many people can't switch to another one at all in the first place. When there's a road closure, you just go around.

December 2017. California "Thomas Fire" shutdown 101 (and trains) for weeks. Taking the 5 inland sucks.

If you need the Bay Farm Island bridge tonight, you are SOL. I guess you could charter a ferry boat.

https://alamedaca.gov/alerts

You can use tether to your phone data plan in a pinch. Enough for email and light browsing for sure.

How could they tell?

I know that other commenters are being flippant, but could this legitimately be them testing their throttling ability for specific types of traffic?

No.

Most likely not. Probably more like a backbone routing issue.

Their rate limiting tools like Sandvine do not causing these kinds of outages. They cause hassle for specific protocols and destinations. (rate limits, tcp resets, blackhole) but not massive outages.

Attributing a nationwide outage to a "feature test" seems like it's more flippant.

Reply on news.ycombinator.com