911 emergency services go down across the US after CenturyLink outage | Hacker News Reader

911 emergency services go down across the US after CenturyLink outage

techcrunch.com

434 pointsLopRabbit213 comments

I'm unable to read the article on my phone (ad?) so I apologize if it is answered there, but why is this a long lasting problem? My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order. (I don't want to ignore the pre- and intra- effort by ops, I'm just saying it paid off)

Is this outage just breaking those places that didn't have such things (as a result of insufficient budget or skill)? Is there some now-regrettable contract(s) that prevented that from being in place?

Why is a system that saves lives working without the backup that systems that dont dictate life are? While I understand that govt has both budget and hiring issues, i've worked in state govt and there are definitely some highly competant admins to be found, so I wouldn't assume that is the problem without word from someone with more detailed knowledge into this particular problem.

Software is hard, and backbones tend to be mono-cultures. This means that there will be bugs which only become apparent when the full fleet has been deployed.

This is why large phone networks have really long rollout periods - to hopefully catch these things before it affects everything.

This is also why emergency services should not be using cell phones for reliable service during emergencies.

Here's another example from 1990, where the phone network got stuck in a reset loop.

https://catless.ncl.ac.uk/Risks/9/62#subj2

Corporations make similar mistakes with their disaster planning. It doesn't matter if you purchase network connections from two suppliers if they go across the same bridge.

My dad spent his career as a natural gas pipeline technician. For a long time they maintained their own radio towers and VHF or UHF (I forget which) radio network. Each district had their own comms guy who maintained the network, and it was very resilient and reliable, allowing the entire company to be in contact essentially anywhere.

In the early 2000s as a cost cutting measure, the company decided to nix the radios for satellite phones on each truck. Of course this proved to be problematic, as the sat phones often had reception issues. They relied on cell phones as backups, which was also quite foolish as the remote compressor stations had very poor reception. They also had issues where someone would leave a voicemail and they wouldn’t get notified of it for days or weeks due to some issue with AT&T.

After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies and dismantled the infrastructure. My dad retired not long after this, but the “corporate bean counter” trope rang quite true here, and in the long run we were all a little less safe because some executive with no field experience wanted to make a name for themself by saving a little money on something that proved to be mission critical.

That makes me sad to hear, because not only was this company left with an inferior solution, the man hours and effort that went into the original project (including purchasing part of the spectrum!) was essentially nullified.

It's an alarming trend among new grads as well, who see existing systems as bloated and in need of rewrites. Who would've thought the bloat was kind of important?

But then we wouldn't have single threaded wonders like node.js

It's part of our culture that the young kids dismantle the 'old people stuff' setup their new stuff to only re-invent the wheel again and again, run into the same problems that were solved decades ago to only be repeated again.

Remember how TCL was going to save us all from K&R C? Then it was perl to save us from TCL? Then Ruby? Python? javascript, Go or whatever is the flavour of the month?

Sure except you've managed to bitch about most major programming languages here without proposing any alternatives.

The poster's entire point was that all of them came from a desire to find another option when a reasonable alternative already existed.

The logical conclusion is that Ruby shouldn’t exist because shakes fist those darn kids should write all programs in C instead.

I wouldn't argue that bloat is important. I wold argue don't let bean counters make technical decisions.

Unfortunately those bean counters probably got large raises and bonus. Then moved on to their next job proclaiming their victories and got out into higher positions.

And the board let the financial wizard violate the old principle [1] of not taking a fence down until you fully understand the reason why it was put it up in the first place (i.e., don't assume that your predecessors were stupid just because you don't yet understand why they did what they did.)

Really sad results there, throwing away a sound investment & body of know how, for worse results everywhere, and less safety for everyone.

[1] https://www.chesterton.org/taking-a-fence-down/

BNSF railway still maintains its own point-to-point microwave network and VHF/UHF repeaters on mountaintop tower sites, if you pull the public FCC ULS database data for any western US state you'll see their 6 and 11 GHz band licenses.

> After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies

This is one of those cases where, if it isn't broke dont fix it was a rule to be followed.

Now its hard for the company to fallback to the old system, since they need to buy another set of frequencies, before they can even plan to rebuild the radio tower network.

Since they had stopped with the good running radio network to cut costs (unwise since it ran fantastic) the chance that they will invest in restarting such a network is not really big, unless they get heavy lossses due to their unreliable current system of modern (but worthless) gadgetry

Seems like the solution is to improve cellular reception along the track of the pipeline and stations.

Great write-up, thanks for the link. TLDR: Race condition when switches reboot during high (read normal) traffic load.

I love the fact the "fix" was to reduce traffic load so that the issue was unlikely to occur and let things go back to normal.

> AT&T solved the problem by reducing the messaging load of the CCS7 network. That allowed the switches to rest themselves and the network to stabilize.

Race conditions strike again. It's just hard, whether it's multi threading or distributed systems.

https://en.wikipedia.org/wiki/Therac-25

We used a vendor to put up two MPLS drops to every office. different providers, different routes from the building, etc. Turns out, further up the chain, an ISP was buying transit from another ISP, and they all went on the same shared fiber that a nice backhoe operator found for us, about 800 miles away.

This is very common because of the way transport is built and sold.

Doing real traffic engineering where shared risk link groups do not exist for the active and standby links is a lot of work and, moreover, is hard to keep working as operators change their networks (for maintenance, for other reasons).

On top of the basic difficulty, there are other issues - layering makes this opaque, the service providers have become completely de-skilled, the business model for MPLS is getting killed by Over-The-Top providers who pretend they can give you equivalent service using multiple links and IPSEC tunnels. And so on.

Trace route can help with that somewhat, I've noticed that a few times with redundant connections

The problem with traceroute is that it won’t show that two different ISPs are leasing strands out of the same bundle.

All of the unexpected shared infra issues I’ve observed have been down at this lower level that never shows up at the IP layer.

Is there a directory or service one can use to get that information?

In my experience this exists for ISPs, but it's not easily available. I've used ISP brokers in the past and they've had this information in some proprietary systems when I've asked for quotes and proposals.

My experience corroborates this.

I worked at two small ISPs, each with a fiber network spanning a few US states. They had their own databases to track this information, but as far as I know there was no central database. One of them used OSPInsight backed by a self-hosted SQL Server database. I don't know what the other place used.

In Australia at least, this is becoming harder as time goes on. Several good alternatives have been purchased by one conglomerate. I know several businesses that did everything right in buying two or even three sets of connectivity from different suppliers, and now what they have are different trading names for the same company providing a service on the same network.

My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order.

Because your workplaces don't rely on funding from taxpayers.

CenturyLink isn’t exactly an underfunded provider though.

> Because your workplaces don't rely on funding from taxpayers.

Right. Their workplaces probably had tighter and more transparent financial constraints with more oversight, more accountability for failure, and more incentive to do it right.

Have you worked in a corporate environment?

On a more serious note, I haven't seen any convincing evidence that your genetalizations hold true with reapect to public vs private organizations.

> Have you worked in a corporate environment?

I have, and been a consumer of them, and have quit them easily.

> I haven't seen any convincing evidence [...]

I guess I could link a page of companies dissolving due to financial mismanagement vs governments or something. The point is not that only one sector suffers mismanagement, it's how that mismanagement is realized/handled/reacted-to, and of course it doesn't apply to every situation. I just get tired of hearing assumptions like a public service is improperly implemented due to funding, especially in this case where I know many of the municipalities are well endowed and should have the redundancy of their systems questioned. Why not ask for convincing evidence for those statements instead?

>have quit them easily.

???

>Why not ask for convincing evidence for those statements instead

Because I'm not replying to someone making such a statement.

> ???

Meaning in many cases adverse effects of private sector failure are easier to divorce than public.

> Because I'm not replying to someone making such a statement.

Right, I was, hence the thread context. The point was wondering why replies aren't there asking, so of course the reply isn't there asking.

>Meaning in many cases adverse effects of private sector failure are easier to divorce than public.

Government doesn't tend to provide "easy to divorce services", it's a selection bias problem, not something inherent to public vs privately ran organizations.

Of course it's easier to change office supply vendors than it is to change water companies. Utilities tend to be government operated (or heavily regulated) because their nature makes a competitive market difficult.

>Right, I was, hence the thread context.

Why did you agree with the person you were replying to then? You could have asked for proof in your reply, instead of affirming the comment?

Because they don't test failover.

From CenturyLink's Twitter feed:

"Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership."

This info is related to their much earlier DWDM/transport system outage, which took down a huge number of 10Gbps lit transport circuits starting around 0700-0800 Pacific time on the 27th. The 911 outage for WA began 10+ hours later.

see outages mailing list archives for more details:

https://puck.nether.net/mailman/listinfo/outages

This effected Washington, and its still very much on going 24hrs later. Geez is it a pain. https://downdetector.com/status/centurylink/map/

Thurston County came back online about 20 minutes ago (0900 PST).

Weirdly that almost sounds like some switch getting infected by a worm. eg then attempting to infect any other gear it can (thus the broadcast traffic). :(

[turns off Spanning Tree Protocol] "What does this switch do?"

There was an incident with similar consequences on April 10, 2014. The cause was a programmed threshold being breached and the impact was 6h of downtime.

Source: "The Coming Software Apocalypse" published by The Atlantic (https://www.theatlantic.com/technology/archive/2017/09/savin...)

walrus01 also linked to https://www.fcc.gov/document/april-2014-multistate-911-outag... in another comment.

> Operated by a systems provider named Intrado, the server kept a running counter of how many calls it had routed to 911 dispatchers around the country. Intrado programmers had set a threshold for how high the counter could go. They picked a number in the millions.

I'm really curious if there's some explanation that makes this sound less catastrophically stupid, particularly the part where they picked a threshold less than INT_MAX.

What do you mean, VARCHAR(8) is a perfectly valid way of storing counters...

much better to future proof it by using a BLOB. You know, just in case you ever need to shove something else in there.

if you're going to use a BLOB how can you know the type and how to decode it? Make sure to serialize that data with something like protobuf.

Nah, describe it using some custom binary format and have a 1000 page spec to document it!

Yay, seralised bluetooth! ;)

Allocating pieces of a resource without accounting for reusing released resources?

Great question. Maybe they did pick a sensible constant, but it still wasn't enough. I.e. INT_MAX wasn't big enough.

This will be a post mortem to read. No details yet but I’m wondering where the broken links are? Is it a VOIP glitch or more than that?

Edit: VOIP indeed. Centurylink outage: https://komonews.com/news/local/centurylink-outage-knocks-ou...

Today I’ve learned that vast swathes of the emergency call network depend on just one ISP. I wonder if the old, analogue and copper-only network wasn’t more resilient.

It was. In the entire history of electromechanical switching in the Bell System, no central office was down for more than 30 minutes for any reason other than a natural disaster or major fire. There are books about Number Five Crossbar and how it worked, and they're worth reading if you design high-reliability systems.

The only central office downtime from a major fire was in 1975 in New York City, resulting in 23 days of downtime.[1] All the incoming cables and the main distributing frame had to be replaced. The switching equipment, on upper floors, survived and just needed some maintenance and cleaning. That was still a crossbar office; it hadn't been converted to electronic switching yet. Worst disaster in the history of the Bell System. Bell switched to less flammable cable coverings, like plenum cable, after that.

Widespread failure of 911 service suggests an overcentralized architecture. 911 requires a phone number to address lookup, so there's a database involved. Widespread failure indicates this was implemented as a remote query service ("in the cloud") rather than read-only database copies of the directory at each central office.

[1] https://youtu.be/f_AWAmGi-g8?t=110

There's an old AT&T video showing the careful coordination and huge amount of manpower needed to upgrade a single telco switch. They had rows of workers with cable cutters, in order to literally cut the entire service over to the new equipment. Total downtime was just 47 seconds.

(And even during the cutover, they're careful not to interrupt any emergency calls.)

https://www.youtube.com/watch?v=saRir95iIWk

I love the "DON'T TOUCH THIS" post it at 3:55

That was absolutely amazing. Thanks for posting.

So, what exactly happened there?

Where were the signals going after the cables have been cut?

The new switch was connected to each line in parallel with the old switch, but it was able to ignore incoming signals until told to take over. The old switch was hard wired enough that it could not be remotely commanded to have no effect on incoming signals. The line relay attached to each line had to be physically cut out of circuit.

That is spectacular - thanks for the link!

Remember when that guy cut the fiber along 280 in about 2009 or something resulting in major outages.

And then again in 2015.

Also, there was that kid who put together areport of all infra cable runs for a thesis and the FBI confiscated it... https://www.google.com/amp/s/www.nbcbayarea.com/news/local/F...

I'd love to read more about the crossbar and switching systems, can you recommend a specific book?

"A History of Engineering & Science in the Bell System: Communications Sciences, 1925-1980".

Here's probably the simplest introduction.[1] When reading this, note phrases like "ORIGINATING REGISTER SEIZES AN IDLE MARKER". Think of that as "originating register asks for an idle marker from the pool of markers". Very little equipment is dedicated to specific lines. Everything is done by requesting a service from one of several identical units. If one of those units fails, the system capacity is reduced, but the switch does not go down as long as at least one of each unit type is still up.

The switch fabric, the actual crossbars, is dumb. It just makes the connections it's told to make.

Shared resources include:

- Originating registers. These provide dial tone and record dialed digits. They parse the incoming number to the limited extent needed to decide when it's finished.

- Markers. The smart part of the system. When an originating register has a full set of digits, it finds an idle marker and sends it the call info. The marker figures out what to do next, in about half a second, and then it's free for another call. Markers tell the switch fabric what connections to make. They're duplicated, and the two halves check each other. If the halves disagree, the marking aborts. If a marker aborts, the originating register tries again with another marker. One retry only. Marker failures also cause data to be sent to a "trouble recorder". As usual, there's more than one of those, and they're "seized" as needed.

- Senders. These send digits from one exchange to the next. They're primitive modems.

- Trunks. Lines between exchanges. Full duplex, four wires.

- Terminating senders. The receive side of senders.

There are also units associated with accounting, coin telephones, routing tables, and other auxiliary functions.

The key takeway here is that there's no single point of failure.

[1] http://wedophones.com/TheBellSystem/pdf/no5crossbar.pdf

This involves cell phone access... which is a little more complicated than a billing address lookup.

> There are books about Number Five Crossbar and how it worked

Can you recommend such a book?

/edit: i see you answered this for another poster

20m10s is definitely my favorite part for humor purposes.

Thank you for sharing, that was an enjoyable watch!

In Kitsap County (just across the water from Seattle) 911 traffic is delivered by Centurylink via SIP over a Wave Cable connection (think Docsis 3.0, not fiber). Why the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink.

Why the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink.

I've consulted with a few rural counties (I don't know a thing about the PNW, is Kitsap a rural or suburban county?) who got their circuits leased to them through a local MSP/IT Shop who had all the best intentions, but overlooked any sort of failover and you end up with an answering center taking calls over a single adtran going direct out to a CLEC.

Just wondering if this is less county refusing to do something, and more like the county implementing services sold to them that probably could have benefited from better planning and integration.

Probably the latter. How many rural county boards have people who actually understand any of this stuff?

It seems more than likely that Centurylink Sales/Marketing convinced a few counties to use them and then sold itself to other counties as "the" provider for such services.

How many rural county boards have people who actually understand any of this stuff?

Not enough, I would wager. It's something I spend a lot of time thinking about, living in one of the largest cities in the country now, with parents who live comfortably in the rural south-but have complained about the internet a couple of times.

To their credit, they'll lose connectivity for hours and not even mind, since mom will just turn on the radio and listen to the local PBS station and tend to her garden while dad goes out to his woodshop.

Others in their community probably aren't as content without connectivity, for such long periods as they are.

If gov at that level can't afford to consult, decisions on that infrastructure should move up a level.

Yes, it does seem rather redundant to have each county responsible for rolling their own solution (even if that's just deciding how to outsource it)

In NW WA you might be surprised. There's a long history of utility self sufficiency with the San Juan islands, and the area and neighboring counties has a lot of dark fiber available for lease from smaller independent providers. Many municipalities and community organizations light up their own WAN links.

There was a always a tension when it comes to outsourcing and considering core competencies in the orgs I saw. Can imagine less appetite for creative solutions when critical things like 911 are involved.

A leader's job is find experts and delegate. Not knowing isn't an excuse, finding people that do know IS their job.

And what amazes me is that the ISP was content to be a single point of failure for emergency services. I'm no expert, but I'd think a primary design criterion for emergency systems is that they keep working in emergencies.

My friend and I were just talking about how they should put a bit of oil/natural gas at the top of all those little 911 repeater/tower/whatever things that are sprinkled throughout the city to turn them into a sort of "Beacons of Gondor" type thing, as an absolute-last-resort communication method. Mostly jokingly.

Well... if you put the oil in a small emergency generator it might be more practical, if less romantic.

It’s not the ISP’s responsibility to ensure the customers’ have adequate HA designs for their own systems.

No, but if you are providing voice and telephony services that connect to the public switching network, as CenturyLink does, you absolutely have certain reliability requirements for 911 calls being relayed to local answering points.

I await the results of all the inquiries into this to find out what took place here.

Edit: Oh and "But $ISP doesn't offer traditional phone service, they sell VoIP", is as of 2005 no longer a valid excuse for non-compliance: https://www.fcc.gov/document/commission-requires-interconnec...

This isn't an issue with servicing their direct clients to E911 services, it's cell phone access from various carriers that went out. It's still a bad thing, but not quite the same as what it looks like you're assuming.

No assumptions being made here, just stating matters of fact as to what requirements CTL has for transmitting 911 service, the E911 edit was implied to be supplemental.

CenturyLink owns large swathes of telecoms infrastructure nationwide, and Level 3, a tfn/did resporg provider

CenturyLink is an incumbent local exchange carrier, as it merged with Qwest, which merged with US West, formed in the 1984 AT&T breakup.

I wouldn't be surprised to find that most emergency call networks rely on the local ILEC, although I would have expected routing to be less centralized.

Actually, CenturyLink has sold their services to a very long list of municipalities which then place CL between the ILEC and emergency services.

the decentralized nature of the e911 system is what made it fault tolerant. moving these systems to a more centralized topology is probably going to cost them alot of money in downtime and fines from the FCC in the long run...

CenturyLink operates that, too.

It was more resilient, in a way.

It had more points of failure, instead of one big point of failure. It was also less available nationwide due to the expense.

If someone dies due to inability to summon emergency assistance, this will truly be a post-mortem instead of a badly named incident retrospective.

CenturyLink is critical to 9-1-1 services throughout the U.S. They have a 9-1-1 division to which a very large number of municipalities outsource. Even military installations outsource parts or all of their 9-1-1 operations to CenturyLink.

This CenturyLink division has historically acted very carelessly in regards to securing these systems. Getting them to take ownership of their contractual obligations can be difficult.

Criticism of their operations is very sensitive. Pointing out any vulnerabilities can be prosecuted as an act of terrorism. A post as simple as this one comes with significant risk.

While I'm not refuting any of your claims, do you have a source for them? It sound like typical big corp behavior, albeit worse because of the importance of emergency services.

I at least found the 911 unit:

https://www.centurylink.com/wholesale/pcat/911.html

To the best of my understanding, they aren’t providing the 911 call center here.

Interestingly the University of Washington put out an email notice nearly an hour before AlertSeattle sent one. You'd think the city would make it a higher priority to let everyone know that 911 is down. Not sure if they did a text message earlier though, currently overseas w/o service.

https://i.imgur.com/2rlxi7f.png

More info on the outage here: https://www.king5.com/article/news/local/nationwide-centuryl...

It may just be that their email service is not particularly optimized for such an urgent case, and so only got to your email address one hour after the person sent the blast. I've seen CRMs take hours to push out a 40k email blast.

How long do you think it takes to send out a few million alerts versus sending a few thousand?

For an emergency response system, I hope the answer is "about the same time". Or, more precisely, "within the allowed latency window for emergency alerts".

UW's email alert list probably tops 100k. They have like 70k students + staff and there are plenty of former students/staff and associated individuals who are on the lists.

In any case it does seem like the tools that have been built for email marketing are a lot faster and maybe even more reliable than conventional text messaging.

Isn't there an emergency alert system -- should take as long as sending a radio signal?

EAS notification would have been virtually instant. But it's supposed to be used only for immediate life-and-death situations (tornadoes, nuclear war, chemical plant explosions).

Amber/cell phone alerts have a lower threshold for pulling the trigger.

I think Seattle eventually activated the local EAS over the radio during the 911 outage, my parents mentioned hearing it, though it was very scratchy and not very intelligible.

There was a reverse 911 alert around here about the outage, loud alarm and everything.

> You'd think the city would make it a higher priority to let everyone know that 911 is down.

Should it be? There is an argument that 911 serves as a deterrent and that it would be better if people (in particular, potential criminals, but we can't isolate that portion of the population) didn't find out it was down until it was working again.

911 is used for a lot more than reporting crimes, e.g. fires, accidents, and sudden serious health problems.

So 911 keeps "The Purge" at bay? I've never heard this argument.

It's relatively well known that any situation that produces a predictable breakdown of police response is expired by opportunistic criminals (looting during disasters is perhaps the best known manifestation.)

A publicized 911 outage would be an obvious opportunity.

I'm not sure this argument really holds up given the dispatch phone number for every police department in the country is still fully functional.

Now, if you DDoSed all the local dispatch lines for every police station and sheriff's department in your county, that might be another story...

Most people dont know it. At the least, 911 being down will add delay to the response. Those minutes might be all someone needs for a quick theft, esp cars or burglary focusing ln high-value items.

Is that really an argument anyone would want to make?

And this is why I've made it a habit to save the the local number to the police station where I live, and after moving around a little I seem to have accumulated a few. Always nice to have a backup should you need it.

Not just the police department... Fire, ambulance, poison control, dig safely, local traffic control tower (they're about ~9 miles away from me), pharmacy, and suicide prevention (every fridge needs one of those magnets).

Seems like the government could very cheaply offer an app that is basically just a geolocated lookup for these sorts of things. The app could basically just be a spatialite database generated from a git repo, and it would only have to be updated periodically, and would never go down.

They should but I think it would be a great community project. I'd be interested in helping out, been working on apps for ~3 years.

Smart. Noting these. Why traffic control tower?

Maybe to call if there's a plane in trouble?

Some airports also have public phone numbers to get weather information, could come in handy to know if you're-for example, living in tornado alley and a system rolls through that knocks out power and cellular service.

If you have a working landline phone, calling a nearby airport could be your only way to know the latest conditions and if it's safe to go outside.

My elderly father is living with my sister's family. To find out if it's safe to venture outside they tie a rope to him and push him out the door.

I've always wondered why there isn't a local version of 911. Like 811 connects you to the local police department. Because no one knows the local number. Besides, what if you are traveling? That would also free up the load on 911 operators.

311 is increasingly used for local services, but coverage is spotty and usage is inconsistent. In Indianapolis, last I heard, it was only for police use, while in some other cities it’s a general government point of contact.

What do you mean? 911 does go to your local dispatcher.

I mean to the local police for non-emergency events. 911 is supposed to be an emergency line. But people call it because they just know that number.

In the UK, ringing 101 gets you through to your local police dept's non-emergency number, and 111 gets you to NHS 24, the national non-emergency medical advice service. Both of those are fairly recent, though - NHS 24 was launched in 2001, and 101 in 2009.

That's kinda what I'm asking for. America really only has 911.

I think the UK definitely benefits from being much much smaller, and the emergency services much more tightly integrated, so rolling out new numbers is not as huge an endeavour as it would be in the States.

> Ajit Pai, chairman of the Federal Communications Commission, which regulates and monitors 911 services, said the commission is investigating the outage.

Until they "resolve" the issue without disclosing any details...

Maybe they will open the case for public comments and insights, then “ignore” those comments because of a “DDOS attack”, and then finally not investigate the alleged attack despite earlier on outlining how it was able to cripple public infrastructure.

Could there be a more mismanaged agency than the FCC?

>Could there be a more mismanaged agency than the FCC?

or maybe they're all just as mismangaged, but you noticed them in partiular becaue you are passionate about the FCC's mission?

I'd be willing to bet if the average American interacted with fincen, FDA, IRS etc they'd be shocked at the ineptitude. America has been strangling their enforcement agencies for a long time - one could argue there's a direct path from a poorly funded IRS only taking on "slam dunk" cases and ignoring complex money laundering schemes to Trump's election.

The fact that it takes a special prosecutor (and the subsequent reporting) to dig into these big money laundering schemes only cements your point even more.

If someone neglects to provide their children with food and water, it's rather inappropriate to get mad at the children for their lack of capacity to provide manual labor.

Ajit Pai will be so hard on them that Century Link might offer him a job after the government "service."

911 service is VITAL, pay die from heart attacks when ambulances are not sent and people are killed by psychos when cops are not notified.

Make it 100% redundant, all the way. No one single provider should have all of this in one basket.

Could this happen to have been part of the massive CenturyLink/Verizon outage that's been rolling across the midwest the last day or 2? I know it's been a major pain in the butt for me, but I've been super productive at work not having to worry about answering calls.

network engineer for pacific NW ISP here: Yes this is absolutely centurylink related, from what I can observe that it's taken down, and I have my suspicions about Intrado. I think centurylink broke things further trying to fix this morning's DWDM/transport system outage.

See the PDF file linked here:

https://www.fcc.gov/document/april-2014-multistate-911-outag...

Was this 10G DWDM gear? Any idea who’s equipment and what root cause was? I’m just curious because I used to work for a vendor that supplied DWDM gear to Centurylink and wonder if this is equipment I designed. 10G DWDM is pretty robust so must have been a pretty massive failure, line amplifier, fiber cut, software upgrade, or something.

nothing confirmed yet, it's all rumors and conjecture.

https://fuckingcenturylink.com/

I know it's not the same orgs, but Centurylink has had the most incompetent people I've dealt with as a retail customer.

My totally subjective opinion: In US/Canadian telecom, any business formed from legacy near-monopoly ILEC (incumebent local exchange carrier) staff, they neither have a clue nor give a fuck. It's a corporate cultural attitude left over from days when ILECs truly were the last-mile monopoly carriers in many cities.

In this case Centurylink is composed of former Embarq, Qwest (USWest) and other ILEC entities.

My opinion also applies to entities like Bell, Telus, Verizon and Frontier.

If you look at the people who run the core routers of their ASes, sure they know what they're doing, but they're so insulated from actually talking to a customer through myriad layers of corporate bureaucracy, that their clue level might as well be irrelevant.

You left off most important part: corruption. Paying politicians to block regulations and lots of competition, esp taxpayer-funded, lets them get away with this stuff. Even more so when one of their own is the regulator.

Qwest was a much better company than Centurylink. All of my conversations with Centurylink quickly turn into some Malkovichian bizzaro world.

Appreciate the link, that was a very informative document (didn't finish all 40+ pages in the ~2 minutes to post this comment, of course).

At least the systems to notify of the outage sure worked well, at least for me.

Got an EAS alert while watching TV and two emergency alerts on my cell phone (one generic telling me to call local police or fire, and one telling me a specific number to call for my county).

Furthermore, I now think that a call that came in on my landline at 10:42 PM was probably also a notification. The caller ID just said "wireless caller" and I didn't recognize the number so I ignored it, but now Googling that number turns it up as a cell contact number for my county's Department of Emergency Management.

The incompetence of the Roman government once became so great that they lost the ability to provide basic public utilities like the keeping of time. Is this that for us? How could 911 possibly go down? Crazy.

911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate. There are thousands of (poorly run, uncoordinated) Public Safety Answering Points across the country, and most tend to contract with just the local incumbent telecom, ensuring any time there is an issue with said incumbent, every call to said PSAP fails.

In the context of most of these PSAPs having already moved to VOIP, there isn't a coherent reason why these calls shouldn't go straight from the carrier originating the call to the PSAP, without middlemen like Centurylink standing by to misroute those calls.

I mean if you want to get into the history of any technology you can point out how archaic and horrible it was initially. We're still in that phase with the internet, in 50 years people are going to read about things TCP/IP and Facebook and Comcast and won't be able to stop laughing hysterically at how bad and flimsy it all was.

> 911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate.

I'm always interested in Chicago history, so I went looking, but couldn't find anything at, for example, https://www.nena.org/page/911overviewfacts . Do you know anywhere this history is written up?

Ah, I may have seen what you mean. According to https://www.countyofunion.org/site/cpage.asp?cpage_id=180009... :

> 1976 Chicago claims to have had "the first enhanced 911 system of any major city" in the United States.

… so I guess in

> > 911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate.

you meant E911 (which, as the linked page points out, is a nebulously defined and variously implemented technology), not necessarily 911?

I'm surprised cell towers can't tell connected phones what the local emergency number is, and then dial that instead when 911 isn't working correctly.

Cell networks (but not towers) effectively do this already - when you dial 911 (or 112), the phone doesn't actually dial the number "911" or "112"; instead it requests a call of type emergency. This is routed by the mobile network itself, according to how it handles emergency calling.

The phone also enters a specific "access class" (access class 10) for emergency calling mode, which lets it join other masts on networks it wouldn't normally operate on. (This is where the message "emergency calls only" originates from)

But if the onward system this "un-numbered" emergency call would be passed onto is unavailable (as it sounds like is the case here) then emergency calls will fail. It might be possible to dynamically reconfigure the mobile network core , but you'd need to configure this on the core, for each and every cell tower or group of cell towers, to route the call to the right alternative lines/number. Whether that system has capacity to handle all these calls is another question.

> for each and every cell tower or group of cell towers, to route the call to the right alternative lines/number

Cell towers have virtually nothing to do with routing calls. That’s all centralized in a region with all the towers back hauling.

Cell towers likely don’t run the actual cellular service; they’re just rf points that run the phone applications on top. And if a provider has a fleet of super duper towers that could do it, there’s a certain operations job to be done that needs to be perfectly done. Not trivial. Doable, perhaps, but that would have to be a well rehearsed job for any hope of consistent fleet wide success.

That sounds like a MITM attack waiting to happen.

Isn't that exactly what cops or others do when they use a Stingray phone tracker and intercept people's mobile phone connections?

I still remember in late 2013/early 2014 when I lived in Kyiv, Ukraine, and my friends at the Maidan protest against the government said everybody within geographic proximity of the protest received SMS messages from an unlisted number telling them their protest wasn't 'legal'. Everybody assumed it was the Ukrainian government security forces using a Stingray or similar device as a MITM between their devices and the network.

If the government is in control of the network or can can order / request the network to do so it would be fairly trivial to send a message to anyone connected to a cell tower.

The thing is if the tower is serving a lot of people you could end up sending the message to people not involved so you could use a IMSI catcher (you can even make them out of a TV Tuner turned SDR receiver[0]) to catch IMSI's in a smaller area or use triangulation to narrow down the pool of devices in an area and then use that list to send SMS from the network itself.

The UK did trails of using location based text messages back in 2013 in order to be used for Public emergency alerts [1]

When you can gain access to the network their is no real need to MITM it, but yeah I believe a sting ray type device could also do the same as they act as a tower and trick your phone into connecting to it.

[0] https://github.com/Oros42/IMSI-catcher

[1] https://www.gov.uk/government/publications/mobile-alerting-t... (link to the info about the trial and the final report)

From my experience for a lot of these 911 centers they are centered around counties(not all, but a lot). Back when they were first being set up with the internet capacity there were two issues. DSL was the only available service within the area, and the centers couldn’t afford large enough pipes. Talking early 2000s here. So a lot of these centers got pretty large grants from the feds.

They then hired one of the companies at the time that were doing these upgrades in bulk around the country when the money was flying because the fcc wanted all of the emergency services to move to narrow band from wide band.

These companies came in updated everything to modern and setup the contracts with the DSL providers back in the day(in my rural town I grew up in they just got 80mb cable internet in 2016, prior to that 10mb dsl was the best you could get there) In the west that was Qwest who became Century Link.

But government being what it is. A lot of these 911 centers have languished because there is no desire to invest in updating these systems as they are expensive and hve a difficult upgrade process. So a lot of these centers are just paying to bills and trying to scrape by with minimal funding. And so upgrades and redundancy aren’t even considered. Until something like this happens. And the feds might throw some money at the problem. All the centers will get grants to have failover and then will go back into dormancy of just working day to day with no budget.

Mass says they restored 911 by 9am, but I just got an alert at 12 noon saying it's down now...

I was in a grocery store when my cellphone, in unison with all the other phones in the store gave the clarion call for attention... I thought Russia had finally launched a 1st strike...

It wasn't just emergency calls. Most Centurylink service around the US was down intermittently yesterday.

A lot of Centurylink circuits are still down in Arizona, and if not down completely, have significant packet loss.

A friend texted me that his POS systems that rely on CenturyLink in Nevada have been down for two days. The stores can only do transactions in cash.

Makes you think twice about the cashless utopia the HN crowd imagines.

Makes me think we should be investing in redundancies (one of which is cash, but also in our electronic networks).

Or to switch to crypto currency that is distributed and eventually consistant

I swear the swiftness with which people swoop into these kinds of threads to suggest crypto/blockchain is bordering on becoming a meme.

If the network is fully down, I don't see how the scenario for this business owner changes with blockchain, they'd have the exact same problem with a register that can't close an ACH: backlogged transactions that need to catch up eventually, meaning some card types are going to decline if the POS can't reach a clearing house utilized by the card vendor.

Even if you wait for 15 confirmations, that's easily fufilled by the others in a market.

Similarly, for it to make it to the chain only 1 of those 15 confirmations has to connect to any larger network sometime in the future.

Can confirm, we've had major issues with calls and call quality across our offices the past couple of days and I have been getting periodic updates from Centurylink about their status in regards to our circuits in my email. Major clients have been effected by this.

Central Kansas reporting. For a short time (hours/minutes, wasn't home, got a text from Dad) the normally 6Mbps CenturyLink DSL was down to 0.14 Mbps.

Why does a congressman not step up and draft a law stipulating:

    * US telecom, an office of US federal government shall
      be founded and it shall serve all federal and state 
      governments' institutions. US telecom shall be funded
      by federal sales tax. Government institutions
      themselves shall not be billed.
    * Emergency services number shall be 911.
    * 911 shall be operated by US telecom and US telecom
      shall pay private communication providers in
      international rate every time 911 is called.
    * Private providers shall not bill
      911 calls to their subscribers.

Is it me or did TechCrunch recently ruin their mobile experience. I tried to read the article but got so frustrated and distracted by the wonky scrolling I gave up. It’s just awful.

Is there any information available about how these systems are typically architected and run as well as what sort of availability they generally achieve?

I can tell you what 911 Call Taking systems look like, they are mostly a call presentation system, with some call handling features. This issue is upstream of any 911 center however.

Nothing fancy, just some PABX. They like to have an independent backup connection and a backup control room.

context: https://news.ycombinator.com/item?id=18775352

Since the current post adds information, we'll leave it up instead of marking it as a dupe, and have moved most of the non-time-sensitive comments here.

Notes from their massive DWDM system outage earlier on the 27th, which preceded the 911 outage:

    2018-12-28 13:35:00 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains.
    2018-12-28 13:27:30 GMT - The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes. CenturyLink NOC management remains in contact with the equipment vendor to obtain updates as restoration efforts continue.
    2018-12-28 11:04:24 GMT - CenturyLink continues to work with the Equipment Vendor to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Client services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes.
    2018-12-28 10:05:18 GMT - CenturyLink NOC Management reports steady progression of node recovery and restoral of client services. In addition to the remote node recovery process, Field Operations continue to dispatch and assist the Equipment Vendor with local equipment login.
    2018-12-28 08:51:29 GMT - CenturyLink NOC Management has advised that repair efforts are steadily progressing, and services are incrementally restoring. The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels at this time. There have been additional restoration steps identified for certain nodes, which includes either line card resets or Field Operations dispatches for local equipment login, that have impeded the restoration process. Various repair teams are working in tandem on these actions to ensure that services are restored in the most expeditious method available. Restoration efforts are ongoing.
    2018-12-28 07:12:32 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. Additional information on repair progress will be available from the Equipment Vendor by 07:30 GMT. Information will be relayed as soon as it is obtained.
    2018-12-28 06:00:01 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains.
    2018-12-28 04:58:44 GMT - CenturyLink engineers in conjunction with the Equipment VendorÂ¿s Tier IV Technical Support team have identified the elements causing the impact to customer services. Through the filters being applied and the removal of the secondary communication channels, it is anticipated services will be fully restored within four hours.Â¿We apologize for any inconvenience this caused our customers. Additional details regarding details of the underlying cause will be relayed as available.
    2018-12-28 04:09:31 GMT - The Equipment VendorÂ¿s Tier IV Technical Support team in conjunction with CenturyLink Tier III Technical Support continues to remotely work to remove the secondary communication channel tunnels across the network until full visibility can be restored, as well as applying the necessary polling filter to each of the reachable nodes.
    2018-12-28 02:53:38 GMT - The Transport NOC has confirmed that cooperative efforts remain ongoing to remove the secondary communication channel tunnel across the network until full visibility can be restored, as well as applying the necessary filter to each of the reachable nodes. It has been confirmed that both of these actions are being performed remotely, but an estimated time to complete the activities is not available at this time.
    2018-12-28 01:58:56 GMT - Once the card was removed in Denver, CO it was confirmed that there was no significant improvement. Additional packet captures, and logs will be pulled from the device with the card removed to further isolate the root cause. The Equipment vendor continues to work with CenturyLink Field Operations at multiple sites to remove the secondary communication channel tunnel across the network until full visibility can be restored. The equipment vendor has identified a number of additional nodes that visibility has been restored to, and their engineers are currently working to apply the necessary filter to each of the reachable nodes.
    2018-12-28 00:59:04 GMT - Following the review of the logs and packet captures, the Equipment Vendor's Tier IV Support team has identified a suspected card issue in Denver, CO. Field Operations has arrived on site and are working in cooperation with the Equipment Vendor to remove the card.
    2018-12-27 23:57:16 GMT - The Equipment Vendor is currently reviewing the logs and packet captures from devices that have been completed, while logs and packet captures continue to be pulled from additional devices. The necessary teams continue to remove a secondary communication channel tunnel across the network until visibility can be restored. All technical teams continue to diligently work to review the information obtained in an effort to isolate the root cause.
    2018-12-27 22:52:43 GMT - Multiple teams continue work to pull additional logs and packet captures on devices that have had visibility restored, which will be scrutinized during root cause analysis. The Tier IV Equipment Vendor Technical Support team in conjunction with Field Operations are working to remove a secondary communication channel tunnel across the network until visibility can be restored. The Equipment Vendor Support team has dispatched their Field Operations team to the site in Chicago, IL and has been obtaining data directly from the equipment.
    2018-12-27 21:35:55 GMT - It has been advised that visibility has been restored to both the Chicago, IL and Atlanta, GA sites. Engineering and Tier IV Equipment Vendor Technical Support are currently working to obtain additional logs from devices across multiple sites including Chicago and Atlanta to further isolate the root cause.
    2018-12-27 21:01:26 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC was engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO and New Orleans, LA a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. Tier IV Equipment Vendor Technical Support continues to review equipment logs from the sites where visibility was previously restored. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.

Sysadmins and networking subreddits were tracking this as well.

https://www.reddit.com/r/networking/comments/a9z6tb/centuryl...

Some areas were fully out. While others (like Denver) saw partial outages/packet-loss as exit nodes depending on the dest were having problems.

That thread has some very interesting tidbits. (like this one: https://www.reddit.com/r/networking/comments/a9z6tb/centuryl... )

My two cents... When a company lays off their most experienced engineers year over year over year... this is what happens. Just my humble opinion of the real root cause.

Given the past and recent news about ongoing infrastructure attacks the US has been seeing, reminiscent of Ukraine, is there any possibility this is related?

Not sure how common the problem still is, but some custom roms for android phones could not use shortcodes, and could not dial 911.

I'd recommend having the full numbers programmed in your phones for emergencies if your playing with new roms.

Also, federal alerts did not go out for the 911 outage, but major cities did. So subscribe to the local alert systems.

"In this case, the outage affected only cellular calls to 911, and not landline calls."

Which means its basically the system which determines which 911 center to route a cell phone based 911 call. That this is still centralized, rather than distributed to the geographical towers is crazy.

After reading about this earlier today I had the amusing experience of tweeting my local PD asking if "our" 911 was still operational given the situation; apparently it is given their reply tweet.

I do still have the actual "direct" local phone numbers of dispatches for the areas I most frequent.

Is this outage for both mobile (cell phones) and landlines?

Does dialing the direct number (XXX-XXX-X911) work?

I've had issues on both when there was no widespread outage to the point I've given up on the utility of 911 and try to put the local police department's number in my phone for emergencies.

From 1990, these guys were so ahead of their time (yea, I know, institutional racism != IT infrastructure, but still...):

https://www.youtube.com/watch?v=CPNK0VspQ0M

Is 911 in the U.S. still that wildly disparate patchwork of networks and operations centres?

You say that as if it is a bad thing. Decentralizing key infrastructure and emergency response systems seems like a good idea.

Absolutely. Especially given this widespread failure, the last thing I want is more central control.

There needs to be some central or regional oversight body to ensure standards are being met, equitable access is being maintained, etc. A lot of poor areas are undeserved by these patchwork networks and several infrastructure deficiencies can compound. Example: poorly maintained roads leads to potholes, those lead to drivers swerving or getting otherwise surprised leading to injuries, which are insufficiently addressed by local healthcare. Imagine if we had a tyrannical electric utility who cut your power for a bill paid one day late.

> There needs to be some central or regional oversight body to ensure standards are being met

You mean something like your local, county, or state governments and their agencies?

>wildly disparate patchwork of networks

This is true of pretty much all types of infrastructure in the USA, from roads to healthcare to electricity.

So, now there is proof that our national emergency 911 system has a possible vulnerability. Hope this will be addressed with some prioritity.

Should cell phone users be plugging in 360-693-3111 as an emergency backup in the event 911 goes down in the future?

I'm quite surprised no fallback/failover was put in place for a critical tier service like this.

time to switch to aws

lol, The Purge (2018)

“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” said Pai in a statement. “The CenturyLink service outage is therefore completely unacceptable, and its breadth and duration are particularly troubling.”

In response CenturyLink is now offering a new service, called "911 Fastlane", that guarantees that your 911 calls will get through and will be prioritized ahead of calls from standard CenturyLink accounts. This service is only an additional $10 a month.

Of course, satire.

Or more likely, CenturyLink is now tacking a mandatory $10 "911 robustness and replication" fee to your bill.

Someone's gotta pay for those duplicated systems.

They already have that. If you look on your phone bill's list of fees, you'll see one for 911. It's not $10, but it's there.

When I read that I always assumed that was a fee that went to the 911 call centers to fund them (paying the people who answered the phones, renting the land they're based out of, etc). Does the entire fee go directly to the phone company?

Here's the relevant California legislation for example (it's regulated at state level): https://law.justia.com/codes/california/2005/rtc/41135-41142... it's pretty straightforward as to where the monies go.

Nobody understands what that is though. So I present to you:

    CenturyLink Peach Fuzz - keeps you peachy and the details fuzzy

That's good enough, what's important is signing popular actors and licensing some really catchy music for the ads.

It is, of course, appreciated that you clarify this is satire, but the fact that you need to is alarming.

As the sibling subthread notes, it isn’t just satire (even if the parent intended it as such), and it’s already happening. They already do charge a 911 fee, they just ... don’t use the money to honor the SLA that it promises.

The ankh morpork firebrigade hard at work.

Absolutely brilliant! You win the internet.

Not sure if your comment is satire or not. Thanks anyway, though.

Judging by the number of downvotes my comment achieved, it seems a lot of folks couldn't tell it was satire. Seemed rather obvious to me.

No, you were downvoted because your comment was content-free and this isn't reddit.

Why did you feel the need to post this weird piece of NN propaganda in a completely unrelated thread?

It seems related to me.

Could you help explain why? I really don't see any connection between NN and this story.

I understand that NN is an important issue for the BigCos employing many HN users, but I'm not really convinced we need their talking points to be brought up in every single vaguely network related thread.

Ok, so it's tangentially related. Even though it was meant in satire, it seems like something that a big telco would try to do.

You're right, sorry about that. Won't happen again. Not satire.

  weird piece of NN propaganda

The irony of the internet ruining itself in a single comment.

I’m not sure that it is pro-NN propaganda, since it implies that some traffic should have priority.

(FWIW, I don’t think you should be downvoted for objecting to the tangent.)

I don't really care if it's pro or against, both sides are primarily pushed by corporate stooges.

Despite the huge push by ISPs and massive media companies like Google, FB and Amazon, this stuff isn't actually really relevant to normal people.

The conversation surrounding NN has been totally poisoned by corporate interests, even here you constantly see comments suggesting that the internet will be destroyed without NN regulations.

Why is it centralized??

Put it on the blockchain.

Strange Apple doesn't push their own tech for this. They could include it in Messages and release on Android. For Android there would be a store button where you could see all the new stuff from Apple and buy an iPhone. Would do a service to the 50% of the world's population who's entire community communication network is owned and mined by FB.

Apple doesn't do this kind of infrastructure. Their stuff only works if the rest of the network works, which seems to be what's falling over here.

Reply on news.ycombinator.com