911 call centers down in Washington state | Hacker News Reader

911 call centers down in Washington state

twitter.com

193 points_eht90 comments

There's another discussion at https://news.ycombinator.com/item?id=18777794. Since that article adds new information, we'll leave it up instead of marking it a dupe.

It’s not entirely clear how bad it is yet, but this is far more widespread than just Washington.

911 has been down today in major cities in various states across the US, including Arizona, Idaho, Texas, Missouri, Oregon... and maybe more.

I only realized this because I used to live in northern Arizona and received an emergency alert earlier today about 911 service being down in that area... then a couple hours later received the local one for King County, WA. After looking around a bit it turned out to be all over the US.

Yep, I got an emergency phone alert (I'm in the Portland area) and I thought it was just a local thing too until someone on facebook mentioned it was national.

https://www.cnbc.com/2018/12/28/nationwide-internet-outage-a...

A friend reported it was down in Frisco, TX last night from midnight to 3am.

Many north Texas communities in fact: https://www.dallasnews.com/news/crime/2018/12/28/911-outage-...

Mass. state police reported the 911 problem has been fixed in Massachusetts. http://www.mspnews.org/2018/12/update-wireless-911-capabilit...

Just got an alert about 911 being out in MA. A bit belated, if state police twitter is correct that it was fixed 3 hrs ago https://twitter.com/MassStatePolice/status/10786404842292224...

Yup.

Same here....I’m in Brookline, where as the MSP stated the earlier outage were for towns much west of Boston.

MEMA just tweeted that it is still ongoing, I was confused about the MA State Police tweets hours before the alert too https://twitter.com/MassEMA/status/1078694654604922885

Broadcasters in WA state equipped with EAS have just had the following pushed to them:

This automated message was sent by a Sage Digital ENDEC. Do not reply to this

message.

Alert Received at 12/27/18 22:54:40 from CAP

Matched filter CIVIL,

Received from CAP.

EOM Received at 12/27/18 22:54:42.

The Civil Authorities have issued a 911 Telephone Outage Emergency for all of

Washington beginning at 10:53 pm and ending at 11:53 pm (fromcap). Washington

State is currently experiencing multiple 911 outages in different parts of the

state resulting in citizens receiving a busy signal. If this occurs during an

emergency please contact the local non emergency number for your area.

We also got alerts here in the Portland metro too. Talk about a mess.

I just got a push notification to my phone about this. According to Clark (County) Regional Emergency Services Agency's Twitter feed this is due to a widespread CenturyLink outage: https://twitter.com/CRESA/status/1078415710064275456?s=20

This is most likely a TDOS attack, given the busy signals [1]. Plus, the transformer that happened to blow last night in NYC adds to the suspicion, as the Ukraine Power Grid attack on December 23, 2015 was a TDOS attack [2], too. Additionally, there was a BGP network hijacking incident to the US Department of Energy today [3].

[1] https://www.nojitter.com/when-911-busy

[2] https://www.wired.com/2016/03/inside-cunning-unprecedented-h...

[3] https://bgpstream.com/event/171779

I've heard about DoS and DDoS but not TDoS. So in case someone else is wondering what TDOS is, it's "Telephony Denial of Service"

It is wild speculation to assert that these events are connected, without any supporting evidence. Let's please keep this kind of thing to a minimum. This should not be considered good content for this forum.

Its speculation, but not terribly wild. This is the world we live in.

The most likely explanation is the holiday effect where people go on vacation and stuff goes wrong.

add to it that there is a group selling a new standard for 911 call systems (which is today managed by a single politician and a team of contractors!). they demoed the proposed architecture on defcon this year.

Yup, my phone went off at 23:34 PST with a warning

We have the same issue here in North Texas... apparently because of centurylink.

The sheriffs office is blaming centurylink.

Perhaps people need multiple communication methods for emergency services.

Don't worry, just remember the new number;

0118 999 881 999 119 725 3

Totally forgot about that one! Thanks!

https://www.youtube.com/watch?v=HWc3WY3fuZU

... 3!

So maybe I'm a little paranoid, but does anyone else think its odd that 911 services are going down AND we had that major transformer explode in NYC? All in the course of a matter of hours?

I'm not saying its anyone one in particular but it is smart to test your weapons before having to rely on them. And its a pretty well established fact that the US power system security is lacking.

Maybe i'm just being paranoid but I'm a security engineer. I'm supposed to be.

I’m going with Hanlon’s Razor on this.

If you wanted to test your cyber weapons, I’d imagine you wouldn’t want to draw too much attention to yourself by testing them simultaneously.

On the other hand, years of cost cutting and other “efficiency” measures can easily explain both the problems with the power grid as well as the 911 networks.

Indeed. Degrade infrastructure overnight and it's a terrorist attack. Degrade it over 20 years and it's "small government".

You can affect this as a foreign power. Pay some think tank to produce material for (so called) media outlets.

I am not a US citizen and not living in the US but I can't recall hearing a foreign official or non-official supporting a reduced US government.

US local politicians are vocals enough about it.

Oh, that would be doing it wrong. Of course it would be presented as locally written journalism, local experts or "concerned citizens" etc.

I'm not saying it's happening, but I certainly see the potential profitability in it.

Obviously they wouldn't do so under their own names, but through Macedonian troll factories, opaquely funded think tanks, Russia Today interviews, twitter bots, and so on.

I absolutely agree this is a home grown phenomenon but there is some evidence that Russia funneled money to Republicans with these beliefs through the NRA. This is just one example we happen to know about because people are looking under every rock after the 2016 election.

https://www.cbsnews.com/news/timeline-of-russian-plot-to-inf...

The NRA is a great example of this; its president is Oliver North, famously involved in a plot to sell weapons to one enemy of America to illegally support a terrorist organisation in another. I'm sure such an upstanding person has America's best interests at heart. /s

>>If you wanted to test your cyber weapons, I’d imagine you wouldn’t want to draw too much attention to yourself by testing them simultaneously.

Unless you wanted to send a message about "we can do this too" ...or a response. Lots of things we don't know but a few people in NSA/CIA might know what this is about--if it was done by a foreign power.

But also things happen by random chance.

I have said this before, but Hanlon's razor is good for politeness, but not very good for assessing what people are actually up to, as people regularly disguise maliciousness as incompetence and find it fun to do so, especially in politics and other power games.

One of the many tricks to power is pleading powerlessness on the things you actually planned ahead of time while claiming full responsibility for things that are accidental.

But what you are describing is already part of the premise of Hanlon's razor. It's true you can't always distinguish maliciousness and incompetence, but incompetence is easier to achieve and so occurs more frequently. That's why any given instance like this is more likely to be incompetence than malice.

It's possible this could be some secret plot disguised as incompetence, but it's also totally reasonable for an event like this to happen from incompetence alone, and I don't think it would surprise anyone if that were the case. So we ought to focus on the reality that this kind of outage is totally possible due to incompetence and implement measures to prevent that.

Not only can you not always distinguish them from each other, but they are very far from being mutually exclusive. Which is somewhere else that Hanlon's razor falls down. It sets them up as being options to choose between, which is obviously a very bad model.

To be honest, when I first heard Hanlon's razor, I immediately wondered what nefarious stuff Hanlon had been up to that he wanted to deflect attention away from.

Is a bit like the old aphorism 'You can't cheat an honest man', which is of most use to con artists trying to put honest people at ease before then cheating them.

edit - also you are misrepresenting Hanlon's razor. It is not an argument that says that stupidity is merely more likely, rather it says - "Never attribute to malice that which is adequately explained by stupidity".

Well an argument to this could be that we cant tell its an attack. In a recent report on US weapon systems it was found that even a simple network scan could crash a system.

"Operators reported that they did not suspect a cyber attack because unexplained crashes were normal for the system."

That exerpt comes from this report which was released two months ago.

https://www.gao.gov/assets/700/694913.pdf

We're no longer in a time when Occams razor tells us that the giant radio active spike over the ocean is a nuclear weapons test. Now Occams razor tells us that every crash is simply due to bad hardware or (and sometimes both) a lack of funding.

>unexplained crashes were normal for the system

this only makes me more concerned

It shouldn't really. If they're really common then everyone already knows how to deal with things when the system crashes, it isn't a big deal. What you need to worry about is what happens when the system that has never crashed and that everyone depends on and no one understands goes down.

All you have to do is go look at various bridges across the US, and roads in urban areas too. The simple fact is that the US absolutely refuses to maintain or repair infrastructure. So we have bridges rusting apart and even collapsing sometimes with vehicles on them.

With a national attitude like that, random unexplained crashes of other critical systems shouldn't be a surprise.

> If you wanted to test your cyber weapons, I’d imagine you wouldn’t want to draw too much attention to yourself by testing them simultaneously.

True, but that assumes this would be a test rather than an actual attack. There are motives where substantial but non-crippling disruption could be the goal.

You don't test your cyber weapons so anyone notices. It's giving away that you have a weapon and gives your enemy time to adapt and protect themselves. Mabe it's someone tired of the boss not listening to the complaints of potentially crippling bugs in the security and is now doing a presentation of what would happen to society if he does not get to fix the bug?

Or a provider of DDOS-protection that is running their own DDOS botnet for kicks? Mirai comes to mind here... :-)

> You don't test your cyber weapons so anyone notices.

Yes, I know. I'm saying it's not impossible that this is an actual attack, not a test.

If you recall the stuxnet attack against iran, it was israel who went ahead and deployed the attack against the wishes of the nsa - as they were chomping at the bit to test it.

Also, they made their tracks known with Torrah quotes in the code.

Unless you knew the us government would know you did it. And this was to show them just how much you've compromised the us...and are making it known.

If I were a hostile country and wanted to create a bit of chaos, I imagine I would try to do it at the most inconvenient time possible. Like, say, when the markets are unusually volatile and the president has been doing things that put everyone on edge, like fire his defense secretary and shut down the government.

Of course, it's also quite likely that this is just a coincidence and the transformer and network outage were just ordinary accidents.

Hostile entity might be a better term considering the times we live in.

The US has been volatile for 2 years running. Cherrypicking examples to create the appearance of rare coincidence is bad stats.

But... we are in the middle of a government shutdown, aren't we? And the stock market is also going bonkers just this last few days. Those are facts, and today this is another one. That's not cherry picking. That's not comprehensive or exhaustive fact-checking either. But they are sure facts!

Or it might be directly related to furloughs somewhere leading to insufficient network maintenance.

can't find the article off hand, but this happened only a few years ago in california; someone cut the local phonelines prior to emptying a machine gun into a substation

Google suggests that was the https://en.wikipedia.org/wiki/Metcalf_sniper_attack in 2013 -- AT&T lines were cut and a "team of gunmen" fired shots into several PG&E transformers, causing substantial damage.

This means people decrying the mindset of seriously asking a question about whether this is an attack are very premature and should let the questions be asked and answered in an orderly and prompt fashion.

But my money is at least partially on federal understaffing during a shut down having a variety of downstream effects.

Given that Russia has threatened such grid attacks continually for over a decade, it has to be evaluated. It reminds me of the Russian robocalls that have threatened school attacks all over the globe but mostly in the United States in order to further degrade the future output of our education system.

If both are related, couldn't it be in the opposite direction? The transformer failure damaging (or powering off) some piece of equipment which then leads to the phone system failure?

(I don't known which happened first, but on the reddit video of the transformer explosion says "8 hours ago", while this twitter post is 4 hours ago, so I'm guessing the transformer exploded first.)

>Maybe i'm just being paranoid but I'm a security engineer. I'm supposed to be.

No, you really shouldn't. Every bad thing can happen unless you protect against it, but being paranoid isn't helpful for this industry.

This was going to happen eventually. If they didn't want this to happen then the funding would be there to have this system built up with triple redundancy (or possibly 7 times, maybe they have triple). We all do the best with the funding we have. The good thing about this is that they assessed the problem and they had remediations for it. They announced it publicly (using several methods) and they have an alternative line.

As a Brit it's interesting to see how the default American response to anything out of the ordinary seems to be "we must be under attack!"

I don't mean this as any kind of criticism - I guess it's just a cultural difference. For example we had someone shut down the second busiest airport in our country last week by flying a drone around and the assumption here (by the man on the street and the media in general) was "idiot with a new toy" rather than "terrorist attack".

> As a Brit it's interesting to see how the default American response to anything out of the ordinary seems to be "we must be under attack!"

As a longtime Hacker News reader, it's interesting to see how strong the urge is for non-American readers to make generalizations about Americans based on cherry-picked data that suits their paradigms.

> For example we had someone shut down the second busiest airport in our country last week by flying a drone around and the assumption here (by the man on the street and the media in general) was "idiot with a new toy" rather than "terrorist attack".

Aside from the fact that (a) there were people speculating about terrorists at LGW, (b) the common response of New Yorkers last night was not to assume it was a terrorist attack, you're also comparing a drone flying over an airport to a massive explosion that lights up the sky for a prolonged period of time - over a minute - with bright pulsating blue light.

I think the 9/11 attacks really left their mark on the public consciousness.

Add to that the near-daily school shootings, miscellaneous other mass shootings, and general sense of frustration & helplessness about ongoing Russian cyberwarfare and interference with our elections (I mean, for God's sake, it seems reasonable to suspect that our President could actually be a Russian intelligence asset), and the zeitgeist has become one of heightened paranoia.

The only problem with the near-daily school shootings and other mass shootings is that those are a choice that we, as a society, have decided we want to tolerate and have in our lives. It's not an attack from outsiders, it's something that we've decided we want to put up with from our own people because we refuse to change our gun laws. Other developed nations simply do not have this problem, for the very simple reason that they don't have easy access to such weapons.

As for the Pres, there again, that's a choice that we made as a society. We elected him after all, and we've continued to defend him no matter what.

In short, these are problems of our own making.

The only problem with the near-daily school shootings and other mass shootings is that those are a choice that we, as a society, have decided we want to tolerate and have in our lives.

I'm getting a little more than annoyed with the constant insinuation, in the face of challenging, difficult and emotionally painful issues in our society getting thrown into a discussion as something we've factually accepted and tolerated as something that just 'is', as if we've thrown our hands up and said "please keep slaughtering our school children".

Especially in the form of throwaway comments from individuals who aren't actually putting forth a solution themselves.

Please stop waxing post hoc ergo propter hoc like this-just because a solution hasn't made its way through the mechanisms of law and procedural solution making, and please stop projecting YOUR cynicism on the issue on to the rest of us.

>we've factually accepted and tolerated as something that just 'is', as if we've thrown our hands up and said "please keep slaughtering our school children".

That's exactly what we've done. We're not making any solutions, because too many voters and politicians absolutely refuse to. To them, this slaughter is the price they're willing to pay for their "freedom" to have guns.

>Especially in the form of throwaway comments from individuals who aren't actually putting forth a solution themselves.

Solutions have been put forth, countless times over many years. The pro-gun side refuses to accept any of them, because it means they can't have all the guns they want. It's really quite simple. We're dealing with extremists, except that these extremists have the vote of about 1/2 the population.

> I mean, for God's sake, it seems reasonable to suspect that our President could actually be a Russian intelligence asset

I don't think so. That is propaganda from the other party. Strongest I've seen in years, honestly.

I agree. I mean, in some vague sense he's an asset because he's been useful- and I imagine that happens from time to time to people with power when the other side has a strategy or a goal you haven't caught on to yet, but I don't think he's aware of it.

He has repeatedly sabotaged our relationship with traditional democratically-governed allies (i.e. NATO) - and avoided criticizing Russia to an extent that is simply bizarre.

For example, despite the repeated statements by American intelligence agencies that Russian interfered in the 2016 election, Trump said in Helsinki that he believed Putin's denials over his own intelligence apparatus. This is just incomprehensible behavior by a President.

https://www.businessinsider.com/trump-putin-press-conference...

> He has repeatedly sabotaged our relationship with traditional democratically-governed allies - and avoided criticizing Russia

The same could be said for his predecessor. See Israel, Japan, hot mic with Medvedev.

I think we see what we want to see and forgive "our guy" because we believe his intentions.

Can you provide specifics on Obama's actions re: Israel and Japan that you find objectionable?

Here's one article that explains the final "f u" obama gave to israel:

https://www.nytimes.com/2016/12/23/world/middleeast/israel-b...

The constant calls for restraint on Israel's part as Israeli children as young as 6 months were being stabbed to death in their beds by "freedom fighters" lauded by PLO and PA media also comes to mind.

The blind eye towards how PLO officials spoke to their own people (bloodthirsty talk of killing all the jews) vs what they said in English (calls for peace if only Israel was willing to compromise).

But just googling "Obama and Israel" will give you tons more.

Obama's weak stance on NK and china at the expense of Japanese security was seen as failing an ally.

I'm wondering if you could provide concrete examples of Trump sabotage. I'm curious mostly because I'm guessing we're gonna see it very differently.

Just so nobody else wastes their time I thought I should point out that that NY Times link doesn't support any of the crazy stuff you're saying.

Just so we're clear, the NYT article is only about Obama choosing not to veto a resolution condemning Israel.

Not sure what I said that was crazy?

here you go:

https://www.haaretz.com/1.5177456

https://www.ynetnews.com/articles/0,7340,L-4041106,00.html

https://www.washingtonpost.com/blogs/right-turn/post/palesti...

For starters, Trump doesn't seem to even comprehend the mutual defense pact that is the foundation of NATO.

https://www.nytimes.com/2018/07/18/world/europe/trump-nato-s...

Apart from that, he was generally a source of chaos at the 2018 NATO summit, 1) demanding that NATO allies increase per-GDP spending ahead of schedule or the US would "go it alone" (and hijacking a meeting already in progress to make these demands), 2) accusing Germany of being "totally controlled" by Russia, 3) and giving a parallel-reality final press conference touting that NATO partners had agreed to significant increases in spending (which said partners denied).

https://www.theguardian.com/us-news/2018/jul/12/how-trumps-n...

The NYT article is more about him questioning why we should do it, he seemed to understand what mutual defense is, just that he didn't want to commit to it. If you recall the parable of the elephant and the mouse, Trump doesn't see the value of the mouse and thus doesn't wanna commit to helping it without question. at least that's what I gleaned from the article.

There is a difference between being "a source of chaos" and actual sabotage of alliances.

1) Still doesn't seem that unfair of a demand.

2) You gotta admit that he had a point that using Russia as a source for oil kinda undermines independence. We ran into the same thing in the 70's, it sucked.

3) Classic Trump. but not sabotage.

A useful idiot is still an asset, perhaps not knowingly.

This predates 9/11. I'm not sure how to best to explain it because it can come off badly because US vs UK discussions aren't very healthy beyond a few jokes.

Americans are just different from other english-speaking countries (the thing people tend to think of as the closest comparison.) It could be a strength or a weakness depending on how you think about it. There are definitely changes that you can attribute to 9/11 such as the Patriot Act, airport security, etc. However, I don't think Americans think much differently that might have done during the Cold War.

Perhaps it would help to frame these discussions as not needing to consider whether aspects of our cultures are strengths or weaknesses, and simply talk about the differences frankly.

Speaking frankly, I can kind of speak to the oddness of the reactions-having once lived within an hour of a couple of air bases and an Army base with an aviation detachment in at least 3 of the four cardinal directions (this was Texas, plenty of military bases) two hours up the road (plus another one on the far side of the state). This was several years ago now, before 9/11.

Now personally, I figured it would be normal seeing military birds flying over head en route to one of the three bases, like I said, we were within two hours drive to one of three installations, by air probably even less.

Evidently the rest of the township at the municipal airport I was working at thought otherwise-and didn't find it so 'normal'. "Jade helm" (speaking of post 9/11 'events') went from a painfully reported news story for three straight weeks to a bit of a joke in our office, but before that nothing could stop people from phoning my desk at general aviation to ask if we were under attack or what business the military has flying over civilian airspace.

"Sir", I would say (or ma'am if appropriate), "we're one hour from Lackland AFB in San Antonio, this is normal aviation activity, those boys are probably just finishing some training and heading back to base for some R&R".

Would wager....15 calls a day, more on the weekends if the hornets were practicing for a flyover of DKR up in Austin for a Texas football game.

Edit:

I completely forgot, the calls got to the point where we had a dedicated phone line installed just so people could call in and get ATIS-ish type information from general aviation-and one of the pre-recorded bulletins, I think the very first thing callers hear was "General aviation does not intercept, record, or log the presence of military aircraft in the airspace immediately surrounding New Braunfels Regional Airport"

They ended up disconnecting that line after a few years. People would figure out another number with a human respondent and badger them instead until one of our hangars started getting phone calls wanting to be transferred to the tower.

The zeitgeist has not recently become one of heightened paranoia, it's the Sword of Damocles that comes with being the biggest target. And you could make a better case that Nixon was an asset of China.

your theory doesn't hold for most things. more school shootings than days in a year? no problem. politicians selling the country for personal gain? no problem.

Americans are pretty chill, dude.

Ouch! (But fair...)

As a Brit it's interesting to see how the default American response to anything out of the ordinary seems to be "we must be under attack!"

I honestly think you are mistaken. It certainly crosses some peoples' minds, and I imagine that any issue will make someone consider it might be terrorism.

But I bet if a bunch of smoke came pouring out of your tube, somebody would wonder if a bombing was attempted. I don't think we're that different that way.

"Idiot with a new toy" is not a very interesting news story in the US. The object is to make sales with headlines. Therefore it's always the worst possible thing. It's not cultural unless you count the media as our culture, which it is part of it.

It definitely crossed my mind. On the other hand, what I've read about the "security" of the infrastructure for controlling power, water, gas and so on doesn't fill me with confidence, and concern for these systems is not unreasonable.

A state-level actor could make the next election day in the US pretty spectacular.

we had someone shut down the second busiest airport in our country last week by flying a drone around and the assumption here (by the man on the street and the media in general) was "idiot with a new toy" rather than "terrorist attack".

I was watching Sky News. It immediately went with terrorism speculation when the drone was first reported.

The question you should be asking is, "what could this be covering up?" Why do this now? For the Lulz... or because there was bigger game afoot, could be either one.

have a look around this morning NY wasnt the only municipality with "transformer fires" last night https://twitter.com/othersideofgrey/status/10786416747236925...

I think it would turn out to be on balance very good for Centurylink's stock price and management if the root cause of the outage was determined to be hostile foreign action instead of, for example, a mix of profiteering and incompetence by company upper management.

emergency numbers are down? that's terrible! How could that ever happen? Oh its US infrastructure? nevermind

Reply on news.ycombinator.com