Alert About Missile Bound for Hawaii Was Sent in Error, Officials Say
nytimes.com
nytimes.com
When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!
There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
0. https://landing.google.com/sre/interview/ben-treynor.html
When is appropriate to fire someone for making a mistake?
Some ideas:
- gross negligence / malice
- ethics
- publicity damage / internet pitchforks
Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."
There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task.
This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...
This isn’t a newbie getting set up on his dev db who was given too much access to production. They’re pointing the finger so that mechanism that facilitates single point of failure should be independently audited
when you're dealing with processes as critical as this one...
there should be no single point of failure.
Even our President has a backup, a failsafe, and a safeguard for the purposes we're discussing here. And that backup has a backup, a failsafe, and a safeguard. Etc etc etc.
It was an open secret that nuclear retaliation could be instigated without White House or even Pentagon approval if the Soviets tried to decapitate Nato.
A rational player would only put effort into a retaliatory strike capability to the extent that actually having a retaliatory strike capability strengthened the enemy's perception that they would be obliterated if they struck first.
I generally agree with the sentiments of this thread - you shouldn't fire people for the kinds of mistakes that all humans frequently make. However, I also think that this was a severe and costly incident - a large number of people thought for a while they were in significant danger, and investigating who is responsible, and making corrections with regards to their employment, may be reasonable depending on the exact circumstances.
I'll give an example. I work in an office building with many floors that have exact same layout. One day I was on a floor that wasn't mine, and went to where my desk should be, and tried to sit down. I was shocked, for a moment, to discover my coworkers had all been replaced by strangers. In this case, I made an absentminded mistake of not paying enough attention to my current location - but the consequences were trivial, so the mistake itself is no big deal. Now, on the other hand, suppose while driving down the road, I absentmindedly ignore a red light, plow through an intersection, and kill a family of five. This is also an absent minded mistake, but the predictable consequences of being absent minded while driving are a lot more extreme than being absent minded while walking about an office building.
To apply the logic of my example to the case at hand - imagine that in one scenario an employee is, during a boring moment, cleaning his keyboard with his chat application open. This results in that employee sending a nonsense message to a coworker. Silly mistake, and no big deal. Now, suppose that same employee has the "Missile Alert" app open and decides to clean their keyboard. This is a similar mistake, but with potentially bigger consequences. I think it's fair to treat these two actions differently.
Of course, the circumstances affect what the reasonable standard of care is. Oopsies by ship captains, surgeons, and even drivers are held to a different standard than me making a typo on some random thing I'm writing or a simple program.
In your example, running a red light is commonly taken as one of those things you just don't screw up on (and that can't be eliminated through automation at this time).
No there aren't. There are certain activities that must be always correct, you implement those by making a reliable system out of unreliable jobs, not by making the people 100% reliable.
We see it all the time in IT - people totally misjudge the 'risk'. 1% risk of breakdown quite often seems acceptable to decision makers until it actually happens. Then all hell breaks loose.
I believe it is much better to fix it (require a second authentication before sending those massages) and search for similar problems within the whole department to make sure something like this never happens again. Make sure everybody, including new hires, are aware of this incident, what the consequences were and that everybody is encouraged to report similar bugs and nobody has to be afraid to be fired when they make a mistake. Otherwise you just create an environment where everybody blames anybody but themselves and the “smallest” guy gets fired for mistakes other’s. This way, things get never fixed probably because everyone ist just afraid of the consequences.
if you direct the blame 1 level further, then it was whoever was responsive for budget/hiring/spec that didn’t ensure that there was a UX designer on the project, but at the same time you can kinda understand why not: the value of UX is frequently not understood, especially when people think “it’s just internal; why make it easy to use if only we see it?”. the answer of course is that you pay for internal systems every day in employee time.
anyway, the point i guess is you can probably trace the “blame” back far enough that it’s too far removed to say it’s anyone’s fault per se, but it proves the point that you need design across the board, and that’s frequently not well understood
From what I understand, we were told the record would never get removed, and we relied upon that. Could our error handling been better? Yes. Could the customer's procedures been better to prevent the record from being deleted? Yes. It was a mistake on both parties.
Who should have gotten fired?
[1] Very scary SLAs with the Oligarchist Phone Company.
[2] It's actually a DNS query. NAPTR records to be precise. Search on that if you are interested.
[3] It was also important because that was the only record we could query and not get charged for it. Everything in the telephony world cross-company (and even cross-department) gets charged. Somebody makes money; somebody pays money.
In this particular case no real harm was done and firing would probably be excessive.
I wish to believe, but probably there was real harm, probably someone will come up with some million figures.
Statistically, probably someone died during those minutes of terror and someone will try to attribute to this incident.
What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. The prankster meant no harm either, but he's going to spend years in jail. Both for sending a wrong alert.
Swatting is intentional, triggering the alert wasn't (at least, so it seems). It's not like the operator pressed the button knowing that it would trigger an alert and expected it not to be taken seriously; he didn't mean to trigger the alert at all.
Ongoing car slams into you and kills your whole family while you were driving drunk; you go to jail for manslaughter and get sued by other driver for all you've got.
Are you kidding? The people who were involved in this didn’t do it to introduce the swattee to new friends. They did it to terrorize someone who made them mad.
What I meant is that “oh I didn’t mean it” going to be interpretted differently.
Edit: if you downvote me thinking that I somehow support prank swatters - you are wrong. Otherwise you fail to see the paradox of intended vs unintended outcomes: https://en.wikipedia.org/wiki/Mens_rea (as I understand it's interpreted very different in the US).
The prankster intentionally sent a false alert and intended, at a minimum, to cause emergency response resources reserved for use in circumstances that pose a grave danger to be misallocated and to cause terror in the intended target; that involves both a diffuse but significant public harm and a narrowly focussed private harm.
It is not at all a case where no harm was intended.
If someone really, as you suggest, recklessly disregarded some procedures because they just couldn't be bothered. Say there's a checklist and they just winged it instead. But maybe there isn't a checklist and they made a mistake.
In general though, firing people to send a message or to throw a sacrifice to the crowd isn't particularly admirable practice.
Why not fire the manager of the person who made the blunder? That would work better on every level.
Her story was regarding an employee adding water to acid instead of the reverse, thereby creating an acidic cloud initiating an evacuation of the building [we're talking high volumes of acid and water here; think pumping in the water using a garden hose].
"You'll never have me doing this again," the person managed through tears.
"No, I'll have you do this from now on as my go-to person," was the response, "because I know you'll never do this again."
This was our "Add Acid" lesson.
The other perspective is that most people would not need to learn that lesson in the first place. This employee is unlikely to make this type of mistake again, but broadly speaking, is he less likely to make mistakes than someone who wouldn't have made this mistake in the first place? Doubtful.
The procedure needs to be reviewed of course. Why is such an important event the responsibility of one person? At least two persons should have to press two buttons and it should be made such that one person cannot press them both.
People saying nothing happened - I don't agree. This could cause serious problems for individuals, heart attacks, accidents. This is not without risk.
A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, "I suppose after that set of mistakes you will want to fire me." Watson was said to have replied, "Not at all, young man, we have just spent a couple of million dollars educating you."
After the sale feel through the VP of sales scheduled an appointment with the sales guy. The sales guy began cleaning up his desk and boxing his things anticipating what the outcome of the meeting would be.
Upon meeting the VP the sales person profusely apologized and accepted responsibility and said he understood why he was being let go. The VP interrupted him and paused for a moment before telling him "I did not invite you here to fire you. I wanted to see how you were doing and ensure that you learned a valuable lesson. The last thing I thought about doing was letting you go after spending millions of dollars on your education and training."
Therein is a key responsibility of the manager: if they've learned, you're 100% right. If they haven't, don't care, don't get it or whatever, then you also need to take responsibility & replace that person.
This doesn't work in all contexts, and I suspect it fails for a large majority of contexts. If the system always prevents you from doing the wrong thing, it will also often prevent you from doing the right thing.
For example, I submit that the philosophy "ability to make a mistake means that the system is designed wrong" completely falls down in the case of going to a new restaurant for lunch and finding out you hate the food there.
I wish this claim, which is very common in discussions like these, came with some analysis of why this odd philosophy which is a terrible idea in so many contexts is instead a good idea in the context at issue.
I think a small group of people feel "mission accomplished".
Could they do it so say, two people have to input a code or something from different machines, or something more physical like turning two keys, hitting a button etc
Didn’t stop people from doing it.
That very afternoon a plastic guard was installed over the button so you had to lift it first.
As I commented below, if this happened when the stock market was open, this could have had a huge impact financially. Over 30 minutes with no updates is an eternity for news to hit Wall Street and algo trades start kicking in.
[0] https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_"A...
This is different than Capt. Asoh's defense because Asoh was directly responsible for
> attempt[ing] an automatic-coupled Instrument Landing System (ILS) approach, something neither [he nor Capt. Hazen] had done before on a recorded DC-8 flight.
In any case, your reference to the defense is fantastic. In the profane spirit of what Asoh told the National Transportation Safety Board investigators--"As you Americans say, I fucked up."--I think it's not too off the mark to observe "What an Asoh". [0]
[0] I tried resisting this but it's just sitting there, y'know. Also, I'm sure my American eyes are mispronouncing Captain Asoh's name. For what it's worth, we should all be so forthright to own up to the mistakes we've made.
I was shocked at the 2.5mi deviation until I realized that this is about 51 seconds of flight at his airspeed of 177mi/hr.
Still, that's a pretty astonishing distance from the runway.
No way we should accept errors like this as some sort of blameless par for course in software engineering. Especially as SV grows more and more into life-critical systems like autonomous vehicles. If there was deep incompetence in architecting a system like this then yes, potentially the employees responsible for that architecture could be disciplined. If the responsible employees were too junior to design appropriate safeguards correctly, then the finger points at the managers who set up the team that way.
It sounds like the parties to blame here lie somewhere in the management chain between the poor unfortunate button-pusher and the director of all emergency management for Hawaii.
There is a lot of fearmongering just lately around the idea that there's going to be a nuclear war with North Korea because the president likes to talk shit on Twitter. I've had so much practice, with my explanation of all the reasons that won't happen, that by now I could do it as an elevator pitch if I needed to. That's why so many people are having this "weird panicked reaction" - panicked, yes, but not weird, when so much effort has been spent to insist that an arrantly implausible counterfactual is not just possible but likely.
(On that note, I expect the next wave of articles and op-eds to revolve around Hawaii's recent "close call", and why it means everyone should be much more scared than everyone is already. Who cares about externalities like the health of the republic when there's attention to be monetized?)
Must be reading some DevOps blogs about blameless post-mortems.
[1] https://en.wikipedia.org/wiki/Emergency_Alert_System#Inciden...
[2] https://en.wikipedia.org/wiki/Emergency_Broadcast_System#Fal...
A similar, if obviously much smaller and less disastrous example, happens at my apartment about 4 times a week: the fire alarms for entire floors of my apartment building are easily triggered by people smoking in the breeze ways or burning their dinner. The result is that I routinely ignore fire alarms because the likelihood of a real fire is has been demonstrated to be exceedingly small.
[0]https://static01.nyt.com/images/2018/01/14/us/14xp-hawaii-su...
[1]https://www.congress.gov/bill/109th-congress/house-bill/5785...
[2]http://www.abc.net.au/news/science/2017-12-03/after-25-years...
[3]https://isogg.org/wiki/How_long_is_a_generation%3F_Science_p...
Clearly there is a whole bunch of new attention being paid towards this.
And rightfully so given the provocations from NK and the general decline in maintenance/quality of the US nuclear and ICBM weapon systems.
In Denmark we have a national system of sirens. They are tested once a year at a specific date and time. In the cold war era they used to test them every Wednesday at noon, but it has been scaled down since then :-)
(This false alarm was a push alert sent to cell phones. What about people who don't have a cell phone on them when the alert comes? ... good luck)
They are primarily for tornadoes rather than nuclear weapons though.
At 5pm they play a 30-second traditional Japanese lullaby. If you ask people, they say it means it's time for children to go home - though as far as I can tell it has nothing to do with actual school schedules. I'm not sure everyone realizes it's actually a warning system.
This is just not true. The area in which the nuke will get everyone regardless is less than a tenth of the area where taking precautions would save you.
This is a bit of a pet peeve of mine. People widely laugh at "duck and cover" as if it was some huge joke. Even at the height of the cold war when the nuke stocks were at their largest, only a small portion of the population would be screwed regardless of what they did. While the majority of the population would be at a distance from the closest explosion where ducking and covering would save them.
It's interesting, too, to read historical accounts of sieges in the Napoleonic and earlier eras; it would take days or weeks of pounding by breaching batteries of heavy cannon to reduce masonry walls to effect a breach and allow an assault over the pile of broken rubble remaining, in a bloody, hand-to-hand melee. And often there would be so much time between firings that the defenders could rebuild new defenses in depth behind the breach. Unless a powder magazine was struck, massive damage was relatively rare.
Having grown up during the cold war, I'd say that people thought "duck and cover" was ridiculous not because of some misunderstanding about its usefulness in protecting one from blast, though that misunderstanding might have existed. The reason everyone thought it was ridiculous was that everyone realized that surviving and having to deal with a completely smashed infrastructure and radioactive fallout was an extremely terrible, shitty outcome that you were not going to save yourself from by hiding under a desk or something.
Tomorrow there will be more people in favor of putting a swift and aggressive end to NK's nuclear ambitions than there were yesterday.
My point being, if you can see a tornado coming, you'll have ample time to take shelter in a basement [or if you're in an unpopulated area, you can get out of the way]. Why people get killed by tornados "still" is usually insufficient shelter (homes without basements or safe rooms), quick touchdowns, them occurring at odd hours when people are sleeping, or the occasional goliath tornado with a massive damage track that's moving quickly, in the wrong direction.
I've also experienced being caught in the middle of a tornado outbreak (30 in 90 minutes, a couple were over mile wide) while driving across southern Minnesota. We sheltered at a Walmart and were quite alarmed that no locals seemed to take it seriously, even when the roof partially lifted off and cars started getting tossed around in the parking lot.
IIRC the State recently switched to more localized warnings (not county wide) due to people ignoring the sirens.
I'm also pretty sure that "you can see tornadoes coming" is poor advice, especially since they often come at night or with significant rainfall.
Everyone cheered when the “False alarm” SMS came through across the phones.
Secure your networks people - this electronic psyops stuff is real.
Is it cover when you’re inside but 30 floors up in a high rise?
http://static3.businessinsider.com/image/58cc34b9112f7043268...
Not common knowledge, though, in part because nuclear hazards have been painted in an exaggerated light by Hollywood (basically, no point in trying to survive, because everybody is going to die and what's going to be left is a 1,000-year lethal nuclear wasteland), and in part because we stopped worrying after the end of the Cold War.
Despite the goofy title, this is a remarkably good book from the 1960s, citing some actual science, that helps grasp the actual dangers and the survivability of nuclear attacks or accidents:
http://www.madisoncountyema.com/nwss.pdf
PS. For folks interested in less apocalyptic emergency preparedness tasks, I maintain a handy guide:
http://lcamtuf.coredump.cx/gcnc/
For something a bit less depressing.
I've often wondered, what does "protection" mean if the building has collapsed around us, so we're now 20 feet below ground, with a pile of rubble on top, and no way to get out?
I don't think you should direct your ire towards Hollywood, at least not in any significant way. I don't think it was until "The Day After" (1983) that any movie showed anything like a realistic depiction of the aftermath of a nuclear war, and even that was deliberately downplayed.
It surely wasn't a Hollywood depiction which caused Dorothy Day and others in 1955 to protest the "Operation Alert" drill, saying:
> We will not obey this order to pretend, to evacuate, to hide. In view of the certain knowledge the administration of this country has that there is no defense in atomic warfare, we know this drill to be a military act in a cold war to instill fear, to prepare the collective mind for war. We refuse to cooperate.
This "The Heritage Foundation" report from 1984 titled "The New Case for Civil Defense" also doesn't mention anything about Hollywood depictions. https://www.heritage.org/defense/report/the-new-case-civil-d...
Finally, the NWSS book you cited says "American official policy, or at any rate the implementation of that policy, is based on the assumption that civil defense is useless." Again, I don't think that policy was influenced by Hollywood.
(BTW, it's funny that Teller says "With the use of American automobiles an evacuation could be faster and more effective than is possible in Russia." - I guess he never saw a city trying to evacuate from a hurricane. Or the plans to evaluate NYC should there be a major disaster at Indian Point.)
Instead, NWSS and The Heritage Foundation (and an essay I read by Freeman Dyson) all say the US policy of MAD was a much bigger influence.
(I'm not going to get into a discussion of the validity of MAD. I only want to point out that I disagree with the idea that Hollywood depiction had much of a role.)
In some cases, these were inaccurate simply because it resulted in a better movie or a novel; but in many other cases, they were probably informed by anti-war or anti-proliferation sentiments. I don't think this deserves any special ire, TBH; it's just our reality. I loved Dr. Strangelove, but it sure affected public perception in a particular way.
But I think there should be a stronger criterion than that before saying that the lack of common knowledge of the effectiveness of DIY civil defense shelters is in part due to how nuclear hazards have been portrayed by Hollywood.
It could be because they aren't effective against the type of nuclear exchange expected during the Cold War.
Most US policy makers, including Eisenhower, were convinced that there was no good civil defense against an all-out nuclear war. This lead to MAD, and the policy of MAD demands that a country not be able to protect its citizens. This was the US policy for most of the Cold War. Which means those Hollywood films reflect US policy.
A problem is, MAD requires an effective nuclear response force, with the expectation that most citizens will die. How do you convince the citizens to fund MAD? One way is to convince them that shelters are effective, even if the high-level planners know that it isn't. This was possible early on because of the secrecy about the nuclear bomb project.
The problem is, civil defense, unlike just about all other aspects of the Cold War, requires convincing the public of its effectiveness. And the government attempts were not convincing. This helped promote anti-proliferation efforts.
Which is why I don't accept your implication that because something is "informed by ... anti-proliferation sentiments" it means that we should ignore it. Those sentiments may have a reasonable basis.
Others believed in NUTS, with the possibility of a limited nuclear exchange, which is survivable for a large country like the US. NUTS played a bigger role during the Kennedy and Reagan administrations, which is why there was more government promotion of civil defense shelters then.
I'm almost certain that Teller would be in the NUTS camp. He certainly had Reagan's ear when he oversold SDI. Even if not, there were plenty of people who were, and those are the sorts of people who would (perhaps optimistically, perhaps reasonably) push that people have a nuclear bomb shelter. Teller's support of civil defense shelters was informed by his full-nuclear-response sentiments, which also "affected public perception in a particular way."
That said, the Cold War context, the idea of having a nationwide civil defense was that, after the few weeks are over and the all-clear signal given, we would help clean up and be able to return to a life that was little different than what we had before.
The reality is that, sure, perhaps a shelter could help millions more people survive the war, but come out to what sort of reality?
And it's not just Hollywood. Even before Dr. Strangelove, there were some widely read fiction books on the topic. The ones I've know are "On the Beach" (1957), "Alas, Babylon" (1959), and "Fail-Safe" (1962). (And a shout-out to "Malevil" (1972), which was the first 'modern' (post-Verne) French science fiction story I read.)
Again, the question isn't if they affected public perception "in a particular way", but rather if they lead to a more complete understanding of the topic.
And I don't think Hollywood's portrayal was much different than what was already well-known at the high policy levels, which is why I don't think it's right to single them out.
That is classic cold war stuff. Before you even get to the foreward by Edward Teller you get a preface stressing (among other things) that low doses of radiation are "healthful."
small doses of radioactivity are hormetic, healthful because they stimulate the immune system. This was proven in laboratories as far back as the 1920's. With the advent of the A-bomb almost all the hormetic research stopped. And only in the last decade has it resumed on a serious scale.
I wonder who they experimented on and whether that stuff is written up somewhere. Ionizing radiation is okay in my book, how about yours? is not exactly a common theme in modern medicine.
In the documentary "Iraq: The Untold Story", the creator shows civilian air raid shelters in Baghdad. They were four stories down, with the upper floors all reinforced concrete. Yet there were powerless against US bunker busting bombs, and the hundreds of civilians in the shelters that were hit all died.
http://nuclearsecrecy.com/nukemap/
If you are hit directly, you're fucked no matter what. But the fireball has relatively small radius compared to the other zones.
Ducking underneath something solid is to protect you if the building collapses, which is likely to occur in the large air-blast radius. This is much like ducking under a desk in an earthquake.
Cover will save you if you're within the larger thermal radiation radius. Even clothing can be enough to protect you from burns, so any cover you can get is good.
Don't underestimate the protection cover can offer!.
Depending on how much time I have, if I had to take cover, I'd either load up supplies in the car and head to the 3rd underground floor of my office parking garage (but away from the vent stacks in 2 corners of the garage), or if I have less time, I'll jump the fence at the apartment complex next door and hide out in their one floor underground garage.
I live in a wood-framed house, so it's going to provide less protection than an underground garage, though still better than running around the streets.
Even more disturbing was the instructions of the initial warning which was to "hide in a safe place".
Terrifying.
It's probably why in school we went through fire drills, tornado drills, etc. So you at least had some idea what to do and what it would be like.
If they had gotten away from the windows and ducked and covered, they would have been fine.
You'd think the people who "pressed the wrong button" would be able to press the same button again?
Absolutely not. Once you say “fire” you need official confirmation to say “no fire”.
That means certification from the military. The official needed being in a meeting or tending to something of greater importance could easily introduce delays.
In any case, I presume the system’s designers didn’t build in an “oops, fat finger” notification. Getting that ready could have easily taken 30 minutes. This happened on a week-end. The coders could have very well been at home.
USPACOM had to check and give Hawaii an all clear.
Yeah, but imagine getting a legit alert, and then someone fat-fingering the "fat finger" notification. Or someone installing malware that sends that notification for all military related threats.
I'm struggling to think what meeting or item would have an importance higher than "incoming ballistic nuclear missiles".
That’s not what happened. No military system detected a threat. “Incoming ballistic projectile” would be a high priority. “Civilian request for official confirmation everything is fine” is not.
Whether they had or needed "official confirmation" the Hawaii EMA still tweeted "NO missile threat to Hawaii" ~13 min after the false alert https://twitter.com/Hawaii_EMA/status/952243912415985664
Would be nice if they could stick to the same medium and ensure timely delivery for this type of correction.
This happened on a week-end. The coders could have very well been at home.
Again, why would you perform a drill involving mass emergency mobilization and not manage that risk? What if this had happened during a morning commute or in some part of the country where people are more easily panicked?
With great power comes great responsibility, remember?
Was this a drill? I understood it to be an unplanned mistake.
The text of the warning was “EMERGENCY ALERT BALLISTIC MISSILE THREAT INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL.” That seems really specific.
Also, the news story we're commenting on describes it as a drill that is conducted every shift change, so yes I think it's a drill because that's how they describe it.
Officials said the alert was the result of human error and not the work of hackers or a foreign government. The mistake occurred during a shift-change drill that takes place three times a day at the emergency command post, according to Richard Rapoza, a spokesman for the agency.
I mean it's probably just a simple mistake, but it's also the kind of thing you would do if you wanted to stoke fears about a nuclear strike.
A screw up seems more likely though.
It makes sense to have drills right now, India (tests with war against China and Pakistan), China (Telling their soldiers to be prepared to die for China), Russia(tests against Nato), and North Korea (getting ready for war with the US) are all having military tests for basically what will quickly become another world war.
We are really at a big crossroads as a large amount of people are rejecting globalization in many different countries. And with such major powers willing to fight hard for resources like Ukraine, South China Sea, Oil Eu pipelines, and not even mentioning the increasing gulf between various countries on core ideologies and creeds.
We really are at a new and dangerous crossroads.
- Officials said the alert was the result of human error and not the work of hackers or a foreign government. The mistake occurred during a shift-change drill that takes place three times a day at the emergency command post, according to Richard Rapoza, a spokesman for the agency.
- In Washington, Lindsay Walters, a deputy press secretary, said that President Trump had been informed of the events. “The president has been briefed on the state of Hawaii’s emergency management exercise,” she said. “This was purely a state exercise.”
Approx. 8:05 a.m. – A routine internal test during a shift change was initiated. This was a test that involved the Emergency Alert System, the Wireless Emergency Alert, but no warning sirens.
8:07 a.m. – A warning test was triggered statewide by the State Warning Point, HI-EMA.
8:10 a.m. – State Adjutant Maj. Gen. Joe Logan, validated with the U.S. Pacific Command that there was no missile launch. Honolulu Police Department notified of the false alarm by HI-EMA.
8:13 a.m. – State Warning Point issues a cancellation of the Civil Danger Warning Message. This would have prevented the initial alert from being rebroadcast to phones that may not have received it yet. For instance, if a phone was not on at 8:07 a.m., if someone was out of range and has since came into cell coverage (Hikers, Mariners, etc.) and/or people getting off a plane.
8:20 a.m. – HI-EMA issues public notification of cancellation via their Facebook and Twitter accounts.
8:24 a.m. – Governor Ige retweets HI-EMA’s cancellation notice.
8:30 a.m. – Governor posts cancellation notification to his Facebook page.
8:45 a.m. – After getting authorization from FEMA Integral Public Alert and Warning System, HIEMA issued a “Civil Emergency Message” remotely. The following action was executed by the Emergency Alert System (EAS): 1. EAS message over Local TV/Radio Audio Broadcast & Television Crawler Banner. “False Alarm. There is no missile threat to Hawaii.” “False Alarm. There is no missile threat or danger to the State of Hawaii. Repeat. There is no missile threat or danger to the State of Hawaii. False Alarm.” 2. Wireless Emergency Alert (WEA) “False Alarm. There is no missile threat or danger to the State of Hawaii.”
9:30 a.m. – Governor makes initial media notification.
9:34 a.m. – Governor’s message posted to his Facebook and Twitter accounts."
http://www.washingtonexaminer.com/hawaii-releases-timeline-o...
There are lots of potential explanations...whatever happened, there is stuff to work on in the aftermath of this. Barring the idea that it was an actual missile that failed to hit its target, if these alert systems are vulnerable, that's a bad thing. This could have caused serious chaos in a large city like New York. Techniques like this could be used to cause gridlock before a terrorist attack, suppress voter turnout on election days, etc.
Source: http://fortune.com/2018/01/13/hawaii-false-missile-alert/
Hopefully instead of seeking to cast blame or personnel firings, etc., folks will learn what it is they lack in emergency response.
A drill is supposed to tell you how close or far away you are from being prepared. A surprise drill is going to be much more effective. The threat is so great, if there haven't been serious drills in a long time then they were overdue.
Now we get to see & analyze the "real life" reaction to this dress rehearsal. I'm sure everyone knew without this that a real alert would be full of SNAFUs, but you don't usually get a chance to see what they would really be if everyone actually believed the alert was real.
Hopefully (and I think they will ) they'll be smart enough to use this rare opportunity.
For folks who want to understand the actual dangers and survivability of an ICBM strike, I strongly suggest a book from the 1960s written by one of the folks involved in the US nuclear program during the Cold War:
http://www.madisoncountyema.com/nwss.pdf
It cuts through many of the Hollywood-perpetuated myths - the certain and painful death in case of a nuclear strike, or the 10,000-year radioactive wasteland that's going to be left behind.
For example, it discusses why the oft-ridiculed duck-and-cover strategy is actually surprisingly effective. The primary threat from an air burst is very conventional - a shockwave and an intense burst of thermal radiation. Shelter - any shelter - greatly improves your survival odds.
The fallout from air bursts is comparatively modest (i.e., tends to be far lower than from an event such as Chernobyl) and while lethal, it decays very rapidly - dropping to reasonably safe levels in a matter of days, not centuries. Staying sheltered for 2-10 days greatly improves your odds, and the thickness of material between you and any surfaces that gather dust (roofs, ground) matters more than anything else. Here's a handy chart:
http://static3.businessinsider.com/image/58cc34b9112f7043268...
In other words, having enough food and water in your home to weather out a nasty stowstorm also makes you well-prepared for the nuclear apocalypse. Mattresses and bulky furniture provide decent shielding when all other options fail.
The long-term effect of fallout tend to be exaggerated, too; water from streams, deep lakes, or wells should be safe or get safe very quickly. Removing a layer of topsoil allows relatively safe crops to be grown. Mild radiation sickness, at the levels where people start experiencing vomiting and hair loss, is actually pretty survivable and has a relatively modest impact on your odds of developing cancer later in life.
(Plus, keep in mind that more than 2,000 nuclear tests have been conducted so far, including around 900 in Nevada alone; while they had some statistically observable negative effects, they have not turned the world into a nuclear wasteland.)
Of course, don't get me wrong - even a single nuclear strike would be awful, and a large-scale confrontation would mean untold damages and loss of life. But the important point is that a lot of people would survive and would be able to do well in the aftermath - more so if we teach them about some common-sense preparedness steps.
The main reason why our understanding of the nuclear risk is so lopsided is because for decades, many nuclear disarmament activists (including many prominent screenwriters, celebrities, and pundits) had a vested interested in portraying the already-awful outcomes of a potential nuclear war as far less survivable and far more hopeless than in reality; the mockery of duck-and-cover, the "barren wasteland" imagery in the movies, and the largely-discredited scientific theories like the "nuclear winter"... all helped to advance (otherwise noble) goals, but at the expense of teaching people that there's nothing they can do save themselves.
Plus, of course, after Cold War, we have fewer reasons to worry. It's hard to top the Cuban Missile Crisis. There's plenty of politicized hyperbole around nuclear tensions right now, but the reality is that a large-scale strike on the US is a lot less likely than throughout a good part of the 20th century.
PS. I have a short summary of NWSS and some other points about this topic (and other, more mundane but plausible hazards) in my "Doomsday Prepping for Less Crazy Folk" - http://lcamtuf.coredump.cx/prep/
Did you write this? This is great!
Why did you need to add this? It was a great post otherwise.
Some of these portrayals were exaggerated simply because it resulted in a better story, but some were almost certainly colored by anti-war and anti-proliferation sentiments predominant among the cultural elites of that time. This wasn't coordinated or meant to advance some sinister agenda, but for better or worse, it skewed our understanding of what we can do in the unlikely case that any ICBMs actually fly.
An argument can be made that another factor was the government's desire to discourage the Soviets from ever trying to attack us, but I'm unconvinced - their generals, politicians, and nuclear scientists sure had a more realistic understanding of what would happen. Besides, the anti-nuclear and anti-war sentiments hurt the government in many other ways (nuclear power generation, nuclear weapons testing, Vietnam...).
Now, I'm not particularly angry at that, and I sure loved Dr. Strangelove.
Besides the fear and chaos potentially caused to people in Hawaii there are financial implications if the stock market were open.
Looking through the Github and Slack system status pages for this month so far ... I don't know that's exactly the model to follow.
Status lights will still all be green
And look at the results! Those companies never have outages or push out defective code into production or allow security vulnerabilities through speculative execution say.
I'm being a bit sarcastic but glass houses, etc.
In my mind the focus should be on the process for sending such an alert and how it could of gotten through by accident.
I hope this event wasn't too stressful for you and your family.
At Korea's level of technology, for example, you'd probably just pick a secondary target in some other place entirely.
Surely this wasn't one step gone wrong.
[clear] [send and cause panic in 1.4 million people]
[Send and cause panic] [Details]
The civilian notification system sent out a false positive. Saying “false alarm” would require, at that point, asking the military for affirmative confirmation nothing is wrong. That is a lower priority than responding to an active threat.
Could. It didn’t. This was a mistake. You’re being irrational.
No, that is the active threat and anyone in power would know it. A panicked civilian population can do about as much damage as a real bomb and the military knows it. Such a long delay in the all-clear message is incompetent and dangerous.
Honolulu was vaporised? Keep your perspective.
Responding was a priority. The error was corrected in 13 minutes. It’s just not as high of a priority as an incoming ballistic projectile.
Was this a drill? I understood it to be an unplanned mistake. In either case, I agree. This was a fuck-up. I am simply saying the cause for the delay, once the mistake was made, is understandable.
> You keep trotting out this excuse
This is a shitty thing to say.
I have seen your user name on HN before. I thought simple etiquette might apply to our discussions. I am sorry for presuming your civility.
http://thehill.com/homenews/administration/368907-white-hous...
Furthermore, allow me to quote from the NY Times story on which we are commenting:
Officials said the alert was the result of human error and not the work of hackers or a foreign government. The mistake occurred during a shift-change drill that takes place three times a day at the emergency command post, according to Richard Rapoza, a spokesman for the agency. (emphasis added)
Now you're accusing me of lying. Previously you thought my disagreement with your dismissal of the issue was shitty and you accused me of being irrational. You are in no position to complain about civility.
Please retract your false accusation.
You then accused me of “trotting out an excuse.” I noted the incivility of that statement and, as you repeated false and specific facts, highlighted your lie.
Let me go one layer deeper. You are upset a civil notification system malfunctioned, by saying false things. That is reasonable. But in reacting to that, you said false things.
These are unfortunate mistakes. Learning should result.
Retract your false claims.
[1] http://www.chicagotribune.com/news/nationworld/ct-hawaii-inb...
The facts support me, not you. I have highlighted the sentence in which the words 'mistake' and 'drill' appear since you seem unable to read it on your own.
HEre is another quote from the same article:
Vern T. Miyagi, the administrator of the agency, said that during the drill, an employee — whom he did not identify — mistakenly pushed a button on a computer screen to send out the alert, rather than one marked to test it. He said the employee answered “yes” when asked by the system if he was sure he wanted to send the message.
I see no indication of the NYT report having been discredited or the NYT making any effort to retract or correct it, and your story from a different publication in no way contradicts it - if anything, it supports my basic point about the irresponsibility of the long delay in notifying the public of the error.
I am not the one lying here.
I'm hoping the withdrawal of exercises of the USA and peace talks actually get us somewhere, but we've been to the table 100 times, I don't see much changing.
'rm -rf $HOME/ foo' vs 'rm -rf $HOME/foo'
If that ain't an option, jumping into the ocean doesn't seem like such a terrible idea.
That’s a terrible idea. You’d be directly exposed when at the surface and end up inhaling fall-out.
>Salt in seawater readily absorbs neutrons into both the sodium-23 and chlorine-35 atoms, which change to radioactive isotopes. Sodium-24 has a half life of about 15 hours, while that of chlorine-36 (which has a lower absorption cross-section) is 300,000 years; the sodium is therefore the most dangerous contaminant since it has the shortsest half life.
For that - take shelter inside, anywhere. Make sure you shut all windows and doors tight, and ideally tape shut vents. Cover openings with a wet towel. Stay put (don't go outside!) and shelter in an interior room for at least one and ideally 3-5 days. Radiation follows an exponential decay curve, so waiting longer significantly reduces the amount of radiation you are exposed to. You have about 45 minutes after the blast (~1 hour after launch) before fall-out starts arriving in significant quantities: enough to take shelter and do basic preparations, but not enough to travel any significant distance.
The biggest physiological danger comes from inhaled alpha-particle emitters, i.e. radioactive dust. Alpha particles have only a few cm range and are easily stopped by wood, paper, or skin, though, so don't breathe the dust. Beta particles are also easily stopped by clothing or a few meters of air. There's not much you can do about gamma rays or X-rays (this is what your hypothetical lead-lined vault is for), but they are physiologically less dangerous than alpha and beta particles.
https://nwschat.weather.gov/p.php?pid=201801131825-PHFO-NOHW...
In the first case, we'll probably only be told about it if they actually make an arrest and the system is patched.
The second is an interesting case, we'll come back to.
The final one: it was intentional and used to track what happened on the islands, and to also watch that information propagate back to the mainland. I feel like all of America has forgotten about PRISM/the NSA. With their immense data collection architectures and ties to large tech companies, they could easily filter the keywords and images they want, and get measurements of how quickly this information propagates and how people react.
Even more freaky, they can use language processing to even get numbers about what people might be feeling! I think we've all forgotten about this and it should be a chilling affect.
In the final case, we will, of course, never know. Anyone who suggests it will be called a conspiracy theorist (even though there is evidence the US government has preformed operations like this in the past, e.g. COINTELPRO) and the official line will be either reason 1 or 2 .. possibly with a patsy to arrest if need be. The truth will be declassified in 30~40 years so some people will be able to say "I told you so," but long beyond the time period anyone will actually care.
The pattern is fairly predictable, too. Twitter users will hear of this first, followed by pockets of people with close friends in Hawaii. Then it will make the news sites and TV news.
The least nefarious reason for a 3 letter agency to do this would be to dry-run a reaction plan to such an event, and take what they learn from reactions & apply what they learn to improve said plan
Then again, it could be a dry-run to measure peoples sentiments toward a war with NK. Less likely, but still possible.
If there is anything nefarious here (seems like it), this is the work of a foreign state actor probing our infrastructure for exploitable weaknesses and sizing up our response. I'll put my money on Russia, who has already been attacking our elections¹, energy infrastructure², social media, and spearphishing candidates and elected officials³.
They have conducted similar attacks in the past, sending out tweets and text message alerts to hundreds of people for nonexistent emergencies⁴.
[1] https://www.bloomberg.com/news/articles/2017-06-13/russian-b...
[2] https://www.washingtonpost.com/world/national-security/russi...
[3] https://www.secureworks.com/research/threat-group-4127-targe...
[4] https://mobile.nytimes.com/2015/06/07/magazine/the-agency.ht...
That escalated quickly.
Or it was "just a prank, bro!"
So the plausibility of Russia doing it would be based on whether they would bother or not.
It doesn't mean you stop looking for causes or eliminate human error though.
The viability of Russia/NK depends on how the EAS network is set up in Hawaii. If they have internet-facing servers, it's completely possible.
https://www.theatlantic.com/technology/archive/2017/01/the-v...
https://www.forbes.com/sites/kalevleetaru/2017/01/01/fake-ne...
Its my personally held opinion that Operation Mockingbird never died. I feel that the intelligence arms are the deep state running the show through misinformation campaigns. My personal opinion held in this, the 2nd paragraph, is 100% conjecture. I wouldn't be surprised if it were found to be true, though.
(It's also in the interest of three-letter agencies and external state actors to raise the suspicion that they have the capability to do something even when they don't.)
Ok, but what would be the point of that?
You would either discover that people are terrified of nuclear war, in which case... you can't do anything useful with that information. Or, you'd discover that people don't respond to the message, in which case... you can't do anything with that information.
Why would anyone who wasn't a particularly stupid villain intentionally do this?
I work in nuclear security so this is kinda my thing. If you're in the fireball zone you're pretty much toast, other than that it's mostly about avoiding the blast pressure wave (as well as secondary effects like buildings collapsing on you) and exposure to airborne fallout, which is most dangerous in the first 24-48 hours. After that it becomes much safer to move around.
I'm curious, what does that mean? Designing shelters? Teaching people what to do in nuclear attacks?
In my case I'm in research; specifically I develop algorithms for locating radioactive sources using sensor networks distributed through a city. My educational background focused heavily on non-proliferation and nuclear forensics. One of the things we studied in my graduate career was how to optimally sample fallout in the immediate aftermath of a detonation based on predicted dispersal in order to get the best quality samples (for forensics work) while minimizing the risk to first responders. Spent a decent chunk of time learning about all the sorts of dangers you expect from a nuclear detonation.
Or, you know, a vodka soda.
I would probably head to one of the local train stations and go down like 3,4 levels deep.
Although FEMA is actively developing more robust controls for their IPAWS system, the controls on user behavior and what functions can be triggered in the system have limited capabilities to enforce restrictions (one would be the requirement for a digital signature to accept a CAP message as valid). If you listen to the press conference [2] Hawaii says they will now be using a 'two person rule' which indicates that the most significant controls they have are manual/behavioral (not automated in the system by user roles or automated workflows based on state policies). Few information systems do much more than have a few coarsely-permissioned user roles, though, so it's not like FEMA or Hawaii has tried to cheap out on the functionality - it's just not a very common capability and emergency alerts isn't a mission where you want to be using 'interesting new tools' that aren't well tested.
There are several alerting systems - the Emergency Alert System (EAS), the Wireless Alert System (WEA), and Non-Weather Emergency Alerts (NWEM). States use FEMA's IPAWS system for sending alerts, which [1] this one seems to have been sent through (localities don't necessarily participate in IPAWS, which is voluntary, but the Hawaii EMA was the one that sent this). Some questions I would have about this would be: - IPAWS messages must have a digital signature to be accepted by the system, however based the Hawaii EMA press conference and articles which say 'an employee made an error' I would guess that the digital signature is not used in a way that is actually tied to the official authorized to declare the alert but to is accessible to their whole emergency Operations Group. - Are they sending test messages with that signature? With a 'two-person rule', it sounds from the press conference that it isn't enforced by the machine (not like having two keys which both have to be turned to send the message) but by the first person stepping away from the machine and letting the other person push the "are you sure you want to send this?" button. That doesn't seem much better, but changing the system to do that gets away from the basic CAP architecture and isn't likely to happen soon.
The FCC is currently working on a proceeding regarding updating WEA to allow more geographic targeting of alerts, the way the other alerts can be targeted at specific locales. The current system dates back to 2011 or 2012, and is pretty coarsely-targeted, which is probably why you're getting Amber Alerts on your phone for a town that is 6 hours away just because it's in your state. You can find it at Proceeding Numbers 15-91 and 15-94 [3].
[1] You can see the message here, which archives messages sent via IPAWS: http://ipawsnonweather.alertblogger.com/?p=18764 [2] http://dod.hawaii.gov/hiema/press-conference-missile-alarm-l... [3] https://www.fcc.gov/fcc-announces-comment-dates-rulemaking-s...
This gives some more technical details on IPAWS and the Common Access Protocol that these messages use: https://www.fema.gov/pdf/emergency/ipaws/ipaws_cap_mg.pdf
Context: I tweeted to someone who was upset about it all, "All your alert are belong to us", then immediately received the second false alarm: https://www.youtube.com/watch?v=KXGnAMp9Nvk
An ICBM (which is what you're usually talking about with nuclear bombs) takes about 30 minutes to reach its destination, which will be a large population centre.
So, you have less than 30 minutes to get out of a large city. Because you won't be the only one to try this there will be major congestion on top of the usual city traffic. Public transport could be an option but I wouldn't be so sure that trains keep running either once such an alert has been issued.
Here[1] is an account by Akiko Takakura. She was less than 300 meters from the hypocenter (the point on the ground directly below the nuclear blast) of the Hiroshima attack. She and others survived because they were inside a well built bank. She was 20 at the time and lived another 40 years.
There are a number of things you can do to contribute to your survival and minimize long term consequences. First is distance; the further you are from a target the better. Assuming you have sufficient distance then you need shelter, anything you can put between you and the blast will help mitigate or entirely eliminate the impact; a hill can make an enormous difference. Should you have the good fortune to survive the blast then you need to deal with fire (nuclear blasts ignite many things simultaneously) and avoiding the bulk of the fallout until you're clear of the damage. If you manage all that you can indeed survive and live a long life.
Much contempt is shown toward the old cold war era videos one can find of school children practicing nuclear attack drills by diving under desks and covering their heads. The fact is that these are entirely legitimate tactics that will mitigate the effect of the light flash, thermal pulse, flying glass and other debris.
"Per White House pool reports, Trump was out on a golf course when the alert was sent."
I don't like the guy, but it's not on him to immediately and personally react to this kind of situation. Maybe a nice reassuring speech after the fact (which I doubt will be forthcoming or satisfying).
"Why did this happen? What are you doing to make sure it doesn't happen again?"
Right now there are way too many people apologizing for his complacency (I guess people have normalized his uselessness at this point?), when it needs to be clear that he's responsible for this shit, too.
Fun fact, in my small hometown you could and possibly still activate the tornado warning system using DTMF and a transmitter on some frequency in the 149mhz range.
To clarify, I never actually attempted it but I had a uniden radio scanner in my teens and noticed the pattern for the 12:00 test.
That refrigerator would have been very handy here too:
"DŌ-OH WAS ABOUT THREE-FOURTHS OF A MILE FROM THE HYPOCENTER, INSIDE A MITSUBISHI TORPEDO FACTORY. THE MASSIVE...FACTORY COLLAPSED ON TOP OF HER AND THOUSANDS OF OTHERS".
https://news.nationalgeographic.com/2015/08/150809-atomic-bo...
Why does everyone assume the bomb is going to fall directly above their heads or in some very small radius? It's a whole bloody island, it could drop anywhere.