The Threat to Americas Electrical Grid Is Bigger Than Imagined | Hacker News Reader

The Threat to Americas Electrical Grid Is Bigger Than Imagined

foreignpolicy.com

91 pointsweatherlight45 comments

Ted Koppel (retired new anchor) began research for a book on this subject after retiring; the book, titled "Lights Out" was published late last year, i believe. Two things about this book were remarkable to me--one was the exhaustive research; the second was a remark from Koppel in the book (i don't recall the exact words he used nor where in the book it is) that it is difficult to imagine a better design for a Power Grid whose primary criterion was maximum exposure to catastrophic attack than the US Power Grid (used collectively to refer to three geographic grids).

Thanks for the recommendation and insights on that book.

The critical piece of evidence presented in his book was the blackout of 2003. Ted Koppel's version was short and implied that the power grid system is so vulnerable that all it took was some overgrown trees to bring the northeast down to its knees for days. He then spent rest of the book talking about SCADA glitches and computing vulnerabilities and disaster preparations (focusing namely on the Mormon community, which is always prepared for a disaster.)

Problem I had with the book is that he either misunderstood or misrepresented that triggering event. Yes, there were overgrown trees, high summer demand and temperatures that made power lines heat up and stretch. Wind speed dropped from 5 to 2 knots and cooled the line less than it normally did in that place. Cables stretched more than usual and grounded. These are all true facts.

However, that's not what brought the power grid down that day. Here's what did.

At 12:40 pm, an engineer working for MISO (transmission company, sort of like a traffic regulator) switched a regulation system to manual mode to make a correction. Note that the correction was made because the system had an errant piece of information regularly enough, which through a phone call the operator knew was false and corrected.

The regulation system was called a "state estimator." It basically takes tons of factors, loads, schedules, downtimes, capacities, etc and optimizes power transmission. Like intelligent traffic control. Of course, all systems have input garbage from time to time, and he corrected this thing.

At 1:30 PM that engineer went to lunch, forgetting to re-enable the automated mode.

At 2:02 PM, a 345k-volt line tripped after touching a tree and started this cascading reaction resulting in a system blackout for a week. Other operators looking at the system readouts didn't know that the state estimator was now sending a false picture, being frozen in manual mode.

By the time the unnamed engineer (from the NERC report) returned, it was too late. Other regulators, operators and power plants had been making uninformed decisions, preparations and adjustments for the power grid based on outdated, non-realtime information. However, the blackout didn't happen instantly -- it took four hours for a cascading chain reaction to trip every safety device in the system.

Lets not even go into SCADA vulnerabilities. Here's some positive outcomes from the blackout: investigators made a staggering list of 46 recommendations -- none addressing tree trimming because vegetation events happened all the time. Those were routine. They did focus on computer security, even though there were no computer faults in this incident.

Secondly, during the blackout the Department of Energy ordered state of Connecticut to energize one of their power plants despite their objections. They complied, and that restored power to metropolitan areas. This action is now a precedent for any future outage events: the legal framework for prioritizing power restoration exists and is in place.

SCADA systems show up on HN often enough to be a point of ridicule. Good news is that the outdated SCADA systems are getting replaced with synchrophasors for the most critical power integration functions. Since 2003, nearly 1700 phasors were added to the national grid, and they're well on their way to be fitted in the roughly 7,000 power plants in the U.S. What it will do over SCADA is improve recovery times of an unstable power grid segment.

Edit: derp, northeast, not nw. Also http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/...

needless to say, i lack the domain knowledge to critically evaluate the conclusions in Koppel's book, so always welcome to hear such evaluation from someone who does.

The "synchrophasor" is interesting and caused me to think of the grid as a giant PLL: better detection of phase discrepancy allows the control system to improve the phase margin.

(grid blackout == PLL loss of lock, too)

I just checked Koppel's book out of the library and finished it. Great book and it pisses me off that we spend so much money on corporate welfare, human welfare, and wars overseas, but we can't spend much less money to harden the power grid.

I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.

> I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.

Our food production, waste and water treatment, hospitals, etc run on electricity, sure. But 80% doesn't seem anchored in reality.

I feel like this irrationality about the risks posed by a blackout is what the article is meant to address.

I think a US Army report had the figure higher than 80%.

A small thought experiment: imagine that you live in a city and for 1 year there was no electricity, no trash pickup, likely no water to drink and to flush toilets, and no gas for cars and trucks.

I think this scenario is very unlikely, but it is still worthwhile hardening this infrastructure.

I admit to being cautious. If it were up to me, I would also spend a lot of resources hardening computing systems: servers in business, kids' school laptops, home computers, IoT, network infrastructure, etc. would have maximum crypto, and we would have a "going to the moon" level effort pushing those aspects of computer science for general security issues.

A year of blackout? Yeah, 80% sounds about right. See, our modern cities are designed around efficient transport (food, sanitation). Back in the day they added sewers to London because it took longer than a day for the nightsoil wagons to get to the dump sites and back - something more efficient was needed.

Today its 1000X that. There's no way to get food to people without efficient transport, which depends utterly on energy. No way to get sewage out. No way to get water flowing. Heck even the traffic lights are needed to keep transport going.

A year without food and water? Without sanitation? Of course nearly everybody dies.

Manual solutions could be found, but its impossible that they could support the current population density. So folks would have to flee the cities. To the countryside, where they just die a different death. After pillaging and destroying any remaining food pipeline.

Food is pretty easy, from my readings of nuclear war survival during the Cold War. E.g. a single semi-tractor trailer, long before their allowed weight was upped to 80,000 pounds, could carry enough wheat to keep 100,000 people fed for one day.

It's the water supply that would be the big killer, you can't go without it for more than a few days, and that just wouldn't be enough time to get enough to everyone.

I'm not sure they'd be so successful in "pillaging and destroying any remaining food pipeline" ... how do you to that to an interstate highway? At most, to the fueling stations, but those are a fairly small scale, and I would expect easily replaced thing. But that would also help to self-limited the radius of the pillaging and destruction, keep it mostly away from the farms themselves, which aren't exactly defenseless if such is needed for only a relatively short period of time.

Its the farms that are vulnerable. Hungry thousands trampling crops looking for anything to eat. Then a breakdown of hybrid seed - that's a delicate pipeline that took generations to build, and can be destroyed in a single season of neglect. Hopeless to rebuild in time to save everyone. We'd be reduced to a percentage of our previous population. Assuming widespread social failure.

But many if not most cereal grain farms (plus legumes besides soybeans?), the ones that count for getting the most calories to the most people, are quite some distance from big cities. And if you only have to deal with an initial wave, you can do a Maximum Effort to keep them out of your fields, it's long term banditry where farmers by themselves inevitably lose because they can't do both at the same time.

Hybrid seeds aren't used for all crops, not for wheat as I recall, I'm not sure about soybeans. Corn is of course the big one, but at least going forward humans wouldn't particularity want to eat corn if wheat is available.

But I agree with your bottom line, on the problem of big cities, well, Dean Ing, in the appropriately titled novel Systemic Shock, put it this way:

The American public had by turns ignored and ridiculed its cassandras: city planners, ecologists, demographers, socialists, immigrants, who had all warned against our increasing tendency to crowd into our cities. Social stress, failure of essential services, and warfare were only a few of the spectres we had granted only a passing glance. We had always found some solution to our problems, though: often at the last moment. Firmly anchored in most Americans was the tacit certainty that, even to the problem of nuclear war against population centers, there must be a uniquely American solution; we would find it.

The solution was sudden death. A hundred million Americans found it.

I have a hard time believing that people would sit and wait for death because the lights were out. If they had to walk 100 miles, till with their hands, garden with whatever fibrous, undomesticated vegetables the could find, and fertilize with night soil, they'd make it happen.

Down for a year? I can't really see that happening outside of a nuclear winter.

The triggering event is a high altitude nuclear explosion designed for a maximum EMP pulse. I think China and Russia (and a few of our close allies) are the only actors who have this capability and I think there is an astronomically low chance of them doing this.

Still, spending money hardening our infrastructure would improve general reliability, help prevent deaths do to storm outages, etc. Good thing to do.

This is one of the big worries about Iran's nuclear program. 3 IRBMs, which they have, lofting 3 warheads (you need that many to get fairly complete coverage of the CONUS), they might be able to pull it off.

But it would be a terrible gamble for them, and most any scenario besides total failure results in the country being turned into the proverbial "fused glass parking lot".

Everything that has an off switch has an on switch. Attacks against the power grid are weak because, in almost every scenario you can conjure, the remediation is just disconnecting the network and flipping the on switch.

The things to look for are asymmetries. What is something that is very easy to do but very hard to un-do? What is a very important thing that would impact lots of people that is very easy to do or set in motion, but much harder to un-do or stop?

Look for those and you'll find actual problems. There are far fewer of them and they are difficult (requiring creativity and domain expertise) to find, which is why no one talks about them...

> Everything that has an off switch has an on switch.

Not really. The ultimate off switch for every appliance is that appliance burning out. And quite a lot of dangers to the power grid include, or would cause, physical destruction of key infrastructure elements.

that scenario is covered in the second paragraph ;)

Yes. The biggest risk is of process controllers being reconfigured to cause physical damage. For instance, gas turbines can be cranked up to speeds & temperatures at which the turbine blades disintegrate. Or you can shut off combustion but keep fuel flowing, then restart combustion and get an explosion in the exhaust system.

Why not just design equipment to have analog fail-safes? To get the last few percent of efficiency out of many systems you have to push them beyond what can be simply protected with fuses and analog shutoffs, and depend on software.

Nuclear reactors can't just be turned off. You have to keep the cooling pumps running or the core overheats.

Solar thermal farms can be hacked to point all the mirrors wrong and melt the tower.

Large windmills have very complicated wind speed/blade angle controls, and can be made to disintegrate in minutes with the right inputs.

All the effects I mentioned are local to one system. It's not clear if there's an attack that depends on the dynamics of the overall grid. But I can understand why researchers focus on the largest possible abstraction, rather than writing detailed engineering papers about turbine failure modes.

Actually it depends heavily on the design of the reactor. I recall that more modern designs have either far higher or complete degrees of 'passive' safety. The very heat of the reactor being sufficient to circulate the cooling loop at a safe level.

Really we should be planning for the active replacement of //ALL// infrastructure within it's designed service lifetime or less (since invariably it'll take longer than initially estimated to accomplish).

Modern designs do that, but paradoxically it's the political environment of anti-nuclear fearmongering that makes it impossible to switch to them.

Restarting a down grid is nontrivial. Small power stations are used to kickstart the larger stations. Think of the starter motor in a car. Now do all of this while precisely maintaining 60 Hz (in the US) everywhere.

This is called a "black start". [1] Definitely nontrivial.

[1] https://en.wikipedia.org/wiki/Black_start

Can they still do it the old fashioned way with a telephone to coordinate and an oscilloscope to line up the phase?

Back in 2014-ish, when Eskom in South Africa was having particular problems with blackouts, they discussed this possibility. Some estimates on restart times were given at the time, but this is the only reference I could find now: https://africacheck.org/reports/does-south-african-face-an-e...

> Everything that has an off switch has an on switch.

That is very naive. You can say that only because real and well organized attacks are very very rare.

Power Grid on its own is very complex system, and it is not trivial to revive it after blackout.

And second paragraph does not save it. Weak spots are everywhere. A few guys with rented trucks could cause blackout in several counties for weeks.

> not trivial to revive it after blackout

and yet, we do it on the regular.

edit:

> real and well organized attacks are very very rare

and why is that? if the system is as vulnerable as they say and the world is as full of enemies as they say then why is it not constantly going down?

> and yet, we do it on the regular.

No, we don't. When the whole grid is down the power stations have to shut down - they can't run without load. Power stations require power to start up, and without power on the lines there's no synchronisation.

There are facilities designed to start a grid, the utilities are prepared, but it generally takes 6-8 hours to get synchronisation to a level where the main generators can get back online.

As most major generators are a long way from population centres (and their primary load) a coordinated attack could partition most country's grids in a way that ensures a long term blackout.

As it turns out, the world is not as full of enemies as they say. Additionally, most symmetric actors generally also like to avoid their own downfall.

> What is a very important thing that would impact lots of people that is very easy to do or set in motion, but much harder to un-do or stop?

War. WW1 was started by a student with a pistol and resulted in >30m people killed or wounded.

The conflict that started on 9/11 fifteen years ago is still ongoing and has cost over a trillion dollars and a million lives, having destabilised Afghanistan, Iraq, Libya, Egypt and Syria. Turkey may be next.

Terrorism suceeds when it provokes a spiral of escalation. Perhaps a grid collapse is fine - everyone goes on holiday for a day and hangs out with their neighbour, NYC passim, but a grid collapse plus a more conventional terrorist attack?

WW1 was inevitable. The incident that triggered it was irrelevant.

That's pretty much the case that's being made of a widespread grid failure.

> Attacks against the power grid are weak because, in almost every scenario you can conjure, the remediation is just disconnecting the network and flipping the on switch.

In two of the last four winters, I've lost power for extended periods of 4 and 5 days, respectively. It's not fun, and even with friends, family and businesses in the area that did still have power that we could depend on, it was difficult. These types of issues are not flip the switch to reset, there is physical damage that needs to be repaired. Imagine this on a much larger scale and it is pretty scary.

If you've never watched the document Zero Days [1], I recommend it. Towards the end, they mention (and the documentary pulls from quotes and testimony of many intelligence personnel) that Olympic Games (Stuxnet) is just the tip of the iceberg, and that we essentially own Iran top to bottom. We completely own their power grid and could destroy strategic pieces of it in the same way we destroyed the centrifuges..using only code. They state that with something like this it could take weeks or months to fully get power restored. Considering the fact that water plants, refrigeration, businesses, etc. depend on that power, access to clean food and water would likely be severely limited, probably also leading to looting, riots and violence...a not-insignificant number of innocent civilians would certainly die.

[1] http://www.imdb.com/title/tt5446858/

I think the point is that a single high altitude EMP burst would do so much physical damage that the grids would be offline for for a long time.

The US has actually has three independent power grids: East, West, and unsurprisingly, Texas.

Some would say the US has two grids and Texas had one.

This is coincidentally similar to the situation in Canada with Quebec on their own power grid

With all the money we spend on "homeland security", there really should be a reserve supply of large transformers, just in case.

Ideally Autobots.

A lot of power plants do, at least the ones I've worked at. I wouldn't be surprised if these substations don't. Inspection teams come in all the time and practically force companies to do things like obtaining a spare.

I work for a grid operator here in the US.

There's certainly a lot of FUD in that article, but it gets one thing right: critters in the equipment are a major source of failure.

Last year we had a power glitch at our office when a snake climbed up a power pole and shorted across two phases. The snake didn't survive and folks learned to make sure their desktops are plugged into the outlets served by the building UPS.

Today I learned that my dog is doing her part to improve National Security.

Key lesson is that if you're the Russians, train squirrels to attack the power grid, deploy them, and then deny, deny, deny.

In all seriousness, most of the squirrels are already working for the Russians.

Reply on news.ycombinator.com