Hacker News Reader

Comment by doug1001 | Hacker News Reader

Ted Koppel (retired new anchor) began research for a book on this subject after retiring; the book, titled "Lights Out" was published late last year, i believe. Two things about this book were remarkable to me--one was the exhaustive research; the second was a remark from Koppel in the book (i don't recall the exact words he used nor where in the book it is) that it is difficult to imagine a better design for a Power Grid whose primary criterion was maximum exposure to catastrophic attack than the US Power Grid (used collectively to refer to three geographic grids).

View on HN

The critical piece of evidence presented in his book was the blackout of 2003. Ted Koppel's version was short and implied that the power grid system is so vulnerable that all it took was some overgrown trees to bring the northeast down to its knees for days. He then spent rest of the book talking about SCADA glitches and computing vulnerabilities and disaster preparations (focusing namely on the Mormon community, which is always prepared for a disaster.)

Problem I had with the book is that he either misunderstood or misrepresented that triggering event. Yes, there were overgrown trees, high summer demand and temperatures that made power lines heat up and stretch. Wind speed dropped from 5 to 2 knots and cooled the line less than it normally did in that place. Cables stretched more than usual and grounded. These are all true facts.

However, that's not what brought the power grid down that day. Here's what did.

At 12:40 pm, an engineer working for MISO (transmission company, sort of like a traffic regulator) switched a regulation system to manual mode to make a correction. Note that the correction was made because the system had an errant piece of information regularly enough, which through a phone call the operator knew was false and corrected.

The regulation system was called a "state estimator." It basically takes tons of factors, loads, schedules, downtimes, capacities, etc and optimizes power transmission. Like intelligent traffic control. Of course, all systems have input garbage from time to time, and he corrected this thing.

At 1:30 PM that engineer went to lunch, forgetting to re-enable the automated mode.

At 2:02 PM, a 345k-volt line tripped after touching a tree and started this cascading reaction resulting in a system blackout for a week. Other operators looking at the system readouts didn't know that the state estimator was now sending a false picture, being frozen in manual mode.

By the time the unnamed engineer (from the NERC report) returned, it was too late. Other regulators, operators and power plants had been making uninformed decisions, preparations and adjustments for the power grid based on outdated, non-realtime information. However, the blackout didn't happen instantly -- it took four hours for a cascading chain reaction to trip every safety device in the system.

Lets not even go into SCADA vulnerabilities. Here's some positive outcomes from the blackout: investigators made a staggering list of 46 recommendations -- none addressing tree trimming because vegetation events happened all the time. Those were routine. They did focus on computer security, even though there were no computer faults in this incident.

Secondly, during the blackout the Department of Energy ordered state of Connecticut to energize one of their power plants despite their objections. They complied, and that restored power to metropolitan areas. This action is now a precedent for any future outage events: the legal framework for prioritizing power restoration exists and is in place.

SCADA systems show up on HN often enough to be a point of ridicule. Good news is that the outdated SCADA systems are getting replaced with synchrophasors for the most critical power integration functions. Since 2003, nearly 1700 phasors were added to the national grid, and they're well on their way to be fitted in the roughly 7,000 power plants in the U.S. What it will do over SCADA is improve recovery times of an unstable power grid segment.

Edit: derp, northeast, not nw. Also http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/...

needless to say, i lack the domain knowledge to critically evaluate the conclusions in Koppel's book, so always welcome to hear such evaluation from someone who does.

The "synchrophasor" is interesting and caused me to think of the grid as a giant PLL: better detection of phase discrepancy allows the control system to improve the phase margin.

(grid blackout == PLL loss of lock, too)

I just checked Koppel's book out of the library and finished it. Great book and it pisses me off that we spend so much money on corporate welfare, human welfare, and wars overseas, but we can't spend much less money to harden the power grid.

I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.

> I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.

Our food production, waste and water treatment, hospitals, etc run on electricity, sure. But 80% doesn't seem anchored in reality.

I feel like this irrationality about the risks posed by a blackout is what the article is meant to address.

I think a US Army report had the figure higher than 80%.

A small thought experiment: imagine that you live in a city and for 1 year there was no electricity, no trash pickup, likely no water to drink and to flush toilets, and no gas for cars and trucks.

I think this scenario is very unlikely, but it is still worthwhile hardening this infrastructure.

I admit to being cautious. If it were up to me, I would also spend a lot of resources hardening computing systems: servers in business, kids' school laptops, home computers, IoT, network infrastructure, etc. would have maximum crypto, and we would have a "going to the moon" level effort pushing those aspects of computer science for general security issues.

A year of blackout? Yeah, 80% sounds about right. See, our modern cities are designed around efficient transport (food, sanitation). Back in the day they added sewers to London because it took longer than a day for the nightsoil wagons to get to the dump sites and back - something more efficient was needed.

Today its 1000X that. There's no way to get food to people without efficient transport, which depends utterly on energy. No way to get sewage out. No way to get water flowing. Heck even the traffic lights are needed to keep transport going.

A year without food and water? Without sanitation? Of course nearly everybody dies.

Manual solutions could be found, but its impossible that they could support the current population density. So folks would have to flee the cities. To the countryside, where they just die a different death. After pillaging and destroying any remaining food pipeline.

Food is pretty easy, from my readings of nuclear war survival during the Cold War. E.g. a single semi-tractor trailer, long before their allowed weight was upped to 80,000 pounds, could carry enough wheat to keep 100,000 people fed for one day.

It's the water supply that would be the big killer, you can't go without it for more than a few days, and that just wouldn't be enough time to get enough to everyone.

I'm not sure they'd be so successful in "pillaging and destroying any remaining food pipeline" ... how do you to that to an interstate highway? At most, to the fueling stations, but those are a fairly small scale, and I would expect easily replaced thing. But that would also help to self-limited the radius of the pillaging and destruction, keep it mostly away from the farms themselves, which aren't exactly defenseless if such is needed for only a relatively short period of time.

Its the farms that are vulnerable. Hungry thousands trampling crops looking for anything to eat. Then a breakdown of hybrid seed - that's a delicate pipeline that took generations to build, and can be destroyed in a single season of neglect. Hopeless to rebuild in time to save everyone. We'd be reduced to a percentage of our previous population. Assuming widespread social failure.

But many if not most cereal grain farms (plus legumes besides soybeans?), the ones that count for getting the most calories to the most people, are quite some distance from big cities. And if you only have to deal with an initial wave, you can do a Maximum Effort to keep them out of your fields, it's long term banditry where farmers by themselves inevitably lose because they can't do both at the same time.

Hybrid seeds aren't used for all crops, not for wheat as I recall, I'm not sure about soybeans. Corn is of course the big one, but at least going forward humans wouldn't particularity want to eat corn if wheat is available.

But I agree with your bottom line, on the problem of big cities, well, Dean Ing, in the appropriately titled novel Systemic Shock, put it this way:

The American public had by turns ignored and ridiculed its cassandras: city planners, ecologists, demographers, socialists, immigrants, who had all warned against our increasing tendency to crowd into our cities. Social stress, failure of essential services, and warfare were only a few of the spectres we had granted only a passing glance. We had always found some solution to our problems, though: often at the last moment. Firmly anchored in most Americans was the tacit certainty that, even to the problem of nuclear war against population centers, there must be a uniquely American solution; we would find it.

The solution was sudden death. A hundred million Americans found it.

I have a hard time believing that people would sit and wait for death because the lights were out. If they had to walk 100 miles, till with their hands, garden with whatever fibrous, undomesticated vegetables the could find, and fertilize with night soil, they'd make it happen.

Down for a year? I can't really see that happening outside of a nuclear winter.

The triggering event is a high altitude nuclear explosion designed for a maximum EMP pulse. I think China and Russia (and a few of our close allies) are the only actors who have this capability and I think there is an astronomically low chance of them doing this.

Still, spending money hardening our infrastructure would improve general reliability, help prevent deaths do to storm outages, etc. Good thing to do.

This is one of the big worries about Iran's nuclear program. 3 IRBMs, which they have, lofting 3 warheads (you need that many to get fairly complete coverage of the CONUS), they might be able to pull it off.

But it would be a terrible gamble for them, and most any scenario besides total failure results in the country being turned into the proverbial "fused glass parking lot".

Thanks for the recommendation and insights on that book.

Reply on news.ycombinator.com