Power Outages Coincide in LA, New York, and San Francisco | Hacker News Reader

Power Outages Coincide in LA, New York, and San Francisco

inverse.com

91 pointsblawson71 comments

LA's outages are widely scattered, and are being blamed on high winds.[1] Crews are fixing distribution lines and transformers. Probably unrelated.

The NYC subway outages may have originated at two locations. The MTA says they lost signal power at 42nd/Grand Central, and utility power from Con Ed at 7th Avenue. Trains can run through the 7th Avenue station but the station is out of action. (The NYC subway system has three separate power systems - traction, signal, and utility.) No info on cause yet, but the problems are mostly fixed or bypassed now.

SF's outage is still vague, but is being blamed on a fire at the big substation on Larkin. SFFD says they've been using large amounts of CO2 to put out smoldering insulation. SFFD also says they're caught up after getting 20 people out of stuck elevators. PG&E isn't saying much.[2] They list "cause" as "unknown". They probably can't even get into the Larkin substation yet.

[1] https://www.sce.com/wps/portal/home/outage-center/check-outa... [2] https://m.pge.com/#outages

Hmmmm...I'm trying to verify the outage in LA. The tweet about the power outage at LAX is from today, but the other tweet in the example is from 2/22/17. No articles about a power outage in LA are popping up on google. Something about this article feels a bit click-baity.

Seems like something is going on in LA: https://m.sce.com/outage/OutageMap.html

Unless a bunch of outages have cleared up, looks pretty typical to me. The biggest outage I'm seeing on that map is labelled as planned maintenance, affecting 66 people. Nothing major on the LADWP map either.

It could be worse, from the Wikipedia page of notable power outages (yes there is such a page):

"On 7 June (2016), about 100% of Kenya went without power for over 4 hours. The nationwide blackout was caused after a rogue monkey got into a power station and triggered a nationwide blackout. However, it is notable that only about 1 million citizens were affected by the outage as the World Bank estimates that only 23% of the country's population have access to electricity."

https://en.wikipedia.org/wiki/List_of_major_power_outages

I am not seeing any of these power outages being due to crazy al-qaeda terrorists or any of the other bad guys our government clowns get excitable about. It's the rogue monkeys and Enrons I fear.

Let's not forget Cybersquirrel1. #cyberwar4ever

http://cybersquirrel1.com/

Search "power outages" in Google news and scroll through a few days worth. They happen all the time, everywhere. I would be very surprised if there's any correlation across events here.

The outage in SF shutdown the company I work at for most of the day. I work remote so was unaffected but everyone else ended up being unable to work.

This was an outage of one substation for less than 12 hours. Now imagine an 8.0+ earthquake anywhere in the bay area. The whole city could be shutdown for prolonged periods.

You don't put all your servers in one datacenter so why would you put all of your employees in one building? I am not sure why companies don't encourage at least 50% of their employees to be remote. Seems like basic DR planning to me.

Risk.

The seeming reluctance to spend lots of money on infrastructure maintenance and modernization really worries me, especially when I read these kinds of stories.

Isn't this a self-correcting problem? The grid will become more robust when there is an economic need to do so.

Alas, the "economic need" will be immediate when the outage happens, and the damage will be substantial because it hasn't been updated yet. Possibly expensive enough to make an update unlikely.

The "move fast and break things" attitude does not work for infrastructure. And no, the market will not magically correct all things. Major infrastructure problems are expensive enough that damage to them causes feedback loops.

I think the chance that it leads to lots of quick fixes rather than a redesign for the new demands and taking a long-term view is considerable.

Only if there's an immediate feedback-loop. (As in a AI-run dictatorship.) In practise this kind of feedback can take up to a decade to cause actual implementation in most democracies, I would guess.

I lived in NYC during the 2001 blackout. It was the first power outage since 1977. The lights don't go out in NYC...it's a really big deal when they do.

you mean 2003

You're right, it was 2003. I was only 21 when I moved there and it was all a bit of a blur.

Meanwhile, if you read the subway service advisories, there's always some sort of issue causing delays. I saw the notifications about the power outages today and thought absolutely nothing of it. If the entire system went offline, that's kind of a big deal. But it's one station without lighting.

I was once informed by someone working at a major energy company that if the utility frequency dropped to a certain specific value it could cause uncontrollable systemwide cascading blackouts. I don't remember the value, but I wish I did.

A large outage like SF, if it wasn't done in a controlled manner, could be grid overload. Or to prevent grid overload, utilities might need to shut down a regular on a high voltage supply line, which causes an overload somewhere else in the system, and automatic cutoff switches force a blackout.

Similar events have happened over the years starting in a remote state and ending in a California blackout. Often it's due to aging infrastructure which goes down unexpectedly, which then causes a chain reaction that hits other parts of the grid. If winds took down power lines in LA, this could cause a backup which could take out SF. It's possible that either LA or New York could have caused the SF blackout, or vice versa. If one part of the grid starts to go down it puts pressure on the rest, and things start to pop.

Note: I'm talking out my ass here, but it's based on news reports of previous outages.

The power grid does not work like that. See this tutorial, "PJM 101" for background.[1] All of today's events are in distribution, not the HV transmission network. The NYC event involves tiny loads by grid standards. PJM isn't reporting any major emergencies today. Neither is CAISO.

[1] http://pjm.com/Globals/Training/Courses/ol-pjm-101.aspx

Could be another warning shot from someone - 2013 silicon valley substation sniper attack http://money.cnn.com/2015/10/16/technology/sniper-power-grid...

North Korean retaliation for us hacking their missile launch last week?

North Korea doesn't exactly have a large pool of technically skilled people available.

What makes you think that?

It's a country where only top level officials have access to the internet, most places don't have working electricity, and there's virtually no interaction with the rest of the world.

Hasn't there been increased electromagnetic activity on the surface of the sun throughout the week?

http://www.solarham.net/older.htm

Is the interference causing unexpected surges in old, unshielded equipment and infrastructure?

Anyone else read ted koppel's book?

Yea, came here to post about it. Was a riveting read with some great interviews. It being Ted Koppel, he got access to some really high placed people. [1]

1: https://www.nytimes.com/2015/11/22/books/review/lights-out-b...

One thing I liked about it was how he dove into how the incentives and limits on federal authority combine to make it very hard to get the power industries more tightly regulated wrt security/resilience

Crazy conspiracy theory: the Chinese and/or Russians have most likely already compromised our power grid and other infrastructure control systems and are waging covert economical war by disrupting important services.

That's a wild theory, but I can't help feeling this way either...

The biggest part of that economic war would be letting everyone know the grid has been compromised, not the actual outage, there's no advantage to keeping it secret.

There's a benefit to keeping it out of public channels if it's a warning rather than an economic war.

I'd argue that taking out the power in three major economic centers at the same time would be the opposite of keeping it secret. But I completely agree with your reasoning.

Capability demonstration

But the downside to making it public is you'll galvanize the victim nation to start taking computer security more seriously, and your window of opportunity will start to close. It would make sense to try to "boil the frog" and gradually and slowly increase the frequency and severity so that by the time you "find" the threshold that snaps the victim nation out of their slumber, you've gotten away with a lot of damage cumulatively.

It's a valid hypothesis and not crazy at all. If this were random I'd expect totally random cities, not the three most economically significant.

It might not be covert economic warfare, but more of a covert demonstration whose audience is intended to be the US government. "We have lots of dangerous 0-days, and there's a lot more where these came from." In that case public credit would never be claimed.

It reminds me of how GitHub kept getting DDOSed for no reason. Turns out GitHub was a popular demo target for vendors of DDOS botnets to show their power.

Edit: could be exactly that... a demo of wares by a black market vendor of 0-days for a potential buyer.

Edit #1: a non-malicious explanation might be a vendor pushing out a bad software update to something.

"If this were random I'd expect totally random cities, not the three most economically significant."

I would be very surprised if there weren't at least 3 power outages in random cities on a daily basis, you just don't hear about it until it's 3 major cities.

That is also possible. I can't find any comprehensive history of blackouts anywhere so it's hard to check probabilities.

There's this: http://insideenergy.org/2014/08/18/data-explore-15-years-of-...

Which includes a link to this Google spreadsheet: https://docs.google.com/spreadsheets/d/1AdxhulfM9jeqviIZihuO...

Great find.

At a glance it looks like 1-2 dozen per month, so a coincidence is definitely plausible.

Have to wait and see.

You could argue that none of the ones today are weather related. If you remove the weather related outages, it's easier to support the notion of a conspiracy. I don't personally think this is one, but...

This is along the lines of my thinking... I think before jumping to conclusions, it would be worth digging up some semi-current data on frequencies of outages. Not sure how conveniently/centrally that data is available, however.

> I would be very surprised if there weren't at least 3 power outages in random cities on a daily basis, you just don't hear about it until it's 3 major cities

I wouldn't even be surprised if it regularly were 3 major cities at about the same time and it's just that statistically someone had to end up taking notice and trying to make sense out of it when it's actually a somehow random† distribution of events, the latest of which happening to be clustered.

But digital warfare seems to somehow look more exciting to many, probably because it's intuitively less frightening than waving nukes at each other's nose, while it may very well impact people's lives for the worst in a very large, non-obvious and pernicious way.

† Some events may be random but aren't independent across because of interconnections and dispatching woes[0]. Managing a power grid or three is a fascinating thing in itself.

[0] https://en.wikipedia.org/wiki/2006_European_blackout

Could be a digital weapons test.

not the three most economically significant

I don't think SF ranks in the top 3, I'm almost certain that Chicago would rank higher in economic significance, and I'd guess that Houston is above SF, probably Boston too.

Despite SF being so popular in the technology space, it's really quite a small city -- only around 800K people, compared to nearly 4M in LA, and 2.5M in Chicago.

EDIT: I found a ranking from 2010 and SF is #9

https://www.theatlantic.com/business/archive/2012/06/america...

New York LA Chicago Wash DC Houston Dallas Philadelphia Boston SF

Not likely because of Hanlon's razor: "don't attribute to malice that which can adequately be explained by stupidity".

[1] https://en.wikipedia.org/wiki/Hanlon%27s_razor

This takes precedence: "Once is an accident. Twice is a coincidence. Three times is an enemy action"

https://en.wikipedia.org/wiki/The_Moscow_rules

The rule you quoted is not on the list there.

I think you meant to link to this [1].

[1] https://en.wikipedia.org/wiki/Goldfinger_(novel)

Like with birthday paradox, denominator is important. Considering how many large cities and days there are, this might not be an unusual coincidence.

Hanlon's Razor is not an established law of reasoning or of philosophy. Quoting it is not a valid counter argument without additional reasoning given.

That reminds me of this cartoon:

http://i.imgur.com/aI80YDO.jpg

Of course not, it's a colloquial heuristic. Do you disagree with this comment though? https://news.ycombinator.com/item?id=14168483

No. But it does not apply in this case. It may turn out to be coincidence or stupidity but claiming that it is likely to be thus because of Hanlons Razor is wrong and bad thinking.

That's why I said "not likely" rather than "impossible". It's just conservative Bayesian prior.

Remember last time nefarious hackers infecting computers at power plants were in the news, that later turned out to be antivirus finding some spam on completely unrelated computer?

It's hard to imagine you really understand Bayesian statistics when you mention what happened the last time as support for your point. It is quite hard to believe that you know with reasonable degree of accuracy the probabilities involved.

Ok I'll bite. Let's back of the envelope the probability.

In the past 17 years I experienced 5 blackouts, so return period is about 1,241 days. Among the cities that were mentioned, San Francisco is the smallest, with population about 865,000. Let's look at all US cities with population above 500,000. There are 34 of those.

The probability of having simultaneous blackout in 3 of 34 cities during 1,241 days is 1-exp(-C(34,3)/1241^2)=0.4%, or about 2% for the same 17 year period.

The above calculation assumes independence, but in reality probability is much higher because independence assumption rarely holds in real life. For example, we already observed multiple large cities in the Northeast having simultaneous blackouts in 2003.

At best Hanlon's razor is a bet on the odds that most cases are stupidity and not malice (probably true). It doesn't mean there is no malice in the world and only stupidity.

Can it be called crazy if exactly that has happened in the recent past? Russia did it to the Ukraine just a few months ago.

Crazy reality here: SF was caused by a substation fire.

http://sanfrancisco.cbslocal.com/2017/04/21/power-outage-shu...

A substation fire isn't a root cause. What caused the fire?

Possibilities I know of would mostly require failure of safety measures, but would include such things as hapless squirrels & birds, problems elsewhere in the system, and yes even remote operation of substation equipment by humans in dangerous ways. (Including both incompetence and malice, though in the latter case you would have to assume the safety measures didn't fail but were overridden.)

Addendum: I'm not coming out full force in favor of a conspiracy theory, I'm coming out full force in favor of keeping an open mind, not getting overly comfortable in one's beliefs, and remembering one knows a lot less than one thinks one knows.

It'd be pretty easy to trigger a substation fire by feeding lower values into the voltage sensors than how much was actually flowing. Similar to how Stuxnet faked the values it was overriding so that everything seemed normal when in reality the centrifuges were spinning out of control. As another commenter pointed out, it's just keeping an open mind...

Could you explain this further? I'm not sure how innacurate voltage readings would cause a fire...

I believe the idea is, you make the monitoring values read too low (lower than the actual voltage in the wires), so the automatic control system adjusts by making the voltage higher. If you can get it to make the real voltage higher than the equipment can safely handle, you may get a fire.

I'll take a stab at it. I'm not an electrical engineer but I do have some experience with industrial control systems.

From Wikipedia: "Generally substations are unattended, relying on SCADA for remote supervision and control." These SCADA systems may or may not be connected to Internet, which could allow an attacker to remotely access and modify the code that controls transformers and other electrical equipment.

In another comment, someone mentioned the power company was, in this case, pumping C02 into the substation in order to contain smoldering electrical insulation. This means that, most likely, the copper conduit heat up beyond defined tolerances. This could be due to more current being carried than those conduits are rated for. Normally, the SCADA system would be responsible for keeping these currents within tolerances. What I am saying is that they could deliver a payload to the PLCs via the SCADA that could trick the transformers into taking more load than they could handle.

This recent USA Today article claims our grid is attacked every three to four days:

https://www.usatoday.com/story/news/2015/03/24/power-grid-ph...

As mentioned in another comment, Ted Koppel's book "Lights Out" is a really fascinating read on the threat.

I have a friend with a very important deliverable due today and he was not ready. I'm not saying he's responsible, but in the realm of things that make you go hmm...

Interestingly, in India this would have close to zero impact. Because of the rate of industrial growth combined with a massive shortfall of energy - almost everyone in urban India has private diesel-fueled electrical generators (yes - including individual homes and small startups).

Plus as a cultural thing, Indians have designed power failure into their daily lives. Interestingly trivia - the most popular Nokia phone in India was the one that had a dedicated torch.

Coincidence.

It's impressive how the article is reporting "as of 1pm" in SF, at 12:30 PDT...

Reply on news.ycombinator.com