Hacker News Reader

I attended a lecture about this a few years back. NOAA has a satellite system to warn us about large solar flares... but it's old and severely underfunded. Which is somewhat surprising, because that 20 minute heads up is the difference between GPS satellites and electric grid components being safely taken offline vs permanently damaged.

Yeah, when I started getting involved in web operations tools, and realizing all the levels of redundancy that the big players like Google and Netflix have, it really got me thinking about our hydro and communications infrasture.

For instance, Netflix has it's chaos monkey, than randomly kills servers and availability zones and generally wreaks havoc. They set it loose, and learn to keep things operating. I really thinkg it would be a responsible societal effort to introduce chaos monkeys into our archaic infrastructure outside digital.

And as for that underfunded satellite, it's sitting in the sweet spot between the gravitation tug of the earth and sun called the L1 langrangian point, where it takes no effort to keep it exactly in place. But we really need more drills to learned whether the hydro/comm infrastructure on one hemisphere of the earth can actually shut everything down in 20 minutes, day or night.

The 100-year old underground electric wires running under the street where I work have caught on fire three times in the past year. Something tells me our archaic infrastructure has quite a backlog of existing chaos monkeys to deal with already.

The spirit of the idea is good, but it would not work well for hardware. With software, if something breaks, you restart the servers. With hardware, if something breaks, you have to buy a new one. Throwing chaos monkeys to see which transformers blow up would end up quite expensive. :)

Haha, was thinking the same thing. This would make a great xkcd.

Then at least simulations on how it affects electric grids coyld help quite a bit. Are this simulations taking place?

I interned at Areva T&D in 2006 (now Alstom Grid). While I was not working on that specific project, I remember they had a very powerful EMS (Energy Management System), which let them simulate incidents (e.g. "what happens if these power lines are destroyed").

Doing a quick search, this seems to be called a "dispatcher training simulator": http://en.wikipedia.org/wiki/Dispatcher_training_simulator

http://en.wikipedia.org/wiki/Energy_management_system

http://www.alstom.com/Global/Grid/Resources/Documents/Automa...

There are a lot of modern facilities providing solar flare predictions, many are old and many are new. Some are space based and a lot are ground based (radio telescopes and coronagraphs) Government agencies amalgamate data from all the sources to come up with the best predictions. We always have more than 12 hours to prepare after spotting something from surface on the Sun, the accuracy of predictions improves as time goes by, the 20 minute mark is generally the time it takes for a fast CME to go from L1 Lagrange point to Earth. L1 is where multiple aging solar observatories collect insitu data, but we usually know before then if a strong event is Earth directed.

What could we do to satellites in 20 minutes that would protect them from a coronal mass ejection? It doesn't seem to me that simply powering them down would protect them.

We can't save satellites but when those begin to fail, we have 20 minutes to power down things before earth itself is hit.

You can save satellites - you can order them to rotate instrumentation away from the flare vector, and you can shutdown sensitive electronics or high-gain radios and the like (most satellites have many levels of shutdown for exactly this reason).

under adverse space weather conditions, launch personnel may delay a launch, and satellite operators may postpone certain operations (e.g., thruster firings). For the spacecraft industry, however, the primary approach to mitigating the effects of space weather is to design satellites to operate under extreme environmental conditions to the maximum extent possible within cost and resource constraints.

http://www.nap.edu/openbook.php?record_id=12507

The relevant department within NOAA is the space weather prediction center, http://www.swpc.noaa.gov (beta site: http://origin-www.swpc.noaa.gov).

I toured their facilities in Boulder, CO a few years ago (I develop a satellite data product, http://link.springer.com/article/10.1007%2Fs11207-014-0529-3, that is used in forecasting). The forecasting is done partly with numerical models, partly with observational data, and partly with old school intuition and experience. It partly involves a small group of grizzled experts sitting in front of a bunch of monitors displaying images and time series.

They are trying to automate it as much as possible and bring in new data sources, especially near real time data sources, from NASA and NOAA satellites. It's too much to go into in the scope of a comment, but suffice it to say that, with as many photons as the Sun puts out, you can gather a lot of information about the entire 3D solar atmosphere and magnetic environment. One day, these measurements would feed a forward model of the same type used in terrestrial weather prediction.

> Can you imagine how disastrous this would be? People would likely wake up without power, and think little of it at first. "Oh, it's a power outage. Better light some candles and wait it out." Then hours, days would pass. There would be no internet, television, or even newspapers to report what happened. We would be reduced to hearing the news from Shakespearian harriers!

> I wonder if this would cause mass revolt. What happens when you completely disable modern technology on all of earth? If this is as likely as they say it is, there needs to be some serious research into mitigation strategies not only of the immediate technological effects, but also of the sociological effects.

I really doubt it would be that terrible. It's not like electricity would have completely disappeared for ever (like in this great novel apparently called in English "Ashes, Ashes" [0]).

Important things and urgent things like communication means would quickly be operated by emergency generators, so people would immediately have information through radio (and most of the receptors are battery operated), and probably also TV, although they would probably have to go to the local townhall, hospital, school or whatever place would have a working generator to watch it.

Cars would still work so people could move around, at least while there are refined gasoline reserves, but refineries would probably be one of the first industries to get powered again anyway.

Probably for a good while there would be restrictions on electricity usage, depending on how hard existing plants would have been hit and whether they are fixable or have to be completely rebuilt, but I really don't think it would "destroy life as we know it".

I wonder how nuclear plants would handle this, though.

[0] http://en.wikipedia.org/wiki/Ravage_(novel)

> I wonder how nuclear plants would handle this, though.

Yes, could all those aged reactors deal with large scale grid or component malfunction? Fukishima didn't do too well. FUD: Would we have hundreds of Fukishimas on our hands after such event?

A power plant makes its own power, so the grid going down wouldn't affect it. For other components, nuclear reactors have many failsafes. Fukushima exhausted at least 4 that I know of. Since a CME wouldn't cause physical destruction like the earthquake and tsunami did, there's a much lower chance that all the failsafes will fail at once.

A nuclear power plant cannot run at say 10% to just power its own systems after it was separated from the grid. That's why diesel generators and batteries are so important to keep the reactor from melting.

It's hard to find a source on this, I apologize. The reason is that it only has large turbines to generate power, and these cannot be operated for arbitrary power output.

Batteries only last so long. And last time a hurricane took out power in my city, the gas stations quit working.

The biggest problem would be high-voltage transformers. They're made mostly be hand, by people who spend several years learning to do it, and we don't have many spares. If they go down continent-wide, we could be without grid power for years. I'm a little less sanguine than you about how well we'd do in that situation.

http://spectrum.ieee.org/energy/the-smarter-grid/a-perfect-s...

> _Cars would still work so people could move around.

Any car that was running at the time of the CME would have all of its electronics fried and become inoperable.

> at least while there are refined gasoline reserves

As another poster hinted at, any gas station in operation at the time of the event would have its pumping equipment fried. Even if the pumps weren't harmed, you still need electricity to run them, so you'll have to wait until the grid comes back up.

To defend against it you just need a fuse, or a relay.

Put one on each transformer that cuts power if it exceeds limits, this would save the transformer and would allow us to restore function pretty quickly.

That seems like a cheap solution that would protect not only against solar storms, but lightning strikes as well. Are these fuses common in transformers? If not, why?

I don't think they have them - just watch any transformer fire fueled by electricity.

I say "fuse or relay", but for the power levels involved these are NOT simple components.

I work in commercial electrical power. No they don't. Because they aren't required to. It's all about costs. So unless it's about known risks (read PRA), no they don't protect against it. You need to spend money where the risks happen.

Wish I had better news, but hey you protect against "real" risks, like LOOPs, trips, downed trees, lightning, regulators, and floods. Well I guess the quotes are unfair, these things happen often. So these are real things you want to mitigate. End of the world stuff, no one takes that seriously sadly.

Also, don't forget a lot of the protective relays are digital protective relays. So even if you have them, well they might not work in This case. So aside from the digital PRs being "better" companies still sell them and few sell the older SD models, so we end up replacing the old solid state relays with digital drop in ones.

So even if you have a PR it might not work in this case. And frankly few people in the industry, or with the regulators, takes a geostorm seriously in commercial power, even though they should.

Typically, your fuses at that scale aren't fuses per se, but mechanical separation of conductors.

https://www.youtube.com/watch?v=PdiJWQmSi0k

https://www.youtube.com/watch?v=e6Ftv3zt-gI

Here is a video on high voltage circuit breaker evaluation and maintenance. More then you ever wanted to know I'm sure.

https://www.youtube.com/watch?v=PKXPeTvmVQg

These systems are complicated, hard to maintain and expensive.

It's not "cheap" when you have to install a breaker for each one. For the companies it's cheaper to have a transformer blow up every now and then.

It'll only be considered an investment when the damage can out-weight the cost. So it's likely they'll only learn the lesson after the first solar storm hits, unless legislation demands they put it now.

Sounds exactly like the now canceled TV show "Revolution".

That's it. I've had it with cancelled shows. I'm no longer starting any shows until the series finale has aired.

You could build an oven in your backyard and stock up with a load of wood ;) and make friends with your local livestock dealer. Just have a plan for how you're going to still make money once everything shuts down.

Start here: https://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs and work upwards.

Hint: you'll need guns. Lots of guns.

Cities are stocked with food and water for few days, but once the supplies are gone, you'll have waves of hungry and desperate people radiating from cities to nearby farms and villages.

I think you're right that guns are probably a necessity to some extent, but I would rather spend the money on feeding my neighbors as much as possible rather than fighting them. I ran some quick numbers on webstaurantsupply.com, and for a simple "donut diet"[1] you can feed 100 people 1800 kCal/day for nearly a month for $1000 (the price of a decent quality rifle where I live).

I've been trying to work out a way to organize neighborhood-level basic essentials storages on the cheap, so that people in that kind of situation can afford to take care of themselves until the power comes back on. Food is pretty easy, but water and sanitation are a bit more problematic so far.

[1] - just calories to make it over the hump, this is not health food. Ingredients are white flour, white sugar, vegetable oil, and some white rice for the people who can't do the wheat products. Add vitamin pills to prevent deficiency diseases and you can stay functional for a long time at about $0.30/day.

the point is, money stops being worth anything in this scenario. you can't eat it and you can't drink it. money is based on trust and when civilization collapses, trust disappears.

Everybody realises that, the point he was making was that $1000 is better invested in storage now than a gun.

Everybody assumes that in an end of the world scenario that we all end up fighting for food, however the better plan, and to be honest the one that served society well in the past, was that we become farmers. Without tractors we'll have plenty of farmland that needs manual labour.

It would be nice if it was codified into law that our fallback plan was to redistribute farmland. Then there is no need for roving biker gangs fighting for the last can of tuna. We'll just step away from the now dead computers and pick up a hoe.

Except that you need to survive the months between planting the seeds and harvest. Also when 90% of crops fail because of lack of modern agriculture tech and farming experience, don't expect big part of people to just lie down and die from hunger. You will have fighting until the food supply stabilizes (which does involve cutting down population a lot).

That doesn't make sense, why would the current crops in the ground not be usable? Why would the food in storage right now not be usable?

Saying that 90% of crops would fail is kind of an over statement. Organic farms that don't use fertilisers are quite successful. The issue is planing and sowing, all of which the extra man power can assist with. Instead of a tractor you have people.

All that's needed is a plan, where farmers us the extra labour until machines can be gotten back on line. No need for a lawless society.

Bootstrapping the reeducation of enough people to effectively get them intensive-gardening in that kind of a situation would be a massive challenge, but if enough people did the feed-your-neighbors preparedness approach we might have enough time to get it done. If we were looking at a long-term scenario I think I'd want to have at least a year of grace period to get people rolling. Thinking back to all the mistakes I made starting my own farm, if my life had been depending on success I'd most likely be dead, or would have survived solely by eating my sheep...

Cuba is an interesting example of a place that had to rapidly adopt a low-power lifestyle, and they actually seem to have done it with grace. There might be a good case there to study for clues about what works. They have the advantage of not having to contend with long, cold winters though, which is a critical difference for most of North America.

Was your lack of success partly due to no seeking the knowledge of other or not being give the help?

I would imagine that secretly farmers want other local farmers to not do as well, so that there is less competition for their produce, however you'd know better than me.

>>no mention of potential mitigation strategies

The biggest risk is if you think of it on the very long term, staying on earth is like risking it all on staying on one rock. There are far too many risks, and there is little you can do mitigate them on the long term.

The long term solution is to invest money to build technologies to develop inter stellar travel and related technologies to settle humans outside earth. And that really wasn't that difficult to achieve. The problem is we haven't made serious attempts beyond the moon and mars missions. The funding is too limited, and there is no political consensus around the world to work towards these goals. We have become far too busy with wars, conflicts and other pointless spending on earth. While we could have used the money to do something like this.

>>Can you imagine how disastrous this would be?....

It can be far more disastrous than you imagine. Loss of electricity and water are just primary effects. There are many secondary effects of these scenarios. No electricity means you can't pump out fuel. Bulk of your critical production[Food/Medicine/Sanitation etc] Will come to grinding halt because of inter dependencies. Expect mass starvation, riots, loss of law and order kind of scenarios. There will likely be powerful groups controlling critical resources required to survive.

I wouldn't say its a extinction scenario. But it will come pretty close.

It would be far cheaper to build underground bunkers on Earth, or fund prevention of specific risks like this one. Building a permanently self-sufficient colony isn't just ridiculously expensive, it may not even be possible with our technology for a very long time. And it's still only a last resort. Civilization and everyone on Earth still perishes in the apocalypse.

It would take significantly more than this to wipe out humanity btw. Even the worse case scenarios, civilization still recovers eventually.