Hacker News Reader

Comment by dskrepps | Hacker News Reader

I've wondered for a while. Lots of people seem to think a Carrington Event will simply fry all small electronics and make everything stop working, but from my understanding all it'll actually do is cause power surges in long transmission lines and disrupt wireless signals. Apparently telegraph lines sparked and shocked people. So what effect will that actually have now? Will a massive power surge go through my house and destroy everything plugged in, and thus indirectly my desktop computer? Will every house on the grid catch fire, destroying all modern cities? Or will substations and transformers be destroyed but shield homes themselves from catastrophic damage? Will wildfires destroy every forest anywhere near modern infrastructure?

View on HN

DC current on long transmission lines saturates mains transformer cores, turning the high voltage primary into a short circuit at the peaks of the "unlucky" side of the AC waveform.

https://ieeexplore.ieee.org/document/9652509

> When CMEs strike the magnetic field of the Earth which provides protection against the fast-moving plasma becomes compressed and results in a varying magnetic field on the ground. This magnetic field variation generates a geoelectric field on the Earth’s surface and leads to Geomagnetically Induced Current flow through man-made technology. This GIC exhibits a very low-frequency quasi-Direct Current (DC) (less than 1 Hz) with amplitudes of 10–15 A and up to 300 A peak current for 1–2 minutes that flows along conductors and technological infrastructure. The power transformers which are connected by transmission lines are the most affected by GIC events. The GIC enters from the neutral ground point of the star-connected (wye) transformer windings and divides equally among the phases. When the GIC flows through the transformer windings, a DC magnetic flux is generated in the core, whose magnitude depends on the GIC flow magnitude. This DC flux is then superimposed on the AC flux in such a way that the asymmetrical saturation takes place in the magnetic cores of the transformers (half-cycle saturation) and increases their reactive power consumptions critically.

> During saturation phenomenon, transformers draw an extremely large asymmetrical exciting distorted current that is rich in even and odd harmonics. These harmonics can trigger the relays improperly, overheating the generators and transformer’s windings and cores, leading to unstable operations of the power system and could result in long-term damage to the system’s components. These effects may turn into catastrophic failures (i.e., permanent damage or blackouts) if they persist for a few minutes. The 1989 geomagnetic storm in North America that lead to severe economic losses is an example of such catastrophic consequence due to the effects of GIC.

Good grief. Here is a thread of currently 58 comments (off the back of a erstwhile top comment no doubt) talking about the effects of a Carrington event. And granted, this is what is mentioned in the title here. But you just need to click on the article title to see that the title continues, with "A “Miyake Event” Would Be Far Worse", and the article goes on to explain how a Miyake event is orders of magnitude worse than a Carrington event.

One may conclude that if our modern society gets hit with such an event it's likely game over, at least as far as modernity is concerned.

That be as it may - this is the second time in a few days that I experience a severe and disappointing lack of interest in the actual linked text. A downside of the karma/gamification aspect here? Is it getting worse?

It would be the equivalent of doing an EMP strike by detonating a nuke above a city, but worldwide... it would be bad, lots of casualties from lack of energy.

I do believe we are in urgent need to have some way of printing all code, including legacy of basically any consumer and industrial program, because were it to hit, even with everyone today surviving and writing in plain text what they know, we'd be lacking decades of software, that most modern day programmers would have very little idea on how to emulate, at least quickly enough.

Perhaps we should write it all to optical disk and keep a few readers in faraday cages.

The big risk is a bunch of transformers blow up, essentially taking the grid down. Since we would have to bootstrap the grid again without the advantage of the grid... this would likely set civilization back pretty far, a new dark age is far from impossible.

I read that it is fairly easy to protect equipment by grounding it. But that requires properly grounding lots of gear, and the power companies aren't willing to spend the money.

Also, the Carrington event has plenty of notice before arrives at Earth. It would be possible to shut down the grid, and maybe disconnect all the transformers. It would require a cold start, but better to shutdown for a day than forever. The problem is that this requires the power companies to practice for a shutdown and cold start.

A Carrington event has notice, but there are plenty of, rather low probability, space weather events with no notice that can do the same thing. It's best if you're not facing down the barrel of a quasar.

From my understanding it mostly effects large pieces of metal like copper lines but would be unlikely to create a significant charge in small electronics and wouldn't destroy solar panels or the vast majority of components. Any transformer pieces that are in a disassembled state prior to the event would be easy to put together and bootstrap any such event. Some locations like inside concrete buildings or under ground would remain insulated from the event as well. Also it's likely that at the equator no effect will be felt.

The thing is there are very few transformers in a disassembled state ready to go. Solar panels would be fine, but they're effectively useless without the ability to deliver power.

Covid was enough to create an international backlog of transformers, and that was with a normal rate of attrition.

Exactly this.

Lets say there were a few transformers that did survive.

The parts to make new transformers are manufactured all over the place.

So you would need to transport the 'working' transformers where they are needed to get power back to each plant that manufactures some singular component that goes into a new transformer. AND Transport them, when gas stations in-between are all also without power.

The global supply chain is complicated and spread out. There are many places with single failure points.

-> Don't have link, but isn't there an article titles something like "nobody can make a pencil anymore". Which traces out like a dozen connections for even such a simple thing.

And, don't forget, a lot of power plants need power to be re-started. They don't all have backup gas turbines to get going again.

The essay is “I, Pencil”

It’s a rollicking good read. Here it is in pdf form:

https://fee.org/media/14940/read-i-pencil.pdf

Fascinating way to lobby for the dissolution of the USPS.

I rather wasn’t expecting that.

     And, don't forget, a lot of power plants need power to be re-started. They don't all have backup gas turbines to get going again.

That made me think of this Everyday Engineering video "What Is A Black Start Of The Power Grid?"

https://www.youtube.com/watch?v=uOSnQM1Zu4w

Heh, when you start thinking about how much fuel is needed to make fuel it turns into a gnawing concern in your mind.

Then add in how little food most people have available and you see how modern civilization will end.

> Solar panels would be fine, but they're effectively useless without the ability to deliver power.

Perhaps this is an argument for making rooftop solar with storage compulsory then at least we would be able to keep the lights on. Assuming that the building could be automatically isolated quickly enough of course.

There's not nearly enough transformers in reserve to properly rebuild the grid and you can't always just drop in and replace either. New transformers are also not currently able to be produced fast, and having society collapse around you probably wouldn't help while trying to increase their production rate. If we'd lose most transformers, it could be many months or years before large regions could see their power back, enough time to cause vast damage directly and indirectly.

THIS!!

It would be relatively cheap (I've reak ~$500million) to build and store a mostly complete replacement set of transformers.

The difference it would make in a Carrington+ event (or a EMP nuclear attack) is insane. Without them, likely the grid collapses for years, along with distribution, food, etc. Millions die in the years-decades it takes to recover.

With the transformer standby inventory, critical parts of the grid could be restored in hours-days, and full recovery in a few months. The one country that prepared itself with a backup inventory would instantly be the world's superpower for years, likely generations.

Yet, because there is no immediate benefit, only stochastic, there is no political will to even start to get it done. Even a 10% inventory would make a huge difference, but...

Honestly this is one of those events where billions could possibly die.

Green revolution. The level of food we need requires nitrogen. This is broken down by natural gas into fertilizer and requires immense amount of energy and transportation. When the grid gets knocked out, you're on a countdown till you run out of gas/petrol. If you don't get enough power back up before this happens you're running into a game over situation. Once fuel runs low and food runs out, people will start killing each other in massive amounts to deal with the resource constrained world they live in. Once enough electrical engineers die then the system doesn't come back up.

Yup, it could really go down that way.

When animal populations overbreed and/or their feed under-grows, a 10% overpopulation often results in not a 10% shaving of population, but a 90% population crash, because almost every animal goes under the survival budget.

It would very much the same here.

Excellent point about the EEs (and other high-capability engineers, mechanics, technicians required to keep it all running) being in the dying population. Absolutely right - if the critical mass of available experts fails, the system never recovers. Only after society stabilizes, regrows, and re-creates the knowledge; by then, it would be about rebuilding from scratch.

In such a scenario we would just start cobbling together makeshift transformers using available materials. You just need some spools of wire, a hunk of iron, then wrap the wires around. It won't be pretty, last for 50 years, etc. But it would be enough to get electricity back on in all major and medium cities to at least some degree.

If it's really bad people will tear apart motors from clothes dryers to steal wire if needed... a jank transformer like that would be hung off the primary wires for each individual house and you wouldn't be able to run high power appliances but in a true "grid crippled" emergency people would do whatever they must.

In the immediate term those of us with solar and battery would be able to help our neighbors charge cell phones and the like. We'd also be able to charge EVs for short trips to gather supplies.

If needed people would disassemble machines for making wire and transformers then move them my horse cart or hand-pulled sled to factory buildings with electric power (or enough solar panels) to restart production.

Don't get me wrong - it would be a bad scenario - but it wouldn't destroy civilization.

Exactly this. We're in the best possible time to deal with it since everyone has a power pack with an attached solar panel and so many homes have solar now. Generators would work just fine as well. We'd be back up in a week tops.

Lmao ok mate, whatever you reckon. Keep in mind you'll need to do this without the internet to help and people not being super cooperative as they do whatever they can to get food.

Like I said: it wouldn't be a pretty situation but people aren't just going to sit back and accept the destruction of civilization and/or death. People will dig out old textbooks on electricity and figure it out.

Transformers are simple devices. The first electrical gear was a janky DIY affair by modern standards. We wouldn't wait 10 years to manufacture modern switchgear... we'd cobble together literally anything to bootstrap electrical power. It would start off as isolated islands with nearby generating capacity and where enough gear was still operating then spread from there. Major cities would be back online first, followed relatively quickly by medium cities. That might mean that only certain parts of those cities have power initially with neighbors having to share fridges as only every other block is repaired.

Small cities, towns, and rural areas would be proper fucked for a while... possibly a year or more. But before a year was up enough power would be restored to manufacturing that many of those people would install solar grid-forming and/or battery systems.

For whatever it is worth we are better positioned now than in the past. More and more people have distributed generating capacity with solar PV and grid-forming systems like Enphase IQ8 can form a microgrid just from solar energy without a battery (at reduced capacity depending on sunlight conditions). Even 10-15 years ago solar was useless without a grid to tie into so a full grid outage would be close to a 100% outage. 10% remaining may not be a large amount but its a lot more than 0 and enough to move your neighbor's fridge/freezer into your garage and keep food cold for the whole neighborhood.

(Disaster resilience is a feature of distributed solar PV, especially with batteries, that is almost never talked about)

There has been an oversupply of transformer manufacturing plants in the world for decades. No one needs the Internet to start building more. The designs already exist and most of the small manufacturers are still using manual or at most semi-automatic winding machines. For large transformers this is true even in the big manufacturers like ABB and Hitachi. Stacking the cores can easily be done by hand.

The bottleneck would most likely be the supply of core steel. Increasing demands for efficiency have pushed core steel manufacturing into a few high tech manufacturers.

Other things you haven't considered:

Fuel pumps run on electricity, so you need to get them working before you can get any materials, everywhere, or you won't be able to fuel all the ships and trucks necessary to move anything.

Phone lines (and the internet) runs on electricity so you need to get them back up to coordinate everything.

While you are trying to rebuild your transformers most of the food we have in storage will rot. Think you can fix the grid fast enough to prevent all your factory workers leaving to forage?

It's incredibly naive to imagine that this is an easy problem to solve. The amount of stuff that you have subconsciously assumed to make this scenario work is staggering.

Yes... so priority will be to find diesel generators to get refineries back online. Individual pump stations will, if needed, scavenge solar panels or generators to get their own pumps up and running.

Governments will prioritize food and electrical infrastructure, likely banning transport of anything not critical to these two categories. Trucks traveling to transformer plants (and plants converted into switchgear plants) will carry food to the plant if necessary.

I'm not trying to paint a rosy picture... that everything will be OK everywhere. Some areas will manage better than others. Some areas may well turn into riot zones as people fight each other for food. Recovery will initially be spotty and uneven.

But I'm 100% confident it will not be a semi-permanent collapse of civilization. I'm also 100% confident some metro areas (and the farming areas nearest to them) will be back up and operational within 1-2 weeks up to 1-2 months depending on local circumstances.

We are probably close enough to be able to deploy solar with storage in sufficient quantity that isolated islands would be viable to re-bootstrap.

This also has the advantage that it helps your economics during normal operation rather than just during a crisis.

are uninstalled transformers even safe?

Provided they’re not connected to lengths of conducting material, they ought to be ok.

Hard to be certain though, as transformers are more than meets the eye.

> transformers are more than meets the eye.

Really?

Transformers are very simple electrical machines. The design process is complex but the object itself is made of mild steel for the tank and supporting frameworks, special steel for the core, copper or aluminium for the windings, paper and resin for insulation, and oil for insulation and cooling.

There are a few ancillary components to do with detecting faults and switching taps and sometimes pumps to circulate the oil through radiators.

Apart from the pumps, circulating oil, and tap changer, there are no moving parts.

There are no compulsory electronic components in even the biggest transformers. Apart from improvements in material quality and the design process a power or distribution transformer built now is really not very different from those built a hundred years ago. In fact there are many transformers that were built fifty years ago that are still in service. Occasionally such transformers are repaired rather than replaced when they fail, quite often the original drawings are still available.

That’s a really clear response. I like the details and context you include.

When I was a little kid the original tv series of “the transformers” was one of the most popular children’s shows. The theme song sang, in a sort of robotic chorus: “Transformers: more than meets the eye”. I was trying to make a subtle reference to this, as an inside joke to people the exact same age as me (or people who know the Shia LaBeouf movies, if they even use that same lyric?)

Thanks. Working in the industry for over thirty years has paid off :-)

I'm much too old for the Transformers TV series, that flew straight past me I'm afraid. I was eight when the original Dr Who series started!

(It was a reference to the children’s cartoon)

Right, you have to avoid the Decepticons.

After a couple incidents where snipers shot at substations or hackers turned stuff off, there was a burst of worry about the fragility of our electrical grid. Of the large transformers that make up substations, almost all of them are unique. There is no inventory of spare transformers. Each one is custom made and takes months. The small transformers you see on poles are made by the thousand, and stocked at every utility. The small ones are not the problem. The big ones - that are the size of SUVs and houses - are the ones that the grid depends on.

There's a difference between made to order and bespoke.

Of course, most of the discourse is hand wavy speculation, vs characterization of the existing stock of transformers, like what standards they are built to and the magnitude of an event that would certainly damage them.

Most large transformers are actually unique but this has little to do with the characteristics of the transformer. It is mostly to do with optimizing the design to minimize the total lifetime cost. This requires a new design for each transformer because the relative prices of core steel and copper or aluminium varies over time. When copper is cheap the total cost will be lower if you use more copper and less core steel for instance while the opposite will be true if the copper price is high.

If cost is not as important as prompt delivery the manufacturer can often just take an existing design and repeat it.

I doubt that it's strictly true seeing how quickly Ukraine was replacing their transformers after Russia deliberately targeted them. As far as I can tell it was a strain on the global supply, but evidently not insurmountable.

Don't modern transformers have surge decoupling devices? The same thing happens with lightning and long transmission lines and isn't that why lights will flicker during storms.

Yes but not the same order of magnitude transient as we're discussing here. E.g. a fuse isn't a fuse if you force a gazillion joule transient through it. Current keep on flowing. Same with shunt protection: it can't shunt all the transient energy before it vaporizes.

Aren't transformers protected from this sort of thing nowadays?

I'm not sure if you can. Transformers are basically two differing loops of wire separated by a space. It relies on the EMF of each loop to transfer energy from one loop to the other. If a strong enough pulse comes along it may be too powerful to block. Like trying to stop lightning with a $100 UPS or any UPS to stop a millions of volts and tens of thousands of amps.

If its disconnected from a grid and grounded it shouldn't be an issue... and if the pulse is that powerful to deal with it being grounded it's very likely we've lost most of the atmosphere at that point making the entire subject moot.

If it's disconnected from the grid why would we be concerned about it? The question as I understand is about protecting active in-use equipment.

No.

This will be the most probably scenario based on this book which as based on an EMP, but regardless the same outcome - https://en.wikipedia.org/wiki/One_Second_After

Then you will probably want have - https://www.howtoinventeverything.com

Fixing the transformers would be the difficult part. Restarting a completely blacked out grid is a scenario we have plans for. Off the top of my head, I know for sure that Bonneville Dam has black start turbines, and I wouldn't be at all surprised if all major hydroelectric dams provide the same ability.

> Since we would have to bootstrap the grid again without the advantage of the grid

That sounds exactly like what happens in the game Satisfactory when your nuclear power plant got clogged and you only find out when it's too late!

Thank goodness v8 has grid priorities. Though it would be a massive pain in the ass if you had to go across the map and replace each one of those transformers if an event happened.

Sooo.... widespread rooftop solar would probably help the resilience of at least domestic power?

They would, at best, be used to power small local induction forges and if you're lucky, refrigerators. There's no way you could use them to restart the grid.

Can common rooftop solar setups actually deliver power locally with the grid being down? I would imagine them taking at least frequency synchronization from the grid, and probably also not faring too well without the grid to compensate for variable output when a cloud passes over them etc.

Most rooftop solar can't produce power without the grid. They use cheap inverters that synchronize frequency from the grid. They also don't produce enough current to handle the peak loads from large appliances starting. The battery provides the current peaks. It is possible to buy more expensive off-grid inverters.

Yup. Stanford University discovered this to their dismay. Power grid went out, even with a field full of solar panels they had no electricity until the grid came back.

Typically solar will not feed back into the grid if it is down, to avoid energizing the lines and zapping repair crews. This is an intentional design, I’m sure anywhere with engineers or hackers could work around it.

You probably want a battery, though.

Batteries and inverters would be common in most installations, surly?

Batteries can cost more than the solar cells themselves. Plus, the older lead-acid batteries required some maintenance. The easy path is to just stay grid tied and not bother with a battery.

If all the power lines on earth went down, I bet communities would connect people with panels to people with batteries, though, somehow or another.

having lived in the tech age, I'm not completely against that idea.

starvation and dysentery at the same time appeal to you?

you're assuming that without electricity, I don't know how to take care of myself.

> a new dark age is far from impossible.

Nah, it's just transformers and the power grid, that can be fixed pretty quickly - see the response after natural disasters, Ukraine's power grid being directly attacked, etc.

The only limiting factor would be the availability of spare parts and personnel.

It's vastly different when it's a local outage and there's functioning equipment outside the effected area. How will communicate the needs of grid stations with no electronics, satellites, internet? How will this be coordinated? How will the trucks moving the equipment be able to refuel when the grid is down everywhere, and everyone is rushing to gas stations and clogging the freeways? How can even emergency services coordinate responses?

If transformers do blow, the big ones at substations and stations... We don't tend do keep spares of those on hand. They're very often made to order, with a multi-week lead time, and that's with a functioning power grid to produce and coordinate them, again.

I wonder whether ships (nuclear and diesel) and diesel-electric trains might be an essential resource in slowly restarting the grid. If that equipment survives, it would be invaluable to have a few mobile GW of power to hook up to gas plants.

No utility in the US has enough transformers on hand to replace them in the case that they all fail, hundreds of millions of transformers.

Even right now, disaster responses? All US mainland utilities are part of a emergency response pact (private, not government run) that obligates each other to send workers and equipment _at cost_ to each other. That's how things get fixed pretty quickly right now, they are depending on the supplies and labor of other utilities that weren't affected by the disaster.

That is far different than if the entire country is down.

A global disaster is very different from a local one. I suggest you review your faith in The Almighty We. Civilization has never been more fragile in history than it is now.

But there's nothing in the article to suggest that these events actually affected the entire world. Most of the analysis seems to have been carried out in the northern hemisphere. If one of these events happened, perhaps it'd affect only one hemisphere or we'd have portions of the world unaffected or partially unaffected and able to provide assistance.

Half the world is still very different to one country.

In fact, if the "once country" was _China_ (or Taiwan) it could be a civilisation ending event. We've built a house of cards, and while it does have some redundancy it's not all that resilient.

Keep in mind the modern world as it is now hasn't even existed 50 years, it has never really been tested with a truly international disaster.

We’re just coming out of an international disaster.

The “international disaster” was voluntarily shutting down production for a relatively short period.

Let’s see how we handle something real like another world war or global natural disaster.

You're ignoring (among other things) global food logistics. You delete the ability to produce, deliver, and distribute fuel to a hemisphere for a couple of months and the resulting disruption to agriculture guarantees a global famine.

Would it be global or would it only be the side of the Earth that was facing the Sun at the time of impact?

It hardly matters. Pick a hemisphere, you will probably hit 2 of China, the US, and Europe. No matter what a large chunk of the worlds manufacturing is going down.

Not the side, it's about the hemisphere. And in general the northern hemisphere is weaker to intrusions by powerful electromagnetic events (would have to lookup why again). It also turns out that most of the worlds population is in the northern hemisphere above the tropic of cancer.

Japan after WWII recovered pretty fast even though the entire world had been set back a lot. Human knowledge is the key resource and electrical surges don't wipe that.

That's a very different circumstance though. For one thing, the majority (about 3/4) of Japans manufacturing was actually still intact, it just wasn't up to the task of keeping up with Americas war machine (which it frankly wasn't even at its peak).

They also, as a key point, imported a lot during this period. That's something you can't do if everyone else has also collapsed.

Unless it manages to kill enough hard disks in data centers to truly lose a lot of knowledge...

All our important knowledge for the long-run still resides in physical books and journals. At most we'd lose some e-only scientific journals.

That's like saying the only limiting factor for the Titanic reaching New York is the big hole in the side of the boat.

The weak link in our electric grid are power transformers, because these are large, essential, long operational life equipment, whose inventory grew slowly over a century of electrification efforts. So if a large number of them blows at the same time, it takes the grid down and we don't have enough spares or production capacity to replace them timely.

Amapa state in Brazil got a taste of such disaster scenario, when the local electricity company was privatizated and management decided to eliminate redundancy and cut costs on maintenance to increase profit margins. The main transformer catched fire and no replacement or backup were available and the region endured weeks without power. See https://www.aljazeera.com/opinions/2020/12/18/the-crisis-in-...

Wow. That's exactly the sort of busted-bottleneck situation I was worried about during Covid. I'm used to hurricanes knocking out water and/or power for a few days, but a month is insane. There's no way a normal person with a normal amount of money and space can "prep" to live off-grid for a month. Part of the social contract is that you trade taxes in return for not having to worry about this kind of shit.

I had lightning hit the pole near my house. Anything with a circuit board that was not on a surge protector died.

After reading your summary, I'm thinking this might be the only way to get my kids to take a break from their iPads for a few days. That is terrifying and exciting at the same time.

Reply on news.ycombinator.com