All Of Argentina, Uruguay Without Electricity After Massive Outage | Hacker News Reader

All Of Argentina, Uruguay Without Electricity After Massive Outage

ndtv.com

292 pointsTanakaTarou141 comments

We just went through a couple of outages in Venezuela. The first one was 5 days, the second one 3 days.

After day 2 you start getting worried because everything failed, including the water service (they've been badly maintained and they don't generate their own power) and cell phone service. Lots of already scarce food was lost because people had no way to refrigerate it. At least it made for some nice neighborhood cookouts were everyone cooked and shared their food in the street.

Getting in touch with my parents in another city was like sending messages in a bottle. I sent an SMS, once a day they moved to one of the few places where there was still cell phone service and replied.

Since the main issue that caused the huge outages hasn't been resolved yet, we still get daily power rationing. I'm sitting here in the dark typing, 4 hours so far without power.

Hope the situation in Argentina gets resolved quickly!

That is scary. Our distant family in Venezuela have fled to Chile. Those left behind have died.

Yeah, it's a scary situation. A couple of days into the outage you start to hear about people looting and you can understand how a situation like that can get out of hand really fast.

As a curious note, my sister fled to Bs. Aires a few months ago (along with a the thousands of others that have been leaving), now she's going through this blackout over there.

> At least it made for some nice neighborhood cookouts were everyone cooked and shared their food in the street.

It's always nice to rediscover human warmth within such tragedies.

The situation in Argentina was completely different. This was a one time thing that was sorted out in 6 hours and it's very unlikely that will happen again.

It's very sad that while you are going through all this, in other parts of the world where the situation is far better, the only interest in Venezuela seems to be how they can use it to support their political claims.

For instance in Spain, leftist people refuse to acknowledge how bad the situation is because they link Chavez and Maduro with their ideas and somehow think that it would show weakness in their ideology, specially for some politicians that openly praised Chavez.

On the other side, right wing people want the situation to be as bad as possible so they can use it as a proof of how harmful the left is for a country.

So Venezuela will pop up again and again in political discussions, but the important issue seems to be "my ideology is betters than yours", not helping the people.

I really hope it gets better for you.

Yeah, the situation right now is a political tug of war between the left and the right. The people get stuck in the middle. It really sucks.

Thanks for your thoughts.

In the United States it's the same, but the left don't even take it as an example of socialist failures. Instead they attribute the state of Venezuela to having been the consequence of American interventionism. It's actually a tragedy, because the lessons to be learned from the failure of socialism come at the cost of countless lives, the least we could do is come away with something of utility.

Alas, your argument is tragic because with just a simple switch of a phrase, it becomes even more truthful:

>It's actually a tragedy, because the lessons to be learned from the failure of western imperialism come at the cost of countless lives, the least we could do is come away with something of utility.

We in the West all too often ignore the victims of Americas heinous illegal wars - over 500,000 innocent people have lost their lives because America believes it has a right to interfere in sovereign states across the globe.

Whether you are left- or right- oriented, you will be doing yourself and your culture a favour by stopping the very poor habit of ignoring the real victims of America's wars, who have too long been swept under the covers, because its either too embarrassing or the inherent evils too obvious upon inspection.

Americans of all political persuasions really need to stop America's war machine. It is, undeniably, the most negative influence in the world today.

I oppose all American interventionism. I think it's horrible. That being said, it's not the reason why the Venezuelan state failed.

Venezuela would be in a much better condition today if the CIA was not interfering in its internal politics. There can be absolutely no doubt about this - the evidence is absolutely clear to those who care to look.

To discount the negative effect that CIA interference in the normal operation of sovereign states has on the world, one has to be willing to fall for a hell of a lot of propaganda and lies. The CIA is one of the most evil organisations on the planet today.

CAMMESA administers the Argentinian electricity market, and they have data on demand [1] and grid flows. Compare the graph of flows on the 500 kV lines from 2019-06-16T07:00 local [2] to one hour later at 08:00 local [3]. For reference, see the prior day too, at 2019-06-15T08:00 local [4].

[1] http://portalweb.cammesa.com/Pages/ADemandas.aspx [2] http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Flujo%2... [3] http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Flujo%2... [4] http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Flujo%2...

I don't understand why BsAs temperature would drop just because the e got knocked off, or why it ceases to be an analog function after the cut :S

My guess would be that it's an average of several temperature sensors, and that a sensor going offline registers as zero. So when the sensors come back online, there's a sudden jump in the calculated average temperature.

The power drops so often that Argentina would be quite a bit warmer than officially registered here if it were true.

[2] and [3] refer to June 15; [4] refers to June 16. Is your text correct? What can be learned from the numbers on the 500 kV lines?

If you change the links to june 16th, you'll what they were trying to say:

http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Flujo%2...

Good catch, the links in my previous post are not accurate. The correct links for [2][3][4] are:

[2] http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Flujo%2... [3] http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Flujo%2... [4] http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Flujo%2...

The 500 kV lines form the backbone of Argentina's power grid. The drastic fall in power flows shows the massive shedding of load and generating capacity, as presumably plants disconnected from the grid for safety. One can step through the next few hours to see how generation was restarted.

Argentina is world’s eighth largest country. I find it mind blowing that you can have a power outage that covers the whole country (+ some more).

USA isn't in much of a better position.

Large scale transformer blowouts are a nightmare scenario due to how long they take to replace / repair.

Even worse when a generator goes out (or is destroyed). It can take months to get a new one. Hope we don't have multiple generators go out at once. They're not just sitting on a shelf somewhere waiting to be picked up.

how much redundant capacity does the grid have ?

It depends on what time of the year it is. During Winter Peak on the East Coast (ISO-NE/PJM/NYISO) or Summer Peak in the ERCOT/SPP regions, you might not have a huge amount of reserves left. The shoulder months would be much nicer, but that is when many generators go on outage for repairs, so you have other problems.

One of the biggest issues is that traditional resources (coal, nuclear, gas) can run for long periods of time and in many places have a pretty reliable fuel supply. Renewables are great, but you could be generating lots of power one minute and then it's all gone the next hour. This causes a lot of stress on traditional resources which need to be ready to counteract this. Coal plants and gas plants were never designed for that either and it's causing issues.

This is the nightmare scenario that we would have if a Carrington-class solar flare hit the earth. Many transformers would be blown out by the induced electricity in power lines. The attached article describes economic losses in the trillions. https://en.wikipedia.org/wiki/Solar_storm_of_2012

Additional reading on the Carrington event:

https://en.wikipedia.org/wiki/Solar_storm_of_1859

I work on one of the densest power grids in the world (NYC) on the team which does emergency transformer replacements and can possibly answer questions about this. From failure to in service for a single transformer in Manhattan is down to 2-3 weeks. That includes design and engineering (my part), permitting, and construction.

For fast transformer replacements we rely on two paths, either system spares (identical units, sometimes refurbished or like in kind replacements) and more recently mobile resiliency transformers. Resiliency transformers are trucked in on semis and can operate at 69, 138, or 345KV in our case. (There's some lightweight reading on them here: https://www.tdworld.com/overhead-transmission/siemens-introd...)

Neat job. I've always wanted to poke my head into ConEd's system and see the whole thing from behind the scenes.

Years back I remember ConEd was replacing two or three very large transformers down the street of a large venu I was working an event. Turns out we worked the same event the next year and amazing the same day ConEd again had the second transformer scheduled to moved into place. What an operation. Huge trailer from Bay Crane along with one of their giant hydraulic cranes. Was told the transformer weighed in at 240,000 pounds.

Heh, our most recent replacement a few weeks ago came in at almost 900,000lbs with oil. Depending on the speed of the replacement you can't always move the transformer dry because you need a certain amount of time between filling it and putting it in service.

Air bubbles need to rise first?

That sounds like quite the transformer. That must have been at one of the big 345kV subs. And I assume shipping filled is to avoid the time needed to remove air bubbles.

I know a few people in the ConEd trenches, one guy is a lineman who does all the >=69kV stuff and the other guy worked in substations and was in mott haven last I spoke to him (cushy job).

I'm always amazed how much new tech con ed has installed and how much old stuff is still kicking around. I live in south queens and we're all 4kV overhead fed from area subs in turn fed by 27kV. Some of those 4kV poles have westinghouse transformers from the 30/40's still in service, glass insulators, and leftover wooden pins from older insulators. Some are so old the primaries have fallen off and just hang between two adjacent poles, some the crossarms are half falling off. Amazing this grid doesn't go down more. Then the next pole is brand new, composite strain insulators, cutouts, reclosers, metering, etc. All modern hardware. And I know some areas they upgraded the 4kV overhead to 27kV. The secondaries are in even worse condition, ancient cloth insulation half weatherd/baked off. But it all still works.

I really love how there's this massive network of cabling and systems which orchestrate the flow of power from multiple synchronized generators so it can get to your humble wall socket and charge a phone or turn a light bulb on.

They are shipped without oil and full of nitrogen gas to reduce the weight.

Are transformers like these standardized, or are they generally built as one-off customs for the specific site?

(I'm not an EE, so not sure how many variables one needs to contend with in grid-scale transformer design)

We have purchasing standards which are particular to our system, though specs may match others' by coincidence. Most of these are electrical, e.g. how the primary and secondary windings are configured (delta-wye in most cases) but also what gets mounted to the outside (bushing potential devices, current transformers, gas monitoring units, etc. These are not unique to each site which is why there can be so much work to replace one.

Older stations were actually designed around the transformers in some cases and the physical configuration of a 50+ year old failed unit is rarely like those produced more recently. We might have a unit on hand that is electrically equivalent but impossible to install without essentially rebuilding the entire station.

To give a rough overview of the range of transformers on our system, the first thing we care about is the voltage and MVA rating, then whether the bushings are open air or gas insulated, what the winding configuration of the primary and secondary are (delta-wye being most common), whether the unit it is replacing has a tertiary winding (sometimes at transmission voltage, other times distribution or for light and power to the station), and then maybe what accessories it comes with such as current transformers for protective relaying.

I'm curious how long that process would take if it were do or die. Obviously, permitting would no longer be involved. So the real question is how long is the critical path through engineering->construction?

It varies significantly from project to project. What we depend on is the accuracy of existing records (30-60 year old microfilm scans typically, schematics and physical drawings), similarity of the failed and replacement units, ease of access to the site, support needed by other disciplines (e.g. concrete or steel work), and cost even if it is do-or-die.

A perfect like in kind replacement could possibly be done entirely by construction using proper procedure of documenting their disconnects if there were no structural modifications required to move the units in/out. This is rarely the case however and so we need to check everything from phase clearance on making the high voltage connections to whether disconnecting a failed transformer will trip out another perfectly healthy one, possibly making the situation much worse.

All that being said, we make every attempt to triage the work required and get information to construction as soon as possible so that no time is wasted waiting for engineering. We can have a conceptual layout circulating within engineering in a few hours and out to the field by the next day. Field technicians can tag and disconnect terminals right away while we figure out how to wire up the new unit. Perhaps in a perfect situation you might get all this done in a few days with all hands on deck but that's more luck than aptitude.

How long would NYC be out in the case of a massive solar flare that blows out all transformers in the city?

At 2-3 weeks a piece... let's just say we'll be eating twinkies for at least 2 years?

The interesting thing in this scenario is that if units were lost and unrecoverable in large swaths you wouldn't have to worry nearly as much about matching the legacy system perfectly, accidentally tripping out adjacent units, or things like that. Transformer replacements might even be easy, we'd just take whatever we had and wire it up anywhere it fits and can provide meaningful service. I'd be more worried about the issues to the underground system (nearly 100,000 miles of cable and piping) once oil stops circulating for long amounts of time, generators, and battery systems.

> I work on one of the densest power grids in the world (NYC) on the team which does emergency transformer replacements and can possibly answer questions about this. From failure to in service for a single transformer in Manhattan is down to 2-3 weeks. That includes design and engineering (my part), permitting, and construction.

Interesting. I had always wondered how utilities interact with local permitting. I had assumed they would work something out to have standardized permits for common work which they could leverage during incidents but that seems that's not the case.

It isn't really though. A cascading failure can knock out an entire grid. The Northeast blackout is an example.

  A line goes out. 
  while (there some lines are still live):
    The current redistributes itself among neighboring lines.
    Neighboring lines trip
  end
  BlackStart()

BlackStart() is fun. Having your plants and grid down is like having a dead battery (plants use ~30% of the electricity they gen on themselves) and no one to give you a jump.

Usually the hydroelectric plants are started first. Arg. is lucky, they have a lot of those. Otherwise you need to fire up diesel generators (which hopefully have been maintained).

You give power to the grid first so you can start another plant.

Then you feed hospitals and continue firing the thermal plants.

Throughout all of this you have to keep phase stability (no grid, no phase reference). I'd assume this would limit renewable's use-fullness for re-start (a thousand 5Mw turbines out of phase are best left idling)

As a last step you turn on your nuclear plants (Arg. has a few). These are last because of nuclear poisons that accumulate from decaying wastes that weren't burned while the reactor was off.

It's a bloody hard problem.

The renewables depends on how they're connected to the grid. If they're DC charging a battery bank then presumably said bank can be treated as a single source so they'd be useful. Though it might be more efficient if they were also able to synchronize and direct-connect to the grid as AC.

How is grid phase synchronised?

https://hackaday.com/2017/07/05/how-do-they-synchronize-powe...

Phase is monitored on both sides of the connection, and high voltage contactors complete the circuit when tolerances are met.

But if you have a grid with a large number of nodes, how do you impose which node is running at the correct (reference) frequency and phase? Do the edges of the grid impose a master-slave relationship?

I'm trivialising the problem, but whichever station energises the grid first becomes the reference. An operating grid has what may as well be infinite inertia (it isn't infinite of course), so new stations coming online must match it (or try and change the phase of the grid to match theirs). I visited a power station in Melbourne many years ago, and they have the shattered remains of a turbine and generator that was connected with a phase mismatch.

In the event that there's a phase mismatch, I assume that gets physically transmitted back to the generating device? (Mode subject to type of device) Or am I over-trivializing it?

E.g., will an out of phase turbine have a physical resistance against it, as a consequence of the rest of the grid?

Of a sort. If it's a phase mismatch of a few degrees, the generator may attempt to physically rotate to match the phase.

This kills the generator. Also, less amusingly, anyone nearby at the time.

If it's a larger mismatch, then that's more likely to fry the wiring before too much other damage happens. On rare occasions generators get connected while they are (exactly) 180 degrees out of phase; this is a dead short, and its internals will instantly vaporize. The vaporized copper may then proceed to explore the outer limits of the generator's physical integrity.

(Disclaimer, etc: I've only worked with DC generation, AC would be somewhat different especially if it's three-phase.)

I've synced 10 mw 900 rpm hydro units as much as 7 degrees out of phase. There is a bit more of a thunk as the unit snaps in to sync but it's not the end of the world.

Diesel units in hospitals will sync to each other as much as 30 degrees out of sync because just get thing online because it the load is critical.

I've also been told two second hand reports of the breaker closing drastically out of sync. One was a person closing the breaker manually when the unit was stopped. It essentially just puts a fault on the machine and it trips right away. Another case the close circuit was poorly designed and has only one relay, whose contacts eventually failed but happened to fail welded closed, and in conjunction with a poorly maintained breaker that developed a variable delay between the close coil being energized and actually closing. It made a big boom like faults do but the unit synced after the relay was replaced and breaker maintained and is fine.

Maybe medium and larger units with more inertia have tighter tolerances. Over certainly heard of broken shafts and generators being ejected through concrete walls.

A nice party trick is to do phase matching instead of syncing with slip so I can keep the unit exactly in phase with the grid as long as the grid doesn't wobble too much. This keeps the sync scope needle exactly at 12 o clock so you can close the breaker to put the unit online at will. Unfortunately i don't often get the budget for The additional inputs to get the nice phase difference signal so I can't perform my trick.

It depends entirely on how much inertia there is on the shaft, in various forms. The example I was told about was a 200MW peltier, constructed such that rotating it backwards implies pushing back against the water. The shaft itself also had some sort of extra equipment on it, but I'm not sure of the details.

Well, when it was connected about 30 degrees out of phase, the entire unit pretty much exploded. It tore out of the concrete base, and was found to have rotated nearly halfway to the "correct" position.

I haven't seen anything nearly that dramatic myself, so I can't be sure how much it's exaggerated. There's probably a report somewhere.

And I love that party trick. There's no direct equivalence in DC transmission, of course, but I've done similar things with voltage-matching in Electrical Age. It always amazes me how much that game ends up reflecting reality.

Water is a bit less forgiving than steam, it has great resistance to changes in position or velocity. I suspect that closing the generator created a water-hammer, which physically or mechanically destroyed the machine.

If the phase is not quite matched, you get power swing and mechanical oscillation. If you get it very wrong expect pole slip http://supachaisabua.blogspot.com/2013/09/2.html?m=1

"In the event that there's a phase mismatch, I assume that gets physically transmitted back to the generating device?"

In case of phase mismatch, you have a short circuit and the weakest element along the short gets fried.

Ok but imagine a circular grid of N nodes. Following the circle, the stations are connected to the grid. Of course there is a small, inevitable mismatch at every connection. Then, when connecting the last node, you might get a problem which can only be solved by adjusting the settings at the other nodes.

It turns out that this can be a substantial problem, resulting in wasted loop flows: https://iopscience.iop.org/article/10.1088/1367-2630/18/10/1...

Generally systems try to avoid too much loop power in the first place for this reason, setting up a directed but acyclic graph with some redundancy links carrying zero or near-zero power as standby.

The grid doesn't really pass through each of the points though so you don't really get the frequency mismatch between one side of a node on the grid and another. Instead of a ring of nodes it's closer to a ring with nodes hanging off of it so you never really get two frequencies observed at one power station because it's only connected to the rest of the grid at a single point.

The reference frequency is interconnection wide. To bring a three phase ac generator online the terminal voltage, phase rotation, frequency, and phase angle must be within some tolerance.

The following videos from BTCInstrumentation demonstrate this succinctly.

[1] Synchronizing AC generators -- Part 1 (introduction and sync lamps)

[2] Synchronizing AC generators -- Part 2 (strobe light view)

[3] Synchronizing AC generators -- Part 3 (sync and unsync)

[4] Synchronizing AC generators -- Part 4 (phase deviation)

The first video explains all the concepts and shows a sync, but the later videos bring additional conditions and detail.

[1] https://www.youtube.com/watch?v=RGPCIypib5Q [2] https://www.youtube.com/watch?v=sFohkp2aaU4 [3] https://www.youtube.com/watch?v=GRk_qJxaxh8 [4] https://www.youtube.com/watch?v=RT1ySBc-Bls

What do you do for a living, and how can I learn more about it? This stuff fascinates me and you've explained it really well.

The user is probably a transmission level grid operator. These exist at most decent sized utilities (Entergy, AEP, Duke), although it is possible there is a TOP (transmission operator company) that is not the same as the utility.

There are also organizations like the RTO/ISO that also act like air traffic controllers for the grid by keeping it reliable. They also are usually responsible for committing and dispatching generation for dozens of utilities collectively to save money (by banding together, you need to carry less reserves or backup power and can leverage economies of scale). In North America these organizations are (CAISO, SPP, ERCOT, MISO, PJM, NYISO, and ISO-NE). All of these organizations practice black-start drills (I used to help run them) and have NERC certified operators.

Honestly, I know that entire procedure myself as it's listed on a few different sites, one being the wikipedia blackout article. Then again this kind of stuff interests me so I read up on it.

I'm not sure you fully understand, or maybe it's me that misunderstands what you're saying :)

Each utility generally has its own black-start procedures. One utility might start two diesel generators (only really used for black-start) and then slowly start energizing the path to various loads and other generators. That utility and it's parent RTO/ISO train on that scenario several times a year and have the procedure printed in a binder within reach.

Something I've found interesting is that many of these plans are ~35 years old and we're created before computers were super common place for these kinds of things. The point being people weren't doing graph and optimization algorithms to determine the optimal paths. There has been a little university research and national lab research into this recently and I've heard they've helped certain utilities that they worked with determine some improvements, which is always exciting.

"The user is probably a transmission level grid operator. "

Close. My dad's a power engineer, specifically grid protection.

If he's doing grid protection he's probably in distribution engineering and they would be pretty involved with a black start drill from as in they or the line crews with their boots on the ground as not all of that procedure can be initiated remotely I'd imagine.

The transmission level engineers and operators will call the shots as far as how the high level stuff goes down. In other words they'll issue a directive to get unit x online and then close in breaker b and clear a path to unit y while picking up load at z. The actual process from the end of the people doing that work is likely to be significantly more detailed.

"If he's doing grid protection he's probably in distribution engineering and they would be pretty involved with a black start drill from as in they or the line crews with their boots on the ground as not all of that procedure can be initiated remotely I'd imagine."

And has a nosy son ;)

The EPRI Power System Dynamics tutorial is a free resource that is really solid https://www.epri.com/#/pages/product/1016042/?lang=en-US.

Can confirm. This is one of THE best references out there for explaining a lot of key power systems topics. I didn't think it was publically available though?

It is one of the few products that is available to everyone. And frankly one of best deliverables from all EPRI programs. Period.

Lol. I'm a computational material scientist, so take everything w/ a grain of salt.

My dad's a power (line protection) engineer, everything I know is from talking to him.

Right now I'm reading this book about this topic: Power System Stability and Control by Prabha Kundur https://www.amazon.com/System-Stability-Control-Prabha-Kundu...

Technically there is a last step you didn't mention. At some point after your island is ready, you'll want to reconnect with other islands. I've never done it in real life, but have done so in many simulations.

I forget the exact calculations involved (it's been years and I'm no longer involved with that area of operations), but you want to make sure your frequency matches the grid you're connecting with to within a certain amount. I think the stations which connect neighboring areas usually have extra monitoring equipment for a variety of energy accounting reasons, but also to measure the frequency on each side to see if it's a go or no go. I believe they use synchroscopes (https://en.wikipedia.org/wiki/Synchroscope) to do this.

Edit: it looks like y'all started talking about this below and I missed it, but the information is good, so I'm leaving it up.

To connect two regions, we align the phase of the two islands. As the voltage across our switch bank falls to zero, we close the switches. It's not a big difference from between synchronizing a generator. The phase is aligned by requesting small changes to generator output, and measurement is done by synchrophasor.

Thanks for this, it is very informative and not something I had considered before.

I'm curious about the statement about renewables though because it seems that with wind or solar the production is very irregular even in normal operation. Doesn't that mean there has to be some technology to balance individual generator differences which would make re-starting those installations easier?

Re: Nuclear, how much energy is required to start up a nuke plant? If decaying toxins are a risk shouldn't those be prioritized? I might misunderstand what you mean by toxins, does that reduce production for some time?

There can be issues with a line-commutated inverter or asynchronous generation (wind-farms might have both) if the fault level is too low (synchronous plant too small or too distant). More recent research has shown the benefits of adding grid support (reactive power control, fault response) to the control system of renewable generation.

In future I would expect renewable and storage elements to provide black-start frequency reference.

"Nuclear poisons" refers to neutron-absorbing substances produced through transmutation. When the reactor shuts down, it will continue to produce these for some time after the shutdown -- Xenon-135, particularly.

This proves problematic during reactor startup. Since it's absorbing neutrons, the controllers would need to retract the control rods further out than during steady state in order to achieve a replication rate of 1.0. However, xenon-135 stops being a nuclear poison after absorbing its neutrons... which means the replication rate will increase, and the reactor might run away.

This is part of what happened in Chernobyl, though they'd also disabled most of their safeties. (And dismantled the others.)

Regardless, nuclear poisons are something to keep a very careful eye on.

True.

Xenon poisoning was discovered at the Hanford, Washington, B reactor during the WWII Manhatten Project.

https://www.osti.gov/opennet/manhattan-project-history/Event...

    > Neighboring lines trip
    > As a last step you turn on your nuclear plants

...don't get it: why would you turn off your nuclear power plants, even if the grid failed?!

Won't it make more sense to keep it running as usual maybe just using the power to generate steam that you throw into the atmosphere? ...mainly because stopping and restarting a nuclear power plant is expensive af.?

(It still boggles my mind why are electric grids so focused on efficiency at the cost of resiliency... with abundant nuclear + hydro you could easily over-produce a bit, I'm 99% sure that making electricity dirt cheap for consumers by gov controlled over-production would help fight climate change even if it appears wasteful: there will be huge incentive to replace petrol burning trucks, buses, cars etc. when electricity it dirt cheap... and then prices can be gradually ramped up after everyone made the conversion.)

I'm not an expert, but there's an interesting discussion on Quora: https://www.quora.com/Can-an-American-nuclear-power-plant-bl...

The key factors seem to be:

1) the plant must be capable of running as an island. It seems not all of them are; I suspect the problem is the plant consuming a variable amount of self-power e.g. 1MW being connected to a 100MW generator results in a very unstable system.

2) whatever safety event causes the problem must not trip at the generators themselves but disconnect the plant cleanly from the grid. Since these events are by definition exceptional, this may be very hard to avoid.

> with abundant nuclear + hydro you could easily over-produce a bit

Both of these are exorbitantly expensive to build, and hydro requires specific geology.

Most of the plants I've worked on can't island themselves by design. It costs more to do it and sometimes requires additional or more expensive equipment, more attention to details in the design, and there is no point for a lot of generators that just get paid for MWh and they probably aren't going to generate many MWh islanded over the life time of the plant. Some generators even get paid for the energy they would have generated if the grid is down.

How many people have a manual transmission car just so they can pop the clutch if the batter is dead.

I'm not in this field (university friends are), but reactors require triple redundancy for their power supply, including the grid.

A better question is, should we have nuclear power at all? I took classes in nuclear engineering, by friends are nuclear engineers, ect. but it's not obvious to me that cheap electricity is worth it.

> not obvious to me that cheap electricity is worth it.

Why would you say that? I mean sure, if you can't manage to get cost-effectiveness right then it's not worth it. But otherwise dirt-cheap and readily available electricity is absolutely needed to solve the world's problems! Think about what will happen when large swaths of land loose access to drinking water after climate + pollution wreck things up and the only alternative is ocean water filtration and desalination plants that will need huuuge amounts of power! Also think about massive irrigation and civil engineering projects that will be needed to fight desertifications etc. And to keep heavily populated costal cities above (the new, higher) sea level. That's terrawats upon terrawats of energy that will be needed to constantly shuffle earth and water around, and maybe even to power huge city-scale air filters etc.

We have and still are wrecking up the planet to such a high extent that sooner or later the bill will arrive and we'll need to put heavy effort into re-terraforming Earth to keep it inhabitable... And there really is no other answer than nuclear here, while cities filled with LED-lights and low powered devices and highly thermally efficient buildings can run on renewables, heavy civil engineering projects and land decontamination and producing drinking water etc. will need waaay more power than our blanket of communication equipment and drones.

We could decide not to, but then we'll end up fighting for resources and instead of nuclear power with it's inherent risks we'll get... nuclear wars!

> >...don't get it: why would you turn off your nuclear power plants, even if the grid failed?!

> reactors require triple redundancy for their power supply, including the grid.

Thanks for posting. I learned something interesting today.

I can imagine the consequences of a total loss of power in nuclear reactors...

What type of power plant has 30% of nameplate rating on station service loads?

I've been to about 30 stations under 50 MW and the biggest station load might have been 300 kW if they had all the heaters, fans, pumps, crane, welders, air compressors, and gates operating at the same time.

> BlackStart()

I love how the people coming from different fields invented their own word for "the process of starting a system through external intervention because it cannot support itself at this point".

Kickstart. Jumpstart. Bootstrap. And now I learned a new one: blackstart.

Puerto Rico was without power for months after Hurricane Maria.

They had massive physical damage from a powerful hurricane, their grid didn't just fail. Transmission lines and substations had to be repaired.

That is ignoring issues like Whitefish Energy among other things. Very different situation.

edit: yes, we know that nothing anywhere near comparable to hurricane Maria hit them.

Do we know that there's been no grid damage in Argentina? It is the middle of winter.

Most of the population and the grid are in temperate weather zones that get little wintry weather.

Most of the country has a very warm winter (yesterday I spoke to my mam and she said it was 16 C).

There's been grid problems since I have memory. Doesn't surprise me that the whole thing eventually went down.

Not that the PR grid was a an example of reliability before the hurricane. I had to endure hours and sometimes days with no power rather consistently.

Maria wasnt the only hurricane that left us in the dark. George and Hortense were a 2 month ordeal with no power (aside from small gasoline generators) in the town I grew up.

parts, not the whole island. correct me if im wrong

The whole island didn't have power for a few weeks, probably about a month.

Source: born and raised here

thanks

I was going to post this. Thanks. Looks like Uruguay, Paraguay, south of Brazil, Bolivia are also affected. Several million people. It’s raining, no internet, this will be a long Sunday. Also today is father’s day, will be a celebration with candles like in XIX :)

> south of Brazil

That sounded strange to me, since Brazil imports power from Uruguay and Argentina through somewhat small DC links, and if only them failing is enough to cause a blackout in Brazil, something was probably not configured correctly. So I went looking, and according to https://g1.globo.com/mundo/noticia/2019/06/16/apagao-deixa-m... "deixou sem energia toda a Argentina, Uruguai, Paraguai e sul do Brasil", so Paraguay was also affected. Paraguay shares the huge Itaipu dam with Brazil, and a large amount of power in Brazil comes from the Itaipu dam, so if Paraguay was affected, Brazil (mostly the south and southwest) will probably also be affected.

edit: the realtime graphs at http://www.ons.org.br/pt/paginas/energia-agora/carga-e-gerac... show a loss of around 1.3 GW of hydroelectric generation in the south region starting around 7:00 and ending around 10:00, matching the timing of the blackout.

According to Paraguayan news [1] (Spanish), the failure was on the Argentina side of the Yacyretá hydroelectric (which is shared by Argentina and Paraguay), not at the plant itself. Only a few Paraguayan towns close to the hydroelectric itself were affected, while Argentina and countries it resells electricity to were out of power.

I can confirm, at Paraguay, only two small towns (Pilar y Ayolas) where affected and for a short period of time. Source: I live at Paraguay.

Nice to see Paraguay present here on HN, I live here in Asunción.

I also heard of some people near Luque without power Sunday and Monday, but I'd guess it was an unrelated ANDE incident.

I was on the Internet (In Argentina) the whole day. :)

It isn't Father's Day in Uruguay, Brazil, or Bolivia.

Not every country in the world celebrates holidays on the same day that US Congress designates them.

How resilient has the cellular phone network been in this outage?

How resilient is the municipal water system in an outage like this?

Do most people in Argentina and Uruguay cook with (piped/tank) gas or electric ranges?

At least in Buenos Aires metro area where I live, cell, water and gas are working ok. Water may be an issue if this continues as the pipe pressure is not enough to fill up the buildings tanks, so you'll need to go to ground level with buckets or similar.

In all of the countries affected the standard way to cook is natural gas, so we should be good in that front until power is up again.

I'm getting reports from friends and family that different zones of the metro area are getting powered as I write.

Me, still no luck... Hopefully in the next hour!

My family at the metropolitan area of Buenos Aires run out of water during Sunday. Cell phones where working intermittently.

I haven't tracked the cellphone network status continuously, but here in Buenos Aires, I had a VoIP call from 8:30 to 9:00 which worked fine. Until it didn't, at about 9:00, but that's par for the course with fucking Movistar. I was using cellphone data because the power was out in the house. I didn't realize it was out in the entire country.

Here in Argentina, municipal water is totally reliable (I've never heard of a water service loss due to a power outage), but the pressure in the pipes is so low that every individual house has a pump to raise the water into a rooftop tank. These pumps are invariably electric and powered from either the grid or, on the frequent occasions that doesn't work, a generator. Municipal water typically has service losses a few times a year, but not due to blackouts.

Gas stoves are far more common in Argentina than electric stoves, although my apartment has an electric stove, which has a short which electrifies the stainless-steel surface of the stove when it's turned on. Most people in and near the capital run their gas stoves and on-demand hot-water heaters from public-utility natural gas, supplied by local monopolies (state-owned in some cases). Propane cylinders are common in less-populated areas and in houses where the gas piping has been disconnected from the public utility, usually due to billing disputes or safety concerns.

Over the last few years, the utilities have been sandbagging gas reconnections after disconnections due to safety concerns, often allowing them to leave unprofitable customers disconnected for years at a time once someone has called in a gas leak, while they wait for the government price controls (which have fallen far behind our absurd levels of inflation) to be loosened, making those customers' business profitable again. This has been a significant reason for new installations of electrical cooking and hot-water-heating appliances. A court recently ruled against this practice, and so gas reconnections have been sped up lately.

Uruguay has much less use of natural gas, for regulatory reasons: gas appliances can only be installed indoors if the room in which they are installed has a window to the outside that cannot be closed, thus removing the risk of explosions in the case of a gas leak. Consequently in Uruguay gas stoves are almost unheard of, and nearly all cooking is electric. Many hot-water heaters in Uruguay are still the gas-powered tankless type that is nearly universal in Argentina, though. They're just installed outdoors.

(Actually, I'm not sure if the above information about Uruguay applies outside of Montevideo.)

I hope this information is interesting to you!

Not OP, but deeply interesting to me and especially so as I travel around South America.

I am completely fascinated by the home infrastructure choices in different parts of the world. I'm currently in Sao Paulo and the hot water setup is a little scary. There's no hot water reserve or hot water plumbing at all.

The only hot water source is in the shower. It's heated electrically in real time by a 5000W appliance that sits just above the shower head. The only way to change water temperature is by turning a knob on the appliance. In homes with bad grounding, this results in shocks often enough that my hosts have shown me Brazilian memes about it.

Yeah, electric showers are common in much of Latin America. In theory you could build them to be safe, but often they aren't. I showered with a tank-based electric shower today: you fill the tank, plug it in until it's hot, unplug it, step underneath, and open the valve. This arrangement has less risk of electrical shock, but it's far less convenient. Because the power was out, I filled it with hot water from the tankless gas heater, but it was able to cool enough in the process that it was more like a lukewarm shower than a hot one.

Many of the showers in Scotland are electric too, but they look much more modern and safe than the Asian and South American ones I've seen on the internet.

The difference is that in Scotland and Europe, it’s just one or two insulated heating elements that the water flows past. There are numerous safeties, overheat protectors, usually an RCD - in short, you’d be very unlucky to get a shock.

In South America, I’ve seen two types. One: an element, but it’s uninsulated and has no safety features. This is usually operated by a water pressure switch and operates by heating a thin resistive element up. High danger of faults to earth (you’re earth).

The other kind has me rather gobsmacked - the water is the element. The idea is to just pump in enough juice that the water starts to flash boil, and forms a plasma channel that then heats the surrounding water. Ostensibly safe, if earthed, but I really disliked the 60hz skin tingle - later discovered that they’d hooked earth up to the third phase in that house, when I plugged my laptop in. Metal case, leg, funky chicken. I guess the shower is reliant on skin effect to not kill you.

There’s a really great YouTuber who is forever tearing domestic appliances apart - it’s actually quite amazing how some stuff is built, both in a good way and a bad way.

Links to showers:

https://m.youtube.com/watch?v=ZwuhFLsowRc

https://m.youtube.com/watch?v=cNjA0aee07k

> In theory you could build them to be safe, but often they aren't. I showered with a tank-based electric shower today: you fill the tank, plug it in until it's hot, unplug it, step underneath, and open the valve.

Do any homes have things set up like most homes in the United States - that is, a central water heater of 40 gallons or more that supplies all taps in the house?

Something else we have here, but aren't as common, are "instant on" heaters; they come in two fashions:

1) Tankless - basically similar to the "electric showerhead", except much higher wattage and installed usually below the sink for both sink and shower usage (if in a bathroom). Water flows in and thru a very tiny holding tank, is rapidly heated, then passed to the tap.

2) Recirculation - in this system, the hot water side of things is basically connected in a loop fashion to the main water heater, and is circulated with a pump. The main water heater, IIRC, can also be smaller. The lines are also all insulated. Hot water is circulated around, turn it on, there it is.

In the usual system, though, you turn on a tap and wait for it to "warm up"; our American wastefulness in action I guess (but technically, at least in most non-rural areas, that water goes down the drain and into the sewer system, where it is taken back to the water purification plant, and is essentially turned back into drinking water - while it doesn't all return, due to evaporation, leaks, etc - the majority I would surmise does, so while still wasteful, it isn't as terrible as you might imagine - but you do still have to pay for that extra water, which is where it does hit ya).

I live in a shitty new apartment building without gas (not the one with the tank-based electric shower I mentioned above) and there's a central water heater in the basement for the entire building. Like all the other heating devices in the building, it is electric. I think it's about 2000 liters ("528 gallons" in medieval units). I don't know if it's equipped with a recirculation system, but the basement is about 10 meters below my apartment ("33 feet" in medieval units), and it sure doesn't seem like I have to wait for hot water to come up from the basement when I turn the hot water on, so possibly.

Most US cities discharge their treated sewage into bodies of water, allowing it to evaporate from the ocean before entering a potable-water plant. Perhaps yours is an exception?

Yes, suicide showers... Cuba has those too.

5000W seems like a very low power draw for heating on the spot. Heating water requires about 4000W per degree Celsius for a flow rate of 1 liter/sec; a typical shower flow rate is maybe a third of that, so call it 1000W per degree Celsius of heating. So 5000W would only give about 5 degrees Celsius of heating, which doesn't seem like much.

While you're right that 5000 W is only 1.2 kelvin liters per second on water, typical shower flow rates are about 8 ℓ/minute, 133 mℓ/s in SI units, and at that flow rate you get about 9°C of heating. This may be a reason electric showers are more popular in tropical countries like Brazil, Perú, and Costa Rica: ΔT = 9° gets you from 28° to 37° just fine. And you always (well, usually) have the option of lowering the shower flow rate to get a wimpier flow of much hotter water.

> And you always (well, usually) have the option of lowering the shower flow rate to get a wimpier flow of much hotter water.

Don't overdo it; when I was a child, I burned a shower head by doing exactly that (lowering the flow to get hotter water).

> typical shower flow rates are about 8 ℓ/minute

This is a fair bit slower than a typical US flow rate (which is about 5 gallons/minute or about 20 liters/minute). So yes, with a slower flow rate and hotter water to begin with I can see it being usable.

> I'm currently in Sao Paulo and the hot water setup is a little scary. There's no hot water reserve or hot water plumbing at all.

While that's a common setup, some homes have instead a "boiler" which is a large water tank heated by a electric resistance. The disadvantage is that, once you've used up all the hot water, it takes a long time to heat up again; electric shower heads are instant. Another common setup is a gas-powered heater; I never liked these, since they're dangerous (you have to be very careful to always leave the window open while they're operating, and always be mindful of leaks). Sometimes a home has more than one setup at the same time (like having the gas-powered heater for the kitchen and one service bathroom, while the main bathrooms are on a boiler or use electric shower heads).

> In homes with bad grounding

Or no grounding; I live in a relatively modern building (~30 years old) which predates the newer three-pin power plugs, and all sockets have no ground. I suppose, though I'm not certain, that the power panel for the apartment has no ground wire, with the neutral being connected to the ground only near the meter in the ground floor.

> Uruguay has much less use of natural gas, for regulatory reasons: gas appliances can only be installed indoors if the room in which they are installed has a window to the outside that cannot be closed, thus removing the risk of explosions in the case of a gas leak. Consequently in Uruguay gas stoves are almost unheard of, and nearly all cooking is electric. Many hot-water heaters in Uruguay are still the gas-powered tankless type that is nearly universal in Argentina, though. They're just installed outdoors.

Uruguayan from Montevideo here, and I disagree with this. I've seen very very few electric kitchens. Almost all the stoves I've seen are gas powered. Most use gas tanks, and some have piped gas into their homes. Hot water heaters are usually electric, and installed indoors. You'd only have a gas heater if you have piped gas.

Thank you for the correction! Serves me right for being so credulous.

Tandil, Buenos Aires province: No cellular phone service for Claro (one of the cell phone companies here). Water service out for the entire day. Luckily I have a rooftop tank, but I got worried about running out of water if this outage went on for days.

David E Nye's When the Lights Went Out traces the history and impacts of blackouts within the U.S.

https://www.worldcat.org/title/when-the-lights-went-out-a-hi...

Reminds me of the outage that hit the Northeast U.S. in the early 2000s.

Does it have something to do with the Copa America that's currently going on right now? Maybe more people inside watching the games and using up power.

It's winter in Argentina now, so they don't have AC loads. TVs are small by comparison (Although it's fascinating to see how load changes during large games: you can spot commercials, for example!)

In the UK, this phenomenon is called the TV Pickup: https://en.wikipedia.org/wiki/TV_pickup

Basically a commercial break comes on, everyone turns on their tea kettle and flushes their toilets, and bam: 200-400 MW spike in demand.

There's some blogs out there charting this during big world cup games.

>There is a common misconception that the number one driver of TV pickup is the boiling of kettles. In fact, this only creates a pull on the local network for a short period of time until the water has boiled, and can therefore be managed relatively easily, whereas flushing the toilet causes a longer surge at the water and sewerage pumping stations, and opening the refrigerator lets the chilled air escape, causing the compressor to run. These loads are more of a problem for the grid.

dry toilets and solar concentrator kettles, yesterday

Or battery to handle short spike in demand (backed with gas turbine if risk of prolonged spikes).

Or a wifi connected kettle or toilet that pre-heats pre-commercial or delays tank re-fill respectively.

There was a coronal hole open these days, any chance this is related to sun activity?

seems unlikely given the very low sun spot activity in our current minimum

https://www.swpc.noaa.gov/communities/electric-power-communi...

Might it have something to do with this? https://news.ycombinator.com/item?id=20190025

I just find the timing amusing to be honest. If both Russia and the US are vulnerable to cyber attacks on the power grid, there is no doubt in my mind that Argentina is much more vulnerable.

Power grid hacking exercise?

Maybe. Maybe not. Forensics are on it now.

I really take no pleasure in saying this

THE PROBLEM IS CENTRALIZATION!!!!

In this case, of the electric grids. In USA there are a few huge transformers that can be knocked out with an EMP, carrington event or otherwise, and plunge the USA into a similar situation.

We need to decentralize:

Electric grids (solar, etc.)

Cellphone towers and ISPs (mesh networks)

Social software and collaboration (open source software platforms to replace Facebook and Google)

I am a big believer in this, and I have led Qbix to reinvest nearly a million dollars of our company’s revenues to build towards this future:

https://m.youtube.com/watch?v=pZ1O_gmPneI

https://www.youtube.com/watch?v=WzMm7-j7yIY

https://qbix.com/blog

Happy to answer any questions / comments / threats :)

Are you sure that the problem is centralization and not interconnectedness? I think if Argentina had just one generating plant they would have found it a lot easier to get things back up and running - no coordination problems - and if they had a few dozen independent grids then only a handful of them would have been affected (just one, or even none). The grid, as it exists today, is rather distributed - hundreds of generating plants in both US East and US West, and I imagine the situation is similar in South America.

Considering the current advances in solar technology, I think we'll soon hit the point where a local power grid powered by solar panels and batteries will be more efficient for people living a bit remote, and I do believe that we will see a larger amount of people living "off-grid" in the rural areas of the future.

Communications networks are the big problem, but sattellite-based solutions like Starlink might help in that regard.

Problem is that none of it will help if the Sun decides to throw a bunch of star stuff our way; local grids will be just as fried as the big ones, and it's better to make sure people are able to survive on their own without electricity for at least a few days.

Decentralizing would mean more frequent local power outages as you cannot meet demand with local generation.

Decentralizing would force lower efficiency on power generation.

An incidence/severity trade-off.

Decentralization doesn't mean you can't have a market for energy, and interconnections.

For an electric grid it kinda does.

Why is that?

Multiple sources of energy generation and the option to participate in voluntary exchanges.

Reply on news.ycombinator.com