Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

Spain is mostly back up.

Plant startup is only the first step.[1] It's load pickup.

Here is a PJM training module on load pickup during system restoration.[2] It gives a sense of how touchy the process is. Power network control has good control over generation and transmission, but limited control over load.

When load is turned on, there are transient loads with different time constants. There's a huge load for the first second as inductors, capacitors, and incandescent filaments start up. That tails off in under a second. There's a second load as motors wind up to speed. Ten seconds or so. Then there's "cold load", where everything in HVAC starts trying to get temperatures back to normal. Maybe half an hour.

There's no mention of computer control. Listening to this, you visualize people running around reading meters and throwing big switches. It's probably people looking at display boards and sending commands to remote big switches, but the concept is the same.

Botching this means voltage or frequency goes out of tolerance, protective devices shut things down, and the system operators have to start over.

More PJM training modules on related subjects.[3]

Unclear what caused this yet. Something caused enough system instability to trip protective devices, but there's no good info yet. Once everybody has a chance to compare all the logging data from different points, it will make more sense.

[1] https://pjm.adobeconnect.com/_a16103949/p622tuwooba/

[2] https://pjm.adobeconnect.com/p6e5csm81ter/

[23 https://www.pjm.com/training/training-resources

Sounds like these systems are extremely vulnerable to sabotage. But adding more complexity such as distributed digitally controlled loads will not necessarily make the situation much better.

In theory maybe, but Ukraine are doing a great job of keeping their grid up in a hot war despite attacks on power infrastructure.

The groundwork for the blackout will inevitably come back to a buildout and reliance of renewable sources that do not have spinning mass that can do frequency synchronization.

There are costly means to compensate for the lack of spinning baseload but actually building these devices have been neglected, to no ones surprise.

Not sure why you are downvoted but this rings true to me.

For about 2-3 years the default for people here has been to downvote everything rather than driving discussion; much more reddit-esque than what it used to be. There are no other sites like this one to jump to, unfortunately.

On-topic: Wind power brings in about 20% of Spain's electricity, and that one fluctuates a lot indeed. I hope they put out a detailed post-mortem, I'd be an enjoyable read.

Exhibit A: this comment.

Some even borrowed the essence of it, posted it elsewhere and still downvoted, ¯\_(ツ)_/¯.

Just like there is vibe coding, this is vibe downvoting: “they are probably right, but kinda feels like they are dissing renewables, and I like renewables, better downvote just in case”

You're right, but: the renewables "debate" is a massive tar-pit of bad faith arguments. People are tired of rehashing that. The best solution is not to slow the rollout of renewables but to accelerate the rollout of battery storage.

(also we're a bit early for a proper postmortem, but that's an inevitable reality of social media, got to get the talking points in before the facts)

> The best solution is not to slow the rollout of renewables but to accelerate the rollout of battery storage.

I agree as well. And I think the gp (or the ggp?) post noticed that there is a lack of "spinning mass" to do frequency damping would have agreed as well. They could be merely suggesting to add just spinning mass synchronizers. These could be independent units, doing just that or maybe even convert some old power plants: "just" decouple the turbine from the generator.

The specific issue identified by the parent would not be helped by batteries. Like solar panels, batteries output DC voltage, and have no inertia on the AC side.

The solution in either case is the same. Put enough smarts in the inverter for it to have "synthetic inertia". If we didn't have the ability to do something like that in inverters, the solution would be something like flywheels to augment the chemical batteries.

Why would you need a spinning mass when you have an inverter which can produce a range of frequencies and outputs and it all boils down to coordination?

Looking at power generation reports it's the plants relying on inertia (like nuclear) which were disconnected and haven't been reconnected since:

https://app.electricitymaps.com/zone/ES/72h/hourly

At the time I'm writing this, there appears to be about equal (hydro + gas) = (solar + wind). It appears they are running a significant portion of the grid with high inertia generators.

In a control system when you have an actuator that is much more responsive than the load, you can get into instable operating regimes where. On a small scale, when you are setting up servo motors, there is an inertia matching ratio for stability. In some of the systems I set up it was 10:1 load inertia:motor inertia. If you exceed that, you end up with elasticity between the motor and load that the motor can locally be above set point, and the control tells the motor to slow down, while the load has not yet reached set point. The motor is then too slow, and the control tells it to go faster, and you get the motor oscillating around the speed of the load.

On the grid you can have similar elasticity. An inverter many km away can get too far ahead in the AC cycle, then when that signal reaches the local inverter, the local inverter slows down, but the remote inverter has already slowed down, and by the time that slower wave gets to the local inverter, it is already too slow, and so the local inverter starts to speed up. If you get the frequency "right" you can end up with a positive feedback loop. With a high inertia rotating generator, the system is damped and slow enough to respond you don't change set point faster than the signal can propagate.

Seems like a simple cybernetics problem, especially if it's point-to-point. If there's many different inverters on the same circuit, you might need something like a directed graph, but there should be simpler, more local solutions.

If the response rate is the only problem, you could just add damping to the system with the faster capability (especially when it is already software driven electronics).

bless you for your correct use of the C-word.

I'm sure the inverter manufacturers are working on something.

At the start of solar and wind integration, it didn't matter. Large rotating generators were the main source on the grid, and provided stability. We're in a different world with an all-inverter grid control perspective-wise.

I think there is also still an information issue. The grid is definitely not point-to-point. It also isn't all generators. You also have loads switching on and off at indeterminate times. There is an overall predictability to it, but not at the phase governing resolution. If you have a local inverter and the phase slows down, you don't know if it is a remote inverter that has over-slowed, to which you shouldn't respond as it will fix itself, or a near-by load that suddenly switched on, to which you should respond as demand is now greater, and needs more supply. With big rotating generators, it doesn't really matter, as they are time averaging with the large inertia to stabilize the load/generator balance.

Again, this appears to be a problem of coordination. An inverter doesn't have to:

-react immediately

-use its full capacity all the time

You have a fundamental misunderstanding of both how a grid in blackout mode appears to a lonely inverter and what means of coordinating inverters that are available to said inverter.

Imagine you're trying to coordinate a choir of ten million singers that are scattered across a radius of a thousand kilometer, that all sing into their individual Ham radio. And you need it to come out in a perfect unison that sounds good for those who tune into the mixed broadcast.

"A problem of coordination" is perhaps correct, but it neglects the difficulties involved

Man, if only we had some means of instantaneous communication at a distance. What a crazy world that would be, huh?

Superluminal information transfer would be more useful for other areas than grid synchronization, but yeah we don't have crazy tech like that. And if we attempt to synch to a 50hz frequency over the internet means a 10ms latency difference results in 180 degree out of phase input.

And even if we did have the superluminal grid sync signal the propagation of electricity itself is slower than C and you'll need to consider perceived phase synchronization over distance.

Microwave, hell, even GPS-based time sources exist.

If I can sync ALL my [important] servers across the globe to the EXACT same time (under 2 microseconds variance in the last 7 days), and all that on the off-the-shelf hardware, surely a NATION can devise something better?

Thing is, it's not ten million - it's several orders of magnitude fewer, particularly if we're starting with just a selected list of generators.

Also we know the locations of the generators, so we can calculate the phase shift and control them accordingly. At thousand kilometres it's around 60 degrees anyway, so not catastrophically huge.

But also, it isn't just generators, it is also the indeterminate loads. If you have nearby loads switching on and off that are similar sized to your local inverters, that is a complicated controls modelling problem.

I'm pretty sure in grid terms, 60 degrees out of sync is catastrophic. I'm pretty sure you'll be disconnected by protection circuits before it gets that far out.

Until we start seeing graphs of voltage and frequency, and logs of breaker trips, speculation is futile. All that info is available for major US blackouts.

Inverters overheat, you need to protect them from insects and dust, and they don't last long. The typical expensive IGBT lasts for 6 years. Of course they are much better, but you need the protect them.

Unfortunately that pretty much just leaves solar panels, as wind/hydro/concentrated solar/geothermal all rely on a spinning mass moving at a mostly consistent speed, though at least hydro can black start itself relatively easily

I don't see why. Battery-backed renewables should be much easier to synchronize than spinning generators.

I've always wondered why we can't just have a "master" power station transmit a 50Hz signal modulated at, say, 50MHz. Then all the other station simply sync to that, including small scale such as domestic PV etc.

Then depending on how out-of-sync a given station is with the signal, they ramp production up or down accordingly.

Is the problem here that there is inadequate or slow feedback to the supply side regarding load and therefore you'd end up with under-voltage?

That's more or less how it works. Knowing what 50hz is and how out of sync a generator is is easy. AFAIK, the problem is one of inertia, both electrical and mechanical. A generator is a spinning mass, the more load on the generator, then the slower it turns. How fast it turns decides the frequency of power it generates. Say you get your generators running at 50hz at night, and then during the day it gets hot and thousands of AC units get turned on. Now your generators are running at 40hz.

It's not about getting the generator to spin at 50hz, it's about getting it to spin at 50hz under dynamic load. Most generators are mostly fixed speed I think for the sake of efficiency, so the mechanism the grid has to regulate the frequency of power is largely by controlling the number of generators running at any given time, and how much of the grid is connected to them. Ramping up and down based on the current observed frequency of the grid is exactly how this all works. However it's more complicated because what if two or more power stations see a low frequency grid and decide to ramp up more generators? Well now the grid frequency is too high. If they them respond to that by ramping down, it'll go too low. Rinse and repeat and you get undesirable frequency oscillation, so there needs to be more communication across the grid than just responding to the locally observed frequency.

Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart by the sheer inertia of the power grid forcing it to match speed and phase in an instant. This is also a problem when attempting to connect two isolated grids, of they're too out of phase with each other, catastrophic physical damage will occur.

Thanks for the reply. In the 50Mhz-modulated-50Hz scenario, however, the modulated 50Hz signal doesn't change under load; it has zero inertia (or infinite inertia maybe) because it's electronically generated.

Any individual station that gets out of sync either works hard to catch up, or gets disconnected.

The problem, as I see it, is that the load signal is equal and equivalent to the sync signal and maybe they could be separated. (e.g. the FM modulated signal wouldn't have to be a sine wave, nor would it even have to be 50Hz)

> what if two or more power stations see a low frequency grid and decide to ramp up more generators?

Seems to me it would solve this problem - they both have a single invariant sync signal.

> Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart

Seems to me this problem is solvable with electronics, but I don't know enough to say how much out-of-phase is problematic. What if, as you say, some heavy load is switched on, the local nuclear turbines slow by a fraction for a few 10s of seconds, and so a home PV array starts leading by 0.01Hz (e.g. home PV at 50.00Hz, local power station at 49.99Hz, so after 10s the PV is fully 36º or 0.6rad ahead). Does this result in the inverter (or the small local windfarm) exploding? Is that a realistic scenario?

> there needs to be more communication across the grid than just responding to the locally observed frequency

Precisely - a grid-wide sync signal, modulated at high frequency.

You need to consider that a master clock signal transmitted through air will arrive at a different time than an AC power wave transmitted through a wire.

A millisecond latency will cause a significant out of phase shift to the AC signal too.

The AC data you want is already codified in the power signal

> A millisecond latency will cause a significant out of phase shift to the AC signal too.

Ooh, good point. But then, we have I think micro- or even nano-second clock sync over public internet now. Or use (reliable) GPS timestamps codified in the sync signal.

> The AC data you want is already codified in the power signal

You're right, but:

> there needs to be more communication across the grid than just responding to the locally observed frequency

Maybe large stations need a grid-wide sync signal that is independent of load, but small (e.g. domestic, community) suppliers just latch on the observed AC (which is easy for an inverter).

Not my area of expertise, but I think this is wrong? Modern PV inverters used in rooftop solar could easily start frequency synchronization autonomously, but are regulated to not do that autonomously in (at least) parts of the EU.

More importantly, according to the REE, PV played a mayor role in the current and successful black start.

How?

If the power is out, do you imagine that all the load that is out there disappears too?

If I'm in a blackout and put my multimeter into the grid socket, will the measurement be something that my local inverter can drive an AC load through with no problems?

And if that's not possible, how do we get an AC wave out to my 5000 local solar panel neighbors? And if we don't get any carrier wave to synch it to, then how do all these thousands of individual inverters decide on the unison synchronization necessary to all start jumping together?

And if they try and fail, will all the connected load accept a graceful out of synch mixed power noise at 0-220v until things properly latch together?

The problem isn't like playing a sine tune of acoustic sound that's merely audible to your local neighbor, who can dial in his own tune, and then daisy chaining this until it matches on a grid sized level.

There's a bunch of sources if you search for "grid-forming", "islanding events", "auto-synchronization" and similar terms in combination with PV inverters.

Do you mean solar energy sources? Wind turbines have plenty of "spinning mass"...

But not in the context of the grid. Most (all that I'm aware of) commercial grid-scale wind turbines have an output inverter. That allows the turbine to rotate at optimal speed for wind conditions, while still outputting the required 50hZ or 60Hz grid frequency. The output inverter has the same "inertia" as a solar inverter. From a grid stability and regulation perspective, they are the same.

Thermal and hydro plants use synchronous rotating generators that have physical inertia, and the generator rotational speed is exactly locked to the electrical frequency. (proportional to the number of poles in the generator).

More recent renewable inverters (solar PV and wind) and battery storage both have grid forming power control support. Term of art is "synthetic inertia" and makes spinning baseload obsolete. This has been shown under real world conditions most notably in South Australia with the Tesla Megapack based Hornsdale Power Reserve, which stepped in to maintain grid inertia when a coal plant exploded in 2021. For a more robust grid, you can deploy additional battery storage that can be called upon not only for energy, but also grid support.

(also replaces the need for synchronous condensers where they would typically be sited for grid ancillary services)

https://www.abc.net.au/news/2021-05-26/queensland-blackout-c...

https://www.pscconsulting.com/australia-leans-in-on-syntheti...

https://aemo.com.au/-/media/files/initiatives/engineering-fr...

https://www.ercot.com/files/docs/2024/01/12/Tesla%20BESS%20G...

Batteries on the grid are really more for load smoothing; demand/supply peak flattening and transient spikes. They're really too expensive to do what someone would think of as "storage".

Our recent govt installation is reported at $1.6bn for 2000MWh, which is incredibly expensive, and can't scale to run the grid, simply smooth load/demand diurnal curves.

https://news.ycombinator.com/item?id=43624390

https://news.ycombinator.com/item?id=43624308

https://www.vox.com/climate/408381/energy-transition-renewab... | https://archive.today/sZFxF

https://www.eia.gov/todayinenergy/detail.php?id=64586

https://about.bnef.com/blog/lithium-ion-battery-pack-prices-...

https://ourworldindata.org/battery-price-decline

The $1.6bn for 2000MWh is the current price, so that includes all recent drops.

I'm not saying "at some point" batteries won't be viable as grid "storage", I'm just saying today they aren't.

I'm really not sure about that. There is a recent "renovation" of a pumped storage site in Germany for more than 300EUR per kWh of storage capacity. Unfortunately I cannot find the original "Bürgerinformationen"-slides again. And it is a local, probably heavily politics-influenced provider, so maybe a lot of that is political landscaping. But I haven't been able to find a lot of anything with respect to storage prices beyond batteries. And I was really surprised that this renovation is the same order of magnitude than a battery installation.

You are still correct in the sense that storage is still incredibly expensive per hour of demand stored, but batteries may very well already be the cheapest option out there

German has no height. For their 11m pumped storage a battery may come close. But for the mountainous countries around hydro storage is much more effective and cheaper. They do have up to 1000m and more. And they are terribly effective to hold the peaks and spikes. Austria and Switzerland are getting rich selling their peak capacity to the big countries nearby.

Do you have some sources to their actual pricing? Some while ago I tried and came up blank. I assume your statements to be correct, because otherwise what would be the point of discussion, but I haven't been able to find any hard(ish) data

My source is a manager from a local energy company

Fair enough, unfortunately doesn’t help me. But thanks for the answer anyways

That particular reservoir may not have much elevation difference, but Germany is far from flat. There are many mountain ranges of a few hundred to over 1000 meters of elevation in the southern 65% or so of the country. They aren't impressive, but enough for pumped storage, at least if you only consider elevation differences.

This is our local Australian one, and the build price is an absolute joke which pretty much everyone puts down to incompetence, is 12bn. It's 5x the cost of the above linked battery for 120x the capacity (350GWh vs 2GWh). And this should be in operation for a lot longer than batteries.

https://en.m.wikipedia.org/wiki/Snowy_2.0_Pumped_Storage_Pow...

Reply on news.ycombinator.com