Spain-Portugal blackouts: what happened | Hacker News Reader

Spain-Portugal blackouts: what happened

theconversation.com

56 pointsVagabundoP62 comments

This article starts by saying "we dont know what actually happened here, or why" and then goes on to make various insinuations and suppositions, and then proposes policy based on that.

This not useful. More data is required.

"The diagnosis

This unusual situation points to a perfect storm of poor grid management and inadequate connections of solar facilities to the grid, as well as other unknown faults. In my opinion, there is a good chance ..."

We don't really know, I don't really know, but I'm going to write a long post about it.

Agreed. He's just running off press reports.

We haven't yet seen the sequence of breaker trip events. We haven't yet seen the graphs of power flows at tie points, and of frequency and voltage error.

The closest thing to a useful fact visible from Google seems to be: "The network lost 15 gigawatts of electricity generation in five seconds at around 1033 GMT, the Energy Ministry said on Monday evening, without explaining the reason for the loss." That's not a cause. Losing 15GW all at once, more power than any one plant generates, indicates some previous event had caused breaker trips somewhere. No indication of the previous event.

We do know this wasn't a supply shortage. There was plenty of generating capacity online.

Here's an analysis of the US Northeast blackout of 2003.[1] Until we start to see that level of detail, it's just blithering.

[1] https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

Unless someone diverted all that power for some purpose.

Classic scifi might be a time traveller trying to get home, or someone opening an interdimensional gateway, or maybe an AI becoming conscious and needing the power to ascend.

radio controlled on-load tap changers can be hacked because radio link and protocol running on it is not authenticated / encrypted

> This not useful. More data is required.

I don't agree. You do not need a thorough root cause analysis to acknowledge the fact that a) some invariants in the system were violated, b) some of the people accountable are already making claims that fly in the face of reason.

Also, Spain has a regrettable track record of covering up the responsibilities of state institutions in major disasters.

It makes me realize how ignorant about power grid I am. It is kinda unfathomable to me, that 2 countries completely black out for 10 hours, then somewhat "magically" go up again, and a week later nobody knows what went wrong and how it got fixed. Or maybe someone knows but doesn't want to say? Anyway, it makes an impression that the whole power grid is this magical black box that somehow works on its own, breaks, fixes itself, and humans can only pray gods to let them have good harvest and uninterrupted electricity next summer. Maybe it's time to renew sacrifices at the temple of Jupiter or somebody else.

Yeah. This was a regretful click.

> At that time the price of electricity on the official market was in the negative at around -1€/MWh.

> Current evidence therefore points to a problem in the synchronisation of the grid. All sources feeding power into the grid must be synchronised at the same frequency, 50 Hertz. To facilitate this synchronisation, stable base-load power is required, which is normally provided by nuclear and other large gas and hydroelectric facilities. These sources act as a natural buffer against disturbances

So:

- price is negative, so solar automatically disconnects not to pay for providing electricity

- nuclear is overloaded at unexpected time, it also disconnect due to safety.

Seems like a bug in accounting software for solar power plants. It disconnected too many power plants too quickly! I bet like 40% of solar plants are using the same software for managing connectivity.

I've seen a few articles of The Conversation/Europe on HN, and it seems an interesting news source. Wikipedia also gives some insight. I've just subscribed.

Does HN know about their agenda's, biases, blind spots. Not necessarily to shoot them down, just to be aware of them.

https://en.wikipedia.org/wiki/The_Conversation_(website)

Yeah they seem good:

https://mediabiasfactcheck.com/the-conversation/

> However, variable renewable sources, such as solar photovoltaic, do not have this capability. They generate direct current which is converted to alternating current at 50 Hertz, but they cannot react automatically to frequency variations.

This part didn't make sense to me.

The rotating machinery of hydroelectric, or nuclear's steam generators, produce the frequency of AC in synchronization with their mechanical rotation, but the inverters converting solar's DC into AC are making a purely synthesized waveform under control of high speed digital electronics.

I would think they should be the most able to modify the AC sine wave they are generating.

Maybe someone can comment and correct my logic...

I remember reading an article about exactly that some time last year: https://spectrum.ieee.org/electric-inverter The technology is grid forming inverters, unlike the normal grid following converters. I'm definitely not an expert, but I think the part you're quoting is hinting at the following:

> Grid-following inverters operate only if they can “see” an existing voltage and frequency on the grid that they can synchronize to. They rely on controls that sense the frequency of the voltage waveform and lock onto that signal, usually by means of a technology called a phase-locked loop. So if the grid goes down, these inverters will stop injecting power because there is no voltage to follow. A key point here is that grid-following inverters do not deliver any inertia.

From the linked ieee articale ..... Not yet commercially available ....

For photovoltaics and wind, grid-forming inverters are not yet commercially available at the size and scale needed for large grids, but they are now being developed by GE Vernova, Enphase, and Solectria......

Stick a battery with your renewable plant and they are available. Like the article also says.

You’re not wrong, it’s not that they can’t, it’s just that they currently don’t do it well. The concept is known as as grid forming if you want to read about it.

Thanks for the replies!

I'll do some reading on "grid forming".

You are very right about electronics being much more capable than electromechanical systems. We can sychronize the frequency much better with GPS for example. This wasn't an issue for small scale solar, so apparently nobody implemented it to keep things low cost. We are at a point that we need these.

The beauty of it is the electronics can be connected and controlled much better with each other too, so more optimized control algorithms for supply and demand balancing is possible. Bad thing is, this will incentivize to solve problems by over the air updates. As we know software reliability has gotten so much worse, which makes it more prone to bugs being pushed. Obe should expect weird outages due to someone pushing something buggy.

I'm having a hard time understanding how we can move the bulk of our energy to renewables without some kind of massive storage system. The wind can die for long periods. It can get abnormally cloudy for long periods. There's nighttime and winter.

I'm no expert, but is there any storage system that is practical that can store the amount of energy we'd need to have most (all) electricity come from renewables?

Some hydroelectric plants can pump water back up and both absorb excess supply and provide stabilization during demand spikes.

I have no idea if it’s feasible to fill longer-term gaps during extended cloudy/windless days and nights using that alone, though. Other than that, there are already large-scale battery plants deployed in some cities.

Another approach is to control the demand side: When air conditioning or heating with electricity, minutes usually don’t matter, and dropping/providing extra load at very short notice should be feasible in a smart grid.

It is entirely infeasible to used pumped hydro alone to solve our energy storage problems. They’re a great component of the solution but they’re big expensive environmentally disruptive projects and there are limited suitable sites.

They're also quite inefficient (75% ish, which is not horrible, but still something) as you lose energy both in pumping and generation.

So the more you use them, the more energy you need in the first place, so you try not to use them too much. And the worst thing for a capital intensive business is to not get used much. Even with the arbitrage advantage, it takes a long time to pay it off. The grid may pay a "retainer" to sweeten the deal but you can't just build more and expect them all to get that benefit.

It's similar to the problem that "use excess energy to electrolyse water into hydrogen" has: no-one running a multi-billion electrolyser really wants to run it a few hours a day only when the grid is oversupplied. And in top of that, risk being cut off at the knees decades before breakeven if someone comes along with a cheaper/more profitable way to deal with the oversupply.

That’s why Grid Storage is a major topic in power generation and renewables. The technology and economics are evolving. A lot of people are working on finding a storage solution that is both durable and economical. One solution is probably not going to solve for all situations.

In the meantime, load sharing is used to distribute power across a power grid for cases where one area has more power and another area needs power. For instance, wind may die out in one area but over larger areas there are usually still areas with wind. It’s harder with solar, of course.

RE "....load sharing is used to distribute power across a power grid ....." TO distribute power across the grid - on a large scale fashion , would require a new power ( very expensive) power distribution network. The base problem with renewable solar and wind is its not reliable. an have outages for SEVERAL days

We can do the bulk of it, i.e. more than 50%, by geographic and modal aggregation fairly easily. Doing all of it from renewables is still an unsolved problem that would require better energy storage.

There's always wind or sun somewhere in Europe, there has never been an hour without both across the entire continent in the last 30 years.

> There's always wind or sun somewhere in Europe.

Not necessarily enough wind to be useful. If you look at graphs of wind turbine output across entire large power grids, you see 4:1 variations in a day over the PJM and CAISO regions.

That's irrelevant to this blackout, anyway. This blackout occurred during a period when supply far exceeded demand.

Having enormous synchronized grids helps a bit with that.

It’s highly unlikely to be abnormally cloudy/calm over the whole of Europe, for example, so Europe’s large grid (https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...) can be used to move electricity from where there’s excess energy from solar or wind to where there’s a deficit.

That requires large capacity connections, though that aren’t everywhere yet.I understand electricity was restored in the south of France much more rapidly last week than in Spain and Portugal because it’s much better connected to the rest of Europe.

The reason why the Iberian grid is not better connected to France might have something to do with French Nuclear not wanting to be bankrupted by cheap renewables from southern countries, this was the word on the street a ehi'e back, when the push for renewables in Spain in Portugal was starting up.

This is a political problem, having Mitteleuropa depend on power from Putin is apparently OK, but on lazy southerners is verboten. Hopefully this blackout changes that (but I'm not holding my breath).

I would think the original reasons are the Pyrenees and the relatively low population density around it.

If you look at the European road and rail networks, you’ll find France and Spain aren’t well-connected, either.

The network connection doesn't really depend on the population density around there. I'm not saying it's a simple technical problem, but it's not insurmountable either.

We have very detailed continental weather readings for over a century and can build very good statistical models. So you just ask a supercomputer to find the lowest cost solution for 99.99% electrical coverage without using any carbon sources.

It's quite possible -- over the previous 30 years we've never had an hour where there was no wind & sun anywhere in Europe. So we could theoretically solve it without storage, but that would require an insane amount of overbuild and interconnect. Imagine supplying all of Europe's electricity needs with just wind power from the south shore of Ireland.

Adding a few days worth of storage reduces the interconnect & overbuild needs dramatically, putting 99.99% carbon free within the realm of the possible.

> So you just ask a supercomputer to find the lowest cost solution for 99.99% electrical coverage without using any carbon sources.

Sure... but my expectation is that that 90% solution would include an unreasonable amount of storage (assuming you want most/all renewables). I guess was my point.

> It's quite possible -- over the previous 30 years we've never had an hour where there was no wind & sun anywhere in Europe.

But "some" wind or sun is not enough to support the whole grid.

I wouldn't assume all or even most renewables. I would assume you'd continue to use existing hydro and nuclear. The nuclear would help a little, but the hydro helps a lot. It's capacity can be saved for times when wind and solar are low.

In that scenario 90% is very easy to hit. A couple of hours worth of storage, maybe.

> But "some" wind or sun is not enough to support the whole grid.

The less there is, the more you have to over build to compensate. Doesn't make it impossible.

This is my favourite tech as its cheap and safe:

https://en.wikipedia.org/wiki/Iron_redox_flow_battery

Its commercially available right now as well.

Its not as efficient as lithium or as energy dense, but that doesnt matter if you're putting them in wind/solar farms.

> To facilitate this synchronisation, stable base-load power is required, which is normally provided by nuclear and other large gas and hydroelectric facilities. These sources act as a natural buffer against disturbances, helping to keep the frequency stable in the face of sudden changes in generation or demand.

In theory, it seems like you could instrument a photovoltaic array to carry some "inertia" with the right control system.

If you need to feed power, you run some power point tracking algorithm, and if you need to consume power, you just overbias the cells and heat them up.

Yep it’s called synthetic inertia. Solar and wind can both offer it. Wind can actually offer loads of it it’s just not as easy since it’s not synchronous.

In a system of micro-inverters, they need something to synchronize with. There needs to be a "truth" reference, so they can push their power onto the grid by slightly leading the phase of that.

If some critical mass of PV micro-inverters exceeds the traditional generators, they'll push so hard that the grid itself will change phase, and blackouts are the result.

One possible solution might be to use a better oscillator in the micro-inverter and limit the rate of phase shift. Unfortunately, the grids of the world have been moving in the opposite direction and now allow more drift than in the past, so where do you draw the line?

https://medium.com/@brandonvar/power-grid-stability-issues-a...

apparent vs real power is your answer.

> they'll push so hard that the grid itself will change phase

That's a thundering herd problem: where all invertors have the same set-points and they all are synchonised to push in the same direction at once.

In networks, thundering herd problems are fixed by given each sender different random delays.

For power networks we could choose statistical methods to get individual solar generators to lead or lag so that the frequency becomes an aggregate vote.

Given that part of the blackout was due to large amounts of solar going offline at the same time, it's possible that all invertors with common software were tripped at the same set-point.

So trip conditions also need to be randomly fuzzy. E.g. if frequency drops below 49Hz +/- random spread of 0.5Hz.

Although it's difficult to match financial incentives against random variations (individual generators are incentivised to power outside of boundaries to keep getting paid, and a trip event can be expensive - due to restart costs).

Electricity market design is hard because the design needs to be resilient to perverse incentives.

Conceptually, your idea seems reasonable, but because the goal is to put the power from the solar micro-inverters onto the grid, they must lead the grid phase by a few degrees. If they lag instead of leading, they are draining power from the grid and not supplying power into the grid.

People with PV arrays want to make money by selling power into the grid. Perhaps if they were a little less greedy, they could back off the phase difference if they detect the grid phase drifting from too much "pushing." After all, they can't sell any power at all while the grid is down.

you have panels connected to grid thru inverter and that can modulate output in any way needed in few milliseconds.

same as inverter in electric car providing power to motors.

or inverter providing power to coils in your loudspeaker/ headphones.

inverter can adjust phase, voltage, frequency. it can means it is job of inverter to provide that in normal operation. that is why it is there in first place.

Would it not be better to have power sinks at every solar/wind farm and that will stabilise everything. Big cheap batteries.

> much of the available energy was being used to pump water from low lying river basins into reservoirs – the only practical way to store energy on a large scale. However, this capacity has a limit and, with the reservoirs almost full, it cannot continue to be stored indefinitely.

There has been unusually heavy rain in Spain for an unusual length of time now, and reservoirs are well beyond average capacity.[1]

I wonder if this anomalous situation has anything to do with the blackout. Maybe the inability to continue pumping water led to the shutdown of the solar, maybe for economic reasons as is suggested in some other comment?

[1] https://www.embalses.net/

Can the excess water be dumped out to waste generating just enough power for grid stability?

Fu....g Ghost Riders in the Sky happened.

Yippie Yah Hoo, Yippie Yah Yay!

Try https://www.youtube.com/watch?v=0TPOj4CTGKs if you don't get it ;-)

Systemic failures, too much power generation and not enough stability.

Pretend you're surprised when they ramp up on blocking the sun.

Edit: Downvote this if it will help you remember when they do it soon. But by downvoting you are hereby agreeing to not to act like you are surprised when it happens.

In my zone we got the power back two hours after the blackout.

Sure more data is required, but the article is not useless. This shouldn't have been flagged.

I stopped reading after this, sorry you totally lost me, im blocking this website:

"However, variable renewable sources, such as solar photovoltaic, do not have this capability. They generate direct current which is converted to alternating current at 50 Hertz, but they cannot react automatically to frequency variations. "

Nonsense. Go to any solar installer and ask him about this capability. or go to your roofs PV inverters manual and look up section about inputting "grid parameters".

Writer is either total ignorant or bad actor.

I'm fairly knowledgeable on this topic, and I disagree that the author is ignorant or malicious. I think he was just trying to "dumb it down" for the masses.

https://theconversation.com/profiles/j-guillermo-sanchez-leo...

I read quite lot about connecting Ukraine power grid to EU. Small variations in frequency and phase are quite real problem. It is quite a big problem to synchronise large international grids, and tiny PV inverter is really not build to do that!

Plus from manual, most AC inverters have parasitic power factor on AC side. Some part of power inverter sends to grid, can be out of sync junk!

Large spinning turbines with generator, are capable of absorbing out of sync junk to huge extend. But that may put turbine out of phase slightly.

Current power grid was build around huge wheels spinning at the very same frequency.

I asked O3 for an analysis on how much battery power would have been needed to prevent this from happening again. The TLDR is: 1. The grid went dark because 15 GW of generation vanished in five seconds. 2. Need to bridge for 10mins -> 2.5GWh needed, with margin 3-4GWh 3. Need a 6C system, 15 GW / 3 GWh 4. Cost ≈€3 B all‑in, or €55 per Iberian resident

However the lead time is going to be an issue: Need 7500 Tesla Megapacks and thats 4‑5 months of combined annual capacity for Tesla. Currently there is a 18 to 24 month backlog. Then there is 6-9 month installation time.

So 3 years to fix this, min.

It's a shame that these renewable energy transitions are managed so poorly. This was a predictable outcome and it's negligence that they went ahead with switching so much of the grid to renewable without accounting for the needed for stability. It hurts the mission of switching to renewable energy.

An interesting calculation, but why do you assume all the battery storage would have to come from tesla?

I'm looking at whole-house battery backup for a little construction projct underway here. This is a boom market at the moment, and there are many other manufacturers in production.

I can only assume this is also the case for industrial scale storage...

Exactly finding other non-lithium battery tech to use for these just makes sense:

https://en.wikipedia.org/wiki/Iron_redox_flow_battery

Sounds great, but when will that be commercially available at scale? I didn't find any installations in a quick search.

https://voltstorage.com/en/solutions/the-voltstorage-iron-sa...

These have commercial products already I believe. I'm not sure if they have any installed yet. They have moved from vanadium batteries.

Thanks for the link. Looks promising however it's years away from solving the problems that we have right now. The point I was trying to make above is that even proven solutions with an installed manufacturing base require years of lead-time. Renewable energy projects that don't include battery stabilization are reckless.

Here's the estimate for the lead times for the top grid suppliers. Tesla is the largest and fastest. So no need for Tesla, but they'd be the first according to this AI researched summary. Tesla: 12–18 months (fastest). Tesla’s 40 GWh Megafactory, vertical integration, and proven deployment speed (e.g., Hornsdale in 100 days) make it the leader. A 4 GWh order could be split across multiple sites, leveraging prefabricated Megapacks and software to streamline commissioning. CATL: 18–24 months (close second, if partnered). CATL’s 300 GWh cell capacity could produce 4 GWh in weeks, but integration by partners like Fluence or Sungrow adds time. Direct TENER deployment could approach Tesla’s speed if CATL scales system assembly. BYD: 18–24 months. BYD’s 100 GWh capacity and LFP expertise are strong, but slower project execution and less software focus trail Tesla. Fluence and Sungrow: 24–36 months. Both lack proprietary manufacturing and rely on external cells, extending lead times.

about the battery capacity needed. It needs to be fully CHARGED too - in reserve ALL the time

Why the down votes? Blackouts are not going to help the world switch to renewable energy. Battery backup can fix the problem but only if the people pushing for renewables have realistic expectations and plan accordingly. This is a failure and didn't need to happen.

Reply on news.ycombinator.com