Cooling system leak led to Victorian Big Battery fire | Hacker News Reader

Cooling system leak led to Victorian Big Battery fire

esv.vic.gov.au

110 pointsbjowen80 comments

> "A new battery module isolation loss alarm has been added."

This is the worst fault possible in an EV 101 scenario right here (touch metal -> instant death), so I'm surprised that with all Tesla's experience, they didn't consider an isolation leak as a serious fault. And I'm surprised that they allow the battery monitoring system to be disabled during servicing, especially the IMD (insulation monitoring device).

I am constantly surprised how many systems have no isolation when it would be easy and cheap to add.

For example, typical solar inverter systems don't have isolation between the AC and DC sides. That means any chafed wire can be deadly. They do at least have leakage detection, but set at a 300mA level, which probably won't save your life.

Considering solar inverters already uses high frequency switching inside the per-string MPPT boost converter, it would have cost mere cents to put two windings on that inductor rather than one and get isolation with no downside.

The cynic in me says manufacturers don't add safety features not required by law, even if zero cost, because in the future laws might be updated to require that safety feature, preventing reuse of hardware already sold, which is good for business.

> For example, typical solar inverter systems don't have isolation between the AC and DC sides. That means any chafed wire can be deadly. They do at least have leakage detection, but set at a 300mA level, which probably won't save your life.

This is very yikes. I just installed such a system.

Do you know any way to test if that's a problem? And is it possible to fix without opening the inverter?

It's normally set that high by design because the solar panels have substantial capacitance to ground (they are large flat conductors sat just a few cm off the ground after all). The capacitance to ground wouldn't be an issue if not for the fact the inverter is non-isolated. That means current can flow to/from the grid to charge and discharge that capacitor, which is indistinguishable from AC leakage.

Some inverters will allow you to lower the leakage threshold (usually with settings in a service menu), but then you're limited to just attaching 3 or so panels or getting false isolation trips.

In my case the panels are, in fact, mounted on wooden brackets a good 1.5 meters off the ground. On the other hand, there's 20 m^2 of them.

Huh? 300 mA should barely give you a tingle unless you're talking about some very high voltages. What am I missing here?

A basic knowledge of electricity. 100-200mA will kill you.

https://academic.oup.com/ptj/article-abstract/46/9/968/46379...

I have heard people say "that can't be right, because a car battery can source 100+ amps and touching those terminals doesn't kill me!" The reason is because you have a high enough impedance that 100mA doesn't flow from a 12V source through your body.

"It's not the voltage that kills you, it's the amps!" Wouldn't power (W) (or joules) be a better way to phrase it? Because as you said, you can't have 100mA flowing through your heart unless you ALSO have stiff enough source to actually supply that current - which implies a high compliance voltage, can't escape ohms law.

No, it's not the power, either, as low resistant paths can be extremely dangerous (such as wet skin). Really, it's a 2 dimensional space that makes for a less catchy aphorism. "It's not the voltage that kills you, it's the combination of voltage and conductivity".

Honestly I kinda hate that saying. It's like "it's not the fall that kills you, it's the sudden decelleration on impact." For most situations people encounter, it's high voltage sources with more than enough power which are dangerous.

"The combination of voltage and conductivity" is ... current, no?

GP mentioned impedance, not resistance. You won't feel anything touching the 12/24V terminals in your car because they're DC. Your skin has a high resistance, but when faced with AC, has a low impedance. This makes AC much more dangerous, as you conduct 120V AC much better than 120V DC.

Even 50mA will likely kill you, as outlined on the "death graph" as we electricians call it. This is independent of voltage.

https://upload.wikimedia.org/wikipedia/commons/7/7f/IEC_TS_6...

Blue is harmless and imperceptible, green is harmless, yellow is harmful, red is fibrillation.

Edit: This illustrates why 30mA (or less) RCD protection is common (and required) in various jurisdictions, depending on the kind of circuits you're serving.

Right, and in particular, "Type B" RCD protection is required for solar inverters in various jurisdictions. These provide protection against DC fault currents.

> This makes AC much more dangerous, as you conduct 120V AC much better than 120V DC.

But at higher frequencies, you should experience the skin effect, right?

I can't answer that question, as I'm only an electrician, not an electrical or RF engineer.

That said, we're referring to power delivery here, and the highest frequency I'm aware of in this space is the 400 Hz power distribution systems used on warships.

"It's the volts that jolt, but the mills (mA) that kills" - The phrase a schoolteacher used when I was a lad.

V = I * R

The voltage across the heart will be rather bounded if you fix the current at 100mA, though. Hearts will not differ significantly from each other.

Not really, whether it takes 50V with wet skin or 500V with dry skin, it's the current that kills you, not the watts

Here is medhi sadaghdar (electro boom) putting 11v at 150A through his tongue https://www.youtube.com/watch?v=XDf2nhfxVzg

Embarrassingly, I have managed to shock myself several times during my life on 240V mains. Once quite recently while working on a light fitting at a friend's flat and not realising that someone had flipped the circuit back on at the circuit breaker!

That shock was certainly quite unpleasant, but despite touching my sweaty skin against live wires, it seems to have been far from fatal.

Surely the dangerous situations are live-to-earth shocks, which could potentially run across your whole body (rather than just a hand or finger, as in my case). But that's why we have RCD/GFCI protection, right? Those will trip with tiny fault currents, certainly much less than 300 mA.

Secondly, aren't "Type B" RCDs required on solar inverters? Those will detect and trip on DC fault currents as low as 30 mA, not 300!

you need "Lock Out Tag Out" lol

https://en.wikipedia.org/wiki/Lockout–tagout

> Embarrassingly, I have managed to shock myself several times during my life on 240V mains. Once quite recently while working on a light fitting at a friend's flat and not realising that someone had flipped the circuit back on at the circuit breaker!

Next time, I suggest using a breaker lockout [0] in conjunction with the local disconnecting means (light switch, which you can also lock out! [1]) to do it safely :)

[0] https://www.walmart.com/ip/MASTER-LOCK-493B-Circuit-Breaker-...

[1] https://www.homedepot.com/p/Ideal-Wall-Switch-Lockout-44-789...

> "light switch, which you can also lock out!"

No you can't! Ceiling light fittings (in the UK, at least) are often always live regardless of the switch setting. This is because they are typically wired in a "loop" arrangement with separate loop and switch cables.

Example: https://www.practicaldiy.com/electrics/lighting-wiring/light...

Well, the rest of the world outside of the UK, Ireland, and Hong Kong uses a safer method of electrical circuiting, one that wasn’t invented to deal with post-war material shortages.

> A domestic power supply voltage (110 or 230 V), 50 or 60 Hz alternating current (AC) through the chest for a fraction of a second may induce ventricular fibrillation at currents as low as 30 milliamperes (mA).With direct current (DC), 300 to 500 mA is required.

https://en.m.wikipedia.org/wiki/Electrical_injury

So you'd have to have a wire pressed against your chest with your back on the ground... Unlikely but yes possible hazard when working on solar panels

Think about the path the electricity would have to follow between, say, an arm and a leg, or even both arms.

These, and the GFCI outlets/breakers are calibrated against injury for adults. The thresholds are lower for children.

Mike Holt is an excellent instructor on the NEC and has quite a few videos that speak about this topic.

https://m.youtube.com/user/MikeHoltNEC

Aren't you are mistaking galvanic isolation with insulation?

Galvanic isolation is useful because it means that you have to touch 2 specific points on the affected unit simultaneously to die, rather than 1 point. Galvanically isolated systems can detect leaks on both the positive and negative sides by using a dual channel IMD. Any insulation issue is bad, but this method of detection can usually detect it before it's able to instantly kill someone per-se. Electric vehicles are an example of a galvanically isolated system (from earth) with dual channel IMD, hence you can have massive leaks such as 8-20mA before it becomes a safety issue.

Yes..., but what difference it makes to this fire?

Are you working in EE?

Galvanic isolation probably wouldn't have prevented this fire, it sounds like the cells just shorted together. They would have had to short over several cells, since a non-trivial voltage is required to break down coolant, so having maintenance contactors to break the pack up into 20V or so segments may have prevented this fire.

Using a mineral oil type coolant that is not electrically conductive (such as used in Tesla vehicles) would have also helped to prevent this fire.

The batteries should be galvanically isolated from the casing so that if liquid causes a current flow from any point in the pack, then the maintenance contactors can be opened and the opposite side current path for the leak is interrupted. We can only assume that they are already (galvanically isolated) since anything else would be bizarre.

I spent a bit working on various power supplies and sadly isolation is non-trivial nor cheap. Adding isolation (transformer vs inductor) requires changing control signals, control mechanism, size. It costs far more then mere cents, and loses some efficiency.

An extract from the summary page suggests a few things went wrong, but it seems like the fire was caused by liquid and electricity mixing:

> ESV found a Megapack cooling system leak caused a short circuit resulting in overheating that led to a fire in a nearby battery compartment, which consequently damaged two Megapacks.

> There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive.

> A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.

I would assume that each megapack would have zigbee for telemetry which should be always transmitting, even if the pack were in the off state.

And the cooling system should be continuously monitored for faults. Not just inspected after filling.

That it's hardware instead of software doesn't mean it can't be hacked together.

Thermals are the number one aspect of battery engineering. For something that is to run under load in the Victorian heat, even more so.

This isn't some weekend hackathon contest, Tesla is supposed to have the worlds best engineering.

https://en.wikipedia.org/wiki/Tesla_Megapack

Why would Tesla have the "worlds best" engineering? I don't think I have ever seen anyone ever claim that except for perhaps Tesla PR representatives. How would you even measure "best" in such a field? Cheapest? Best quality? Best value-for-money? Safest? Highest power density?

I am very surprised in this conclusion... Normally if you have two high voltage wires being immersed in water, a large current flows, vaporising the water, and then the current stops when all the water has been evaporated.

For the current to flow for long enough to heat the battery packs to cause a fire, there must have been a jet of water flowing for a long period of time. And no fusible links between that place and the battery.

It really seems like either very bad luck, or bad system design, or both.

Off topic: can we do a floating (mega-)mega-pack, and use passive water cooling? This should help costs, efficiency AND prolong life time

Of course issues with sweet vs salt water

Sure you can do it, but it won't be reliable or cost effective.

Passive water cooling in natural water tends to lead to seaweed and all kinds of sea creatures blocking up all heat exchange surfaces.

In the case of a megapack, cooling isn't critical to safety - as long as you're happy to stop using a battery when too hot, it won't get hotter. The reason to cool is is so you can keep using the battery.

The article puts it this way: "ESV found a Megapack cooling system leak caused a short circuit resulting in overheating that led to a fire in a nearby battery compartment." Nothing here implies that the damaging heat generation occurred at the location of the short circuit. It could have been at any point in the new current path created by the short circuit, depending on its resistance and the share of the overload current it carried.

A cooling system for something that's on fire sounds like exactly where there should be a jet of water flowing for a long period of time :)

offline monitoring is what caused tchernobyl, when will ppl learn ?

Were you going for Чеrnobyl?

Were you going for t̷̨̆̏c̶̥̎h̸̡̄̅e̴̲͖̓r̶̨̎n̵̥̹̆̚o̴͇̐͊b̸̙͖̑̈́ỹ̴̲͛l̸̢̞͊?

Latinized without the "t", as the Germans do, is closer to the true pronunciation.

> "no active monitoring"

What the actual... how is something like this even allowed to be operational without monitoring?

It wasn’t operational, it was being comissioned.

You folks are acting as if some huge crime against humanity happened here. In reality nobody got hurt, the company who messed up will pay to put it right. They already indicated (in the linked report) that they learned from the incident and changed multiple things in their systems and procedures as a result.

It's a live (active) power source even if it's being commissioned. I don't think it's a crime against humanity but it sure created pollution and waste.

This. Lessons learned, improvements implemented.

Nobody ever got internet virtue points by keeping their cool. You gotta be Outraged(TM).

> It wasn’t operational, it was being comissioned.

You're telling me that you don't see a problem with the idea of starting up a bleeding edge system intended to store incredible amounts of energy without the monitoring systems connected?

In general if you don't have a way to monitor an industrial-scale system you go in to your fail-safe state.

> intended to store incredible amounts of energy

Each of these Megapacks can contain 1.5% of the energy contained in a petrol station. :) Let’s not get ahead of ourselves, these are realy nice and big batteries. That being said the energies stored in them are far from incredible.

These boxes were by all accounts manufactured in Nevada. They have been moved all the way to Victoria, lowered to the ground and hooked to the site’s wiring. They were not “charged” yet. The next step as part of the comissioning was to hook them up to the monitoring system before they are checked out.

No matter when you hook them up to be monitored there is a step just before it when they are not monitored yet.

> In general if you don't have a way to monitor an industrial-scale system you go in to your fail-safe state.

That is indeed a great idea. And if you read the report you can see that after they hooked up each box they put them manually into “maintenance” mode. This mode de-energises all systems in the box. It totaly makes sense that they thought this procedure is the correct one to reach the fail-safe state you mention. Turns out they were wrong. They learned this now and changed their procedures.

Reading this it’s a classic systems failure: multiple defensive barriers gone wrong through poor safety planning. It’s reminiscent of the Victorian ESSO gas plant accident https://en.wikipedia.org/wiki/Esso_Longford_gas_explosion

In what way is it reminding you of that incident?

The paralels I can see:

- both involve energy installations.

- both happened in the same state.

The differences I can see:

- one involves a gas utility the other an electric one

- one has killed and hurt workers while the other did not killed nor injured anyone.

- one happened after several years of operation, the other happened during installation.

- one left the state without energy service for 20 days, the other did not affect service at all.

Don’t get me wrong. I love reading about how complex things go wrong, one of my favourite pastimes. So I thank you for your link. It just doesn’t seem that similar to me. Not sure you would be citing the same case for example have they not happened in the same state.

If I have to choose a failure which the Big Battery fire reminds me of I would rather choose the Florida International University pedestrian bridge collapse. Why? It seems these batteries had adequate safety functions and monitoring planned in for their operation. Just these systems were not activated during the installation in the right order. Similarly with the bridge it seems they did calculations with the full bridge in place, but nobody seems to have checked if the intermediate steps during the construction will stay up too.

Another similarility (to my untrained eyes) is the aim to install a system fast. The bridge project wanted to be a flagship for accelerated bridge building, and the big selling point of these Megapack installations is how fast they can be comissioned. With the bridge project we know that this demand for speediness was a contributing factor. Was it maybe also with the Big Battery fire too?

>In what way is it reminding you of that incident?

It's just in group signaling. "Hey this reminds me of <vaguely related thing>" is a more polite way of saying "look I know about X, I'm like you, gimme dat virtue points"

You see this on literally every virtue points (i.e. up-vote) based form of social media that's big enough that any given participant can blend in with the crowd.

The behavior is endemic these days you can't even post a picture of a rotten and falling down deck on the internet these days without some jerk derailing everything by dropping a link to the wikipedia page on the hyatt regency and then several more riding their coattails with quotes copied from the page.

He probably doesn't even consciously realize he's doing it.

So, they'll have to take note from their nuclear colleagues' playbook? About passive safety.

Why? The factory didn't burn down, and there was no widespread radioactive contamination.

It's a battery pack, not a stash of U235 above the critical mass. Every recycling facility is more dangerous.

Because fires are bad, and they can kill workers and shut down facilities?

Well, the conclusions seem to suggest they have passive safety.

"The affected Megapacks failed safely despite total loss."

I think that people understand the level of engineering and technology that goes into batteries and their designs these days. There’s a ton of talent, a ton of continuous innovation, it in the end it too often gets relegated to a black box for energy in and energy out. Which is in some ways a testament to the quality of the engineering.

for people who might say 'oh, but these batteries are so dangerous!'

How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year?

I imagine the largest insurance companies have reasonably good data on this as a cause of total-loss of a vehicle.

How many times have you seen in person, or seen a photo of a burned out RV that somebody pushed too hard up a mountain pass without keeping an eye on the engine temperature?

Read the doc, this has nothing to do with lithium batteries as a component. This was a System Requirements/Hazard Analysis failure.

The fault lies square with Tesla. No amount of RV analogies can dilute this.

The point was that plenty of other energy dense things with radiator-based liquid cooling loops lack proper monitoring systems.

Not excusing Tesla but the actual number of heat producing things that catch on fire every year (in general) because of cooling pump/cooling loop failure is quite a lot. In many categories of equipment.

> How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year?

Almost none. Overheating is highly unlikely to cause a fire. Fuel leaks cause fires.

You have a gross misconception of the relative temperatures of components in internal combustion engines, and you're using a faulty conclusion based on the same misunderstanding as supposed supporting evidence.

I assure you an RV or other gas/diesel vehicle does NOT suddenly go "pop!" and produce an engine fire when the needle reaches the right hand side of the temperature gauge.

The temperature gauge warns you when the coolant is at risk of boiling; it's not a warning that the engine is about to spontaneously combust.

right, that's why every burned out at the roadside RV I've seen in my lifetime (probably more than a dozen now) was on the upward slope of a high mountain pass, or at the peak...

one of the points I was trying to make is that people don't maintain liquid cooling loops in general, and that's one of the more common instances of it.

do you live somewhere flat?

go ask the towtruck drivers who work the coquihalla highway in BC how many vehicle fires they see every year, and where those vehicle fires occur.

Overheating ICEs blow head gaskets and warp cylinder heads, they don't generally start fires.

Fuel leaks and/or exhaust system failures, especially on turbocharged vehicles however...

From the title I first thought this would be about a battery fire that happened in the late 1800s.

In a much cooler reality, it is, and it’s the other Tesla.

> There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive.

A short in an offline device? That seems weird.

> A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.

So a new pack was installed and connected but not monitored? What is the rationale behind that? Failures shouldn’t happen to new devices?

Department #1 does the hardware setup, signs off their forms and the next day department #2 integrates the new hardware into monitoring?

> So a new pack was installed and connected but not monitored? What is the rationale behind that?

Any time a house is built, there's some time between the ceiling going up and the fire alarm installation being completed.

Nothing wrong with that - if it's done properly and carefully.

> Shorter connection times to the SCADA system to help alert Tesla with specific alarms.

Interesting that even after installation Tesla are still involved in operationally managing the megapacks.

Tesla’s model is they both sell the Megapack system and then orchestrate it over its lifetime using their Autobidder platform. The owner is responsible for financing, integration with on site generation, and working with the local grid operator.

Short circuit that led to a fire in an electronic component

Should've used metal flanges instead of plastic heh

how much would you ask to be paid to pressure test the coolant system of a giant bomb in situ?

One advantage of LFP batteries is that they can survive much higher temperatures without degradation.

If you can make battery packs air cooled, that's a big, big weight saver.

Reply on news.ycombinator.com