Hacker News Reader

Story time. Dad worked briefly for the Eveready battery division of Union Carbide, and they had a plant in low-labor-cost (for the time) Asheboro North Carolina. They had a problem with their then-new alkaline battery line where the batteries were exploding, and dad (a chemical engineer) was sent to find out why and fix it. He was there for several weeks looking over the process and line, trying various things to solve the problem.

Eventually his boss came for a visit and dad was explaining what he had done as they walked through the factory floor. Just as they passed a stack of batteries, one of them exploded -- which then set off a chain reaction as the released electrolyte shorted out adjacent batteries. Listening to this series of small explosions, his boss accused him of a setup. Nope. Just coincidence.

I think he said the problem was traced to the water supply and how the city had been varying the amount of chlorine in it.

The most interesting battery they made was for the Union Pacific Railroad. They had signal lights to run out in the middle of nowhere, and they had to run unattended through the Rocky Mountain winters. So they made this massive battery that would be hoisted by crane into a buried vault by the side of the tracks, to be swapped out 5-6 months later.

Wouldn't signal lights require a network to coordinate? That would mean there's some kind of connection to it, so why would it need a battery?

I believe trains would flip switches as they rode along the rail.

More than likely it used Automatic Block Signaling, invented in the late 19th century. Train detection is done by looking for the electrical leakage across the rails through the steel wheels & axles of the train.

https://en.wikipedia.org/wiki/Automatic_block_signaling

It can't handle as many trains as centralized control, but it has the advantage of not needing a network in remote areas - just buried wires. It's the equivalent to relay logic.

(I finally get to put some of my teenage model railroading knowledge to work)

This is the same signaling used by most of the NYC subway system and (if I understand correctly) is the reason they can't simply add more trains to the tracks despite the current overcrowding at times.

https://en.wikipedia.org/wiki/Signaling_of_the_New_York_City...

Thank for the insights!

I think the damage Samsung suffered was simply too high, so they had to release information that would show that they really solved the issue. Otherwise there would be permanent brand damage if people suspected that they don't know what was causing incidents. So what we are seeing is actually some very well thought out PR.

Seconded with the "well thought out PR" part. The piece was worded well, all fault points away from Samsung.

I would be curious to see how they would have handled the PR if Samsung were at fault. I would like to think the exact same.

The original batteries were manufactured by a Samsung subsidiary.

you misunderstood, which means Samsung misinformation worked rather well.

The FULL blame is on Samsung and its subsidiaries.

Seeing as how Samsung did produce the original faulty batteries, you are seeing exactly how they shift the blame away from themselves. Instead of calling out the bad batteries they made, the specifically avoid discussing who made the bad batteries so it looks like they did a lot of investigation into the cause but it doesn't look like they were at fault.

My question is why wasn't their QA process catching this in the factory? Especially after the first recall. It seems like they didn't bother making a special effort to verify the second battery source was producing good batteries.

(I honestly have no idea, I'm not involved in manufacturing.)

Samsung SDI is separate enough from the smartphone makers that I'd guess it just wasn't prioritized as much as it should have been. 2.5 million phones were sold, at 14 watt hours per battery that means that the Note 7 batteries would only be a few percent of their total production (35 MWh of Note 7 batteries produced.)

Catching problems like this can be pretty tricky and at the end of the day it's also possible that a few people getting lazy and programming the machines wrong led to this.

I agree. They didn't talk at all about any QA processes that should have or would have caught issues, or what they have instituted since.

It's worth noting that the number of actual failures is measurable only after millions had shipped. 10-100 dppm is a reasonable guess (~100 phones out of 2.7M shipped) with relatively low "infant mortality" until they were compressed into a phone and used (and abused as normal).

As a tester, you build a 10k ($10M) unit test run and none of them fail, and then another 10k ($10M) run and again none of them fail... statistically you might not expect reliably catch a single failure until you run 10 times.... that's a $100M test you need to get your boss to sign off on.

Note that when you get a failure it will burn up and you won't be able to identify the cause.

Now, if you knew the failure mode, you could test for it directly and you wouldn't have to waste an entire assembled phone in a destructive evaluation, but that's hind sight. The failure defects they see seem like pretty standard problems, but other than sampling, I'm still not sure how they are going to catch them in manufacturing.

What's amazing is that more batteries don't go up (the original Prius was notorious).

Missing insulation especially sounds like a problem that would be in a whole lot of batteries, and should be easy to find in a test run.

It was only a factor after the case had been partially damaged. Let's hope that the testing procedure will start to inflict similar damage to catch this sort of failure more quickly.

It only causes a fire if there are other factors. But it's very easy to find on an intact battery being examined for flaws.

What's the original Prius issue?

Corrosion from leaking electrolyte would eventually short the battery terminals to the protective shield around the battery cells. At that point the battery would catch fire. It just happened to me 3 months ago, 2002 Prius on a battery that was serviced for the recall in 2004. Electrolyte continued to leak, obviously. :/ https://twitter.com/davemethvin/status/782610209525170176

> My question is why wasn't their QA process catching this in the factory? Especially after the first recall

There were 2 battery suppliers, the initial recall was done because batteries from the first supplier (a Samsung subsidiary) had structural/design defects.

To make up for this lost capacity, they ramped up production at the second supplier without much of a heads-up. Unfortunately the Quality Control was not adequate for the higher volumes, resulting in well-designed but sloppily made batteries that were also prone to explode. The problem in the second case was not a structural one, but in the manufacturing process.

Thanks for sharing your thoughts with us. Slightly off-topic, but perhaps interesting enough to keep here: You state that battery fabs are very opaque. Could you tell us some things we might be surprised to learn about battery manufacturing or technology in general? Thank you.

You're welcome!

Unfortunately, I'm mostly on the supply side of the battery equation, consuming those few PDFs that are smuggled out.

The most surprising thing, to me, is that battery tech moves in a zig-zag shape. The best battery that money can buy, for two of pretty much the most important applications (can't elaborate further, sorry) was only made from ~2012 until ~2014. A lot of really expensive things that need that battery are now running off of old stock, which will of course eventually run out. The best comparable battery made in 2016 has less than half its performance.

Another example of this is energy density. For practical purposes, the best density you could reasonably buy was fixed around 90 Wh/kg for a while (until about 1997), then suddenly jumped to ~180 Wh/kg (~2002), then went in 2006 to ~260 Wh/kg... and sat there, where it has sat for eleven years now with practically zero improvement. [0] If I teleported to today from 2006 I'd expect to have some fantastic 400 Wh/kg batteries that would enable all kinds of new technologies... but nope, the best energy density you can get is literally the same as it was eleven years ago.

Worth noting that since 2006 discharge rates of high capacity batteries have gone up, to be perfectly fair. Still though, energy density remains stagnant.

[0] Looking at graphs on Google Images will give you the wrong idea. They typically show the best batteries that year, not necessarily ones you can actually buy, and the numbers are often pulled out of thin air. This graph [1] is the least worst, but that's not saying much.

[1] http://media.greenmonk.net/greenmonk/files/2015/11/Li-IonDen...

Thanks for posting this. A totally unrelated question, if you don't mind answering. Given all this, do you believe that Tesla is changing the battery market dynamics with their GigaFactory. Musk has claimed that their plan is to increase the energy density, but hasn't given out any specifics.

Not parent, but whatever their plan is, energy density is limited by the technology, not the will. It's not like Panasonic could manufacture battery with twice the energy density if they wanted to, with some tech they're hiding in their back pocket.

As for the "gigafactory", it's Tesla trying to get into that closed market, like parent said there are only a few big players and it's not very welcoming to new ones, Tesla's plan is to be so big that they can have a big supply to sell in bulk AND to also be (together with solar city I guess) one of their first big scale customer.

I think you mean the electro-chemical cell. Batteries are just bundles of electrochemical cells (and 1 cell can be a bundle). I am far from an expert on this, but tighter packing of electrochemical cells could improve density.

> Batteries are just bundles of electrochemical cells (and 1 cell can be a bundle)

Correct. The problem with packing cells tighter is heat production: a ridiculous amount of heat is released and the individual cells can get up to 80 degrees C / 176 F. (80C is not a "doomsday scenario"; most cells are rated to sit there running at 80C just fine.) When they're all next to each other, tightly packed in with no place for heat to go, you're screwed.

The other reason is that volumetric density is usually not what people optimize for; gravimetric density is more useful most of the time, and in that case you don't really have any reason to use tighter packing.

Why is gravimetric density more important than volumetric density? That sounds counter intuitive to me.

Because nobody seems to want a 4 pound cellphone battery.

Places where gravimetric density is more important:

- Electric vehicles

- Aircraft

- R/C

- Anything that goes into space

- Large scale energy storage (good energy density typically = less lithium used)

- Generally: anything that moves

Places where both are equally important:

- Consumer electronics (you don't want an ultra slim 2lb phone, or a ultra thick featherweight phone)

Places where volumetric density is more important:

- Batteries in smartcards

- Some medical devices

- Certain flashlights, IoT devices

You'll notice that there are a lot more things where people care more about gravimetric density.

Also, the best Li-Ion cells for volumetric density that you can buy are rather inconvenient shapes; a favorite is the 18650, an 18mm diameter by 65mm long cylinder. Yes, the best 18650 technically has much better volumetric density than the iPhone batteries. But if you swapped the iPhone battery with an 18650 you'd have a hilarious looking semi-cylinder protruding from the back.

> energy density is limited by the technology, not the will

Spot on. Musk is a fascinating guy but he's not magic; energy density hasn't improved in 11 years with ridiculous amounts of cash thrown at it and I don't think Tesla will be the one to make a breakthrough.

> As for the "gigafactory", it's Tesla trying to get into that closed market

Tesla is actually partnered with Panasonic, which is one of the big battery makers and one of the three that actually make cutting-edge cells. Tesla's "plan" as far as I can tell is to make and sell batteries as cheap as they can, especially since the cells they are making right now aren't interesting in any other aspect than price.

From what I can tell, Tesla has been focused on making their batteries _cheaper_ at the moment so they get the best deal possible on their batteries. They aren't really trying to make "better" batteries because they need to scale up their production first to meet Model 3 demand.

You're welcome!

TL;DR They're changing the market dynamics but their cells are boring aside from being cheap and will probably stay that way for a while.

So first of all, yes they will change the market dynamics; anyone selling as many batteries as they (plan to) do will change the market.

Secondly, the gigafactory's cells aren't that interesting (right now) aside from being cheap. The cell lifetime is atrocious, between half to a quarter that of the market standard. The energy density is not great, around 220 Wh/kg for the best cells and 190 Wh/kg for the worse ones; this is slightly better than the cutting edge fifteen years ago. The power density they claim is unacceptable given what it does to the cell life. And since the cells are larger, they require new cooling systems.

Musk is a fascinating guy, for sure, but he's just as human as the rest of us. An absolutely gargantuan amount of cash has been piled into battery research in the last 11 years, with zero appreciable gains in energy density. If the Gigafactory ever produces a high energy density cell, it'll almost certainly be because their partner Panasonic (which is one of the few battery companies on the cutting edge) made a breakthrough and licensed it to them.

Finally, the battery market is a very strange thing: many makers of the 'good' cells won't sell to anyone except a few specific clients. These clients buy the cells (on paper) to make things like power tools or electric vehicles or battery packs. However, these clients vastly over-buy, and then sell off their excess stock to resellers. Anytime you buy a powerbank, or a loose 18650 cell for an e-cigarette, or a battery pack for an ebike, you're almost certainly going to end up with cells that have been resold at least twice.

As such, the easiest way that Tesla could absolutely revolutionize the industry is by simply saying "hey, if your needed capacity is more than $25K per year, give us a call" and selling directly to businesses. They could even put up a quote on their website à la SpaceX. Unfortunately, I've got a feeling that if Tesla did that their partners at Panasonic would make heads roll, but you never know.

That's interesting, to say the least. You can't just call Saft and order some cells then? They won't sell them to you?

I'm not sure about Saft in particular; with how ridiculously bad their tech is I'm convinced they are a French government jobs program, and nobody I know has ever voluntarily used their cells (except years ago for non-rechargeable.) For reference, their highest energy density Li-Ion cell is 169 Wh/kg. I'm not sure how they managed to be that bad; I just literally taped a rock (14g) to the cheapest battery in my apartment (45.8g, the $2.49 NCR18650PF) and the combined energy density of my new Rock-Lithium-Ion Battery is 170 Wh/kg, better than that of Saft's highest energy density rechargeable cell.

But yes; if you want newer battery technology direct from the manufacturer, you'll probably be flat-out ignored even if you're inquiring at a medium-large scale. The way to do it is to hunt around for semi-reputable resellers, who will usually give you what you want at an acceptable price.

My understanding is that the reselling behavior is the bane of the e-cigarette market. 18650's that you can get from most resellers may have good specs printed on them, but the battery won't necessarily have anything to do with the wrapper.

Edit: https://www.e-cigarette-forum.com/forum/blog-entry/list-of-b...

That's quite insightful. I know next to nothing about that industry and/or battery (modern) tech. I know people have always said Saft was a reputable battery company (around motorised speciality vehicles/grip for film industry). I have no idea why, considering what you've just said.

Its likely Saft optimizes for rough environments rather than energy density. Things like stronger casings, more insulation between anode/cathode, wider temperature ranges are more useful to some applications than energy density.

I've primarily seen them used for industrial/military applications.

You're probably right. Just looking at their site and applications they're touting makes it sound legit.

The stagnation of Li-ion batteries can be attributed to industry approaching the theoretical maximum energy density. So instead of optimizing the remaining 25%-30% capacity, focus shifted to Li-air or Li-S research. Li-S in particular can reach up to 2600 Wh/kg theoretical energy density, or 10x of current state of the art.

Maybe you can shine some light on this, why do the search trends [1] for lithium sulfur appear so cyclic? I'm not aware of any uses in industry, or any major conferences/press cycles around Li-S.

[1] https://www.google.com/trends/explore?date=all&q=lithium%20s...

> The best battery that money can buy, for two of pretty much the most important applications (can't elaborate further, sorry) was only made from ~2012 until ~2014. A lot of really expensive things that need that battery are now running off of old stock, which will of course eventually run out.

Don't lithium ion batteries degrade over time? If that's the case, then how is this 3-5 year old stock still the best available?

In 2010 I experimented with Graphene as the Anode of a Li Ion battery. The results at the time were pretty promising but we (the lab at my uni) were synthesizing all of the material ourselves - there was no graphene supply chain and it was very expensive.

We always talked about specific capacity (q, charge per unit weight) so all the numbers I'm quoting will be in that - you can multiply by voltage to convert.

We were obtaining specific capacities around 500 mAh/g. Vs q= ~370 mAh/g max for graphite anode. I recall based on theoretical modelling ~660 MAh/g could have been achievable with graphene.

This pales in comparison to tin or cobalt anodes, which back in 2010 had q = 990 or so. We were not really optimizing for energy density. Rather with carbon based anodes we would see much better cyclical performance due to the lower volume expansion during Li intercalation. So you would be able to get much more charge/discharge cycles without the loss of performance you'd typically see in a higher energy density metal based anode.

I work in a different industry now and have not followed battery development closely. I'd speculate a similar thing has happened in the "real world" the energy density may not have improved significantly since 2006 but the cycle life is probably better.

multiply with what voltage? sorry I'm being dumb - can you say the number?

>The best battery that money can buy, for two of pretty much the most important applications (can't elaborate further, sorry) was only made from ~2012 until ~2014.

VTC5?

Haha, glad someone tried to guess! But nope, there are several cells that beat the VTC5.

Two that come to mind immediately (that beat the VTC5) are the 25R and the VTC6, both of which run cooler and have a higher capacity than the VTC5.

You probably haven't heard of the specific battery I'm thinking of unless you've worked a decent amount with batteries, though; it's not terribly popular with hobbyists (part of that, of course, is that it can be hard to get.)

I understand that some technology is kept secret for competitive reasons, but it seems to me that what is shown here (composition of batteries) could also be found out by simply opening one, of course assuming a somewhat expert is doing it. Or is there something preventing this?

On the other hand, revealing causes of failure seems much more specific and valueable, so there's that.

The basic structure of Li-ion batteries is well known and not really different between manufacturers and physical configurations (e.g. cylindrical versus pouch cells).

What's significant is the details. The exact chemical compositions and dimensions of the different components result in significantly different performance characteristics.

A relatively small number of batteries had the issues. The one we knew for sure had, well, burned or exploded.

> They never release any information of any value whatsoever to the outside, and there are only a handful in the world

Technical explanations are fine, but catching potential problems like that is why QA, and locking processes, exists, on the shipping and receiving end.

Ideally, there be a sampling and analysis of the batteries, for a specific process, for a specific plant, before accepting them at the factory. Xray imaging, mechanical stress testing, accelerated life testing, etc. Sealing that many flawed component in your phones isn't the result of battery manufacturing opaqueness, it's a test escape for QA. Something went wrong. An incorrect sample size from an incorrect confidence, a process change at the battery end, without resampling, or ignoring or not readjusting the confidence after seeing failures.

Otherwise, you know, phones could blow up in people pockets and be a PR disaster for your company, because customers only see the name on the case.

I was lucky to pick up German as a teenager. This episode of Omega Tau (german only) has an amazing discussion about batteries.

http://omegataupodcast.net/222-batterien/

So can you translate the VIP bits to us that don't speak or understand German?

I'm definitely glad they posted it. It's interesting to see what factors contributed to the failures. However, I doubt Samsung published this out of the goodness of their hearts. I would be surprised if publishing this was not a requirement of some consent agreement with the US FTC or some other regulatory body.

Thanks for the post, interesting information. I know nothing about the battery market but wouldn't it make sense for big players like Samsung to buy into that part of the supply chain if it's in the state you describe?

You're welcome!

And they do. The big battery makers are Panasonic, LG Chem, Samsung SDI, and a few companies that make things internally. Apple has tried to get into batteries with a few strategic acquisitions but didn't really get anywhere (as far as we know.)

I'm also surprised by the level of transparency here. The Galaxy Note 7 warning signs are at every major airport I've visited recently, the damage is done. Hope to see them recover in the coming years.

If the battery industry is so secretive, how do you find out where jobs are?

Jobs find you, probably.

Can confirm. My MA research was on an EMD Li-ion cathode material and I get approached by recruiters for companies I haven't even heard of every so often.

I'm not actually working with batteries right now, but as I posted on an earlier HN thread [0] I'm a generalist, and typically when a company hires me one of the first things I do is help them with batteries. Pretty much any company that uses batteries these days can really benefit from my help (for example, reducing the battery cost from $20/battery to $6/battery, or bringing the capacity, given the same space and weight, from 2800mAh to 7000mAh.)

Regarding how people specifically hire me, I've been lucky enough to never really need to search for work; it just finds me. Either through HN, or from people I've worked for before.

By the way, my email's in my profile; anyone in this thread feel free to contact me. You should especially contact me if you're involved in any project that uses batteries, I always love to hear what people are making.

[0] https://news.ycombinator.com/item?id=13410359

Do you have any details or examples of your battery work? Those battery numbers almost sound like switching someone from older battery chemistry to a lithium based formula.

Although sometimes it is that simple, the two examples I quoted above (battery cost from $20/battery to $6/battery, and bringing the capacity, given the same space and weight, from 2800mAh to 7000mAh) were both examples where I swapped out existing lithium ion batteries to custom lithium ion battery packs (I build packs myself.)

Another example is when a company needed a 10kW output for just a few seconds. I was able to reduce their already-optimized battery pack (the largest weight part of the system) to one twentieth of its original weight.

Given a random company that hired me, I could probably accomplish one or two of the below:

- Increase battery capacity by 20-120% (average maybe 50% improvement or 1.5x original, for example 900mAh -> 1400mAh)

- Increase battery max output power by up to 70x (for example output power 15W -> output power 1000W)

- Decrease battery weight by 20-100% (average maybe 40% improvement or 1.4x original, example 500g -> 350g)

- Increase cycle lifetimes by 10x (example 300 cycles to 3000 cycles)

- Increase or decrease operating voltage of battery

- Make a lithium-ion battery extremely safe (not dangerous after extreme drops, heat, cold, short circuits or punctures)

- Change battery form factor into something much more convenient

That list is of course assuming that they're already using semi-decent lithium-ion batteries. If they're not using li-ion or using bad li-ions, the improvements are (usually) much more drastic than above.

If anyone reading this is working on any project requiring batteries, shoot me an email (it's in my profile.) I can reduce cost/weight/charge time and increase power/capacity/durability (whatever you choose) - and the worst case is that I tell you your batteries are already optimal.

Why are EVs considered small fish? Sure, they might have a minority of the car market, but they also have thousands of times the battery capacity of a laptop or a smartphone.

think of the sheer number of applications for batteries then just mass alone of all those supported devices should give a clue. while many longer range EVs will have 400kg or of batteries their numbers just aren't sufficient to matter; yet.

And yet, the battery production capacity is apparently nowhere near enough to supply companies like Tesla, making them become a battery manufacturer themselves. Which plays right into the parent comment - the battery business is huge, and Tesla could probably pivot from a car company to a battery / energy company easily enough. They kickstarted electric cars, the established companies are trying to catch up, but are doing so in part thanks to Tesla, their open patents, and soon enough, their hugely scaled battery manufacturing capacity.

> the battery production capacity is apparently nowhere near enough to supply companies like Tesla

All conjecture, but it could have just been a risk thing, not a capacity thing.

Tesla isn't a problem, it is Tesla's ambitions that are a problem. There is sufficient supply for the Model S/X but not the Model 3 and they were likely having difficulty finding a partner willing to spin up enough production capacity for the later.

If I were to hazard a guess everyone wasn't willing to take on the risk at the usual price so there was enough price difference (to cover the risk of problems) that it made sense for Tesla to spend the capital to build there own.

Heck you might even be able base Tesla's expanding ideas of what to do with batteries on this. "What do we do if we can't sell the cars at the expected price point so we move fewer of them?" "Just make other products that can shore up demand".

Again all baseless conjecture but depending on the quantities involved could make sense. If Tesla said "I guarantee I will by X but you need to be able to produce 5X in case I don't run into any issues" would you bite without compensation for that risk?

So was this a design problem with the batteries or an engineering one? Or both? Did the fact that they tried to pack the battery as tightly in as possible contribute to those flaws?

The phones that have nonfaulty batteries do not explode.

This was mainly a QC problem. Batteries were produced with an unusual rate of defects that resulted in internal short circuits. Cramming the battery in to the smallest space possible may have exacerbated those defects, but batteries for consumer applications should not be so fragile that moderate pressure on their shells causes them to explode.

Maybe the designed tolerances were too close. ie they design something that could not be reliably made.

People had these devices explode and planes had to land due to them, I'd say some candor is in order.

  "I was shocked when I saw the post."

Are you still using a Samsung Galaxy Note7?