F-150 Lightning Recall Due to SK Cells | Hacker News Reader

F-150 Lightning Recall Due to SK Cells

finance.yahoo.com

63 pointsimtig3rman49 comments

Ford recalled 18 vehicles. It knew exactly which ones had the problem. Pretty interesting.

I've toured a couple of car plants because I find them really interesting - Porsche, Mini - the level of refinement, sophistication and tracability of their ERP is always staggering.

Do you need special connections to tour one? I would love to go, but I assume it would be difficult as a member of the public.

I believe Volkswagen's Wolfsburg factory is open for tours

You can tour the Ford River Rouge truck factory, which is where they make the F-150 and F-150 Lightning (PS you can spend an entire weekend at the Henry Ford complex - the whole thing is outstanding):

https://www.thehenryford.org/visit/ford-rouge-factory-tour/h...

I went on a tour of the VW Dresden factory back when it produced the Phaeton and Bentley Continental. It looks like they currently produce the ID.3 there and are still doing tours:

https://www.glaesernemanufaktur.de/en/your-visit/our-tours.h...

At Mercedes-Benz we offer factory tours you can just sign up for. Agree with OP, well worth it!

Developing and iteratively refining a production line from pilot factory to network scale is not entirely unlike software development.

Here's a few clips of a modern one, including a longer doc:

https://youtu.be/UdDvDKC8FVM & https://youtu.be/TWLyRQbvr6o

Anyone can sign up for the Porsche factory in Stuttgart, Germany or the MINI plant in Oxford UK, there are probably many others too. Stuttgart is really dominated by Porsche and Mercedes-Benz, there's an giant M-B star on the town's big clock tower. It's a very business-y town but a nice visit for a car nut.

While I was an exchange student in Germany, my cohort got a tour of an Audi factory.

That was 23 years ago, though.

This is pretty common in industries like this. In the aviation industry, manufacturers record the tension with which each screw is torqued. That kind of fine grained logging is required for maintenance reasons and audited in case of e.g. accidents. Some modern car manufacturers use a similar level of detail in their manufacturing records. Common in both worlds is having very complex supply networks and just in time deliveries for these parts.

> This is pretty common in industries like this. In the aviation industry

I remember reading an account of the Space Shuttle Columbia accident investigation, where it was stated the investigation was hampered by the lack of serial numbers on some components, such as the astronauts' chairs. This made it hard to reconstruct the breakup sequence from the geographical distribution of the debris.

If I were designing a production line, every step of the process would record video footage of every part being made, tested, and fitted.

Then, if any quality issue is identified, the footage can be checked to see exactly which cells that went into which cars were faulty.

It's far quicker to check 30 seconds of video to verify that a worker remembered to tighten some critical bolts than to recall the car and have a service technician pull the whole thing apart to find that this car was fine after all.

Processes like that work for bespoke things like the mars rovers and satellites. But for a mass produced product you will quickly run into a scale and storage issue. Ford is planning on making 150000 F150 Lightnings so around 400+ per day. With ever step expected to be completed in a couple minutes you are going to have to track tens of thousands of videos clips per day and store and archive them. Then for recall work you will have store the videos for around 15 years. The cost to maintain all of that infrastructure and storage is most likely not worth it.

> The cost to maintain all of that infrastructure and storage is most likely not worth it.

Your analysis is likely correct; but at the same time, a system like this has never been more affordable, possible.

Also, imagine getting a video of your brand new car’s construction at the time of purchase.

So Ford is providing pictures for people who ordered Ford Broncos currently. And for a method of keeping customers interested in the vehicle they ordered I think it is a really good tactic.

https://www.motor1.com/news/542024/ford-bronco-assembly-line...

I wonder how well some kind of bespoke batch-processing video codec would work on production line footage where the endless repeats of the same mechanical motions continue for hours and across days/weeks? Encode only the difference between cycles.

Just store the diffs.

just store the one cycle and repeat it. nobody will know the diff

This doesn't address soft failures, though. What if the worker didn't fully tighten them to spec but did partially tighten them? What if the bolt was cross-threaded, how could you tell from a video? The amount of camera angles and quality of cameras needed to actually make these calls seems like it would be nigh-impossible.

The data-to-noise ratio of video is pretty bad (most of it going to be wasted on data for... the factory floor or whatever), compared to something like logging the torque of every bolt on the line (which, they already do in some situations).

Controls engineer who's designed several production lines for tier 1 automotive suppliers and a couple for OEMs reporting in... you'd be wrong, video footage is kind of useless, because you can't query it automatically.

Instead, our PLCs (programmable logic controllers) report to in-line vehicle sequencing databases. Every step of the process is logged to ILVS, and - unlike our work with furniture or consumer goods manufacturers - that is the only thing they care about. It doesn't matter if you can see that there's a physical assembly with properly torqued bolts that we've delivered to Ford's loading docks, we only get paid if the torques are correctly recorded for that serial number in ILVS. Each line tracks each part with serialized 2D barcodes. Any time a measurement is made or an operation is performed, the PLC will trigger an upload to record that data in ILVS, whether that's a barcode quality grade of the 2D scanner camera, or a force measured to drive a clip in, or a temperature measured while setting a heat stake, or the current drawn by a sensor, or...anything. For jellyroll battery winders I've done, that will include obvious stuff like Hipot test results but also minutiae like the row and column and stack height in the tray that transfers the jellyroll between the winding machine and the pack assembly machine.

This sort of database provides not only recall protections, but also ANOVA gauge reliability and repeatability to retain control over the process, and predictive maintenance to know when a machine or mold is wearing and causing a trend in the measured data (even if it's all within acceptable tolerance).

If I was recording bolt torques that were manually driven by a human (rather than a robot), I'd probably use a wireless electronic torque wrench with a UWB RFID tag on the socket (or some discrete tags on the jig) to verify that you'd reached the appropriate torque at the appropriate location. You'd be able to automatically query the database for the barcode of the assembly with any data that came with its forming/casting/other production data, the lot ID of the bolt, the calibration source of the wrench, the employee ID, the time and date that it was torqued, how much time it took the operator to torque each bolt, and on and on.

We do still have video cameras pointing at our machine recording to a ~1 week circular buffer, but that's more so that we can reconstruct the sequence of events that led to a machine fault.

> you can't query it automatically

Considering the more interesting applications would likely have barcodes, couldn't this be done with modern recognition and tagging?

Now you have to make sure that the camera has line of sight to the barcode!

video recordings would be in addition to what you've said, to catch things that nobody thought to record or check...

Things like "we've had a high number of complaints of wind noise in cars from the front right door seal, but only those made on the night shift".

Then you check the footage and see the worker is stretching the seal while pushing it on, breaking it slightly, and then it fails a few weeks later.

You never thought to have a database field for 'number of newtons of stretch force while putting seal on'... But the video can explain it...

Also interesting but pure speculation is that you probably can extrapolate sales and delivery numbers up to the point of the recall.

Had a similar issue with another other Ford vehicle with a battery, there were 729 vehicles worldwide. They knew exactly what was wrong: error in a temperature sensor of a specific welding unit which which could result in a damaged busbar over time due to the poor weld. The vehicles were worldwide, mine was originally from Germany but that causes issues: I never received an email/mail from anyone as it went to the original dealer which did not know me. I did get a warning though the local DMV equivalent so that worked correctly but by then it was already fixed as I read the ford service site from time-to-time.

Was replaced without cost as with this thing, no questions asked. Though the garage failed to bleed the cooling system (they said they did, but you can see if they did it in the Ford online systems, they didn't), resulting in the vehicle stopping once an air bubble reached the coolant pump and it overrotated (it will run for about 30 seconds before stopping).

In my case this also means I got a new battery after two years which is nice, but Ford probably tried to talk their way out of it thus it taking so long. If it's not a fire hazard it goes slowly ...

When I bought my truck (used, from a non-Ford dealer) I immediately signed up for a ford.com account and registered my truck with them. I've received all recalls and "customer satisfaction program" notifications from them since. I also had the truck routinely serviced at a few different dealerships until I found one that seemed no-frills and have used them ever since. Whenever I take the truck in for an oil change they will check for a pending TSB and ask if I want it done.

My experience with Nissan is they will do the former, but dealerships will not do the later. With Honda it was the opposite, the dealership would always ask to perform repairs I had no knowledge of.

The Kuga plugin hybrid? Very nice vehicle.

Barcodes containing batch or unique serial numbers (laser etching is common in auto part supply chains), along with good manufacturing records is how the scope of a recall such as this is contained.

In the food industry, a portion of cGMP (Current Good Manufacturing Practice) covers traceability. Not sure what the practice is called in the Auto Industry but I’m sure raw material tracing is a part of their practice.

Yes. It's not just Ford which had good records here, their battery vendor must have had great tracking to limit the blast radius so much.

Though potentially they just all got lucky and caught a problem early enough that most units with the problem batteries haven't left Ford control yet.

Of course, the supplier would have to have some SOPs to say how to make the assemblies and then a QA department to catch the Deviation and write it up.

Every electric truck has a centralized computer that calls home for ‘updates’ with a GPS. Pretty simple

If I were to guess, this issue will be because metal tabs inside the cells were cut to shape with a stamping machine. The stamping machine die was probably worn, leaving burrs on the edges of the tabs. Those burrs are sharp and perforate plastic membranes, causing a short.

Lesson:. All stamped parts should be deburred.

Don't the drawings say "Debur all edges"?

>Lesson:. All stamped parts should be deburred.

This take is like the manufacturing equivalent of when people say we need a law to prevent something that's already illegal.

The run of parts was mostly in spec. All they have to do is change their tools a little more aggressively or tighten up the parameters on their existing QC process.

Adding an extra step to a low margin high volume part where cost is dominated by number of operations in the manufacturing process is needless and wasteful when you can just do what you're already doing but at an ever so slightly higher level of quality.

Also for all we know they could already be doing this and the defect actually occured because of it, E.G. misaligned deburring head bending the metal into an edge

>Due to SK Cells

I predicted Ford would have problems with SK batteries two years ago the moment Bolts started going up in flames https://news.ycombinator.com/item?id=28547711 :

>you will be happy to know Ford is using SK batteries after switching away from LG. SK just recently settled for $2billions a case that revealed they stole LG battery technology. Bolt uses LG batteries.

>LG had a chain of costly recalls and replacements for all of their clients

>"LG Energy Solutions, the company that makes the battery for the Bolt and Kona EV, has not had a good year. First, they agreed to replace the 82,000 batteries sold to Hyundai for the Kona EV, Ioniq, and Elec City buses. Although the initial rumors were from a faulty battery separator, Hyundai later said that the problem was badly folded tabs. GM emphatically pointed out that they use a different separator, and a different factory. Thus neither of those problems should apply to the Bolt fires.

>Porsche recently initiated a recall on a loss of power in its Taycan LG batteries, and Ford also moved from LG in its Mustang Mach-E to SK in its Ford F-150 Lightning."

Really annoying that they don't put the number of vehicles recalled (18) in the headline.

Aren’t really large batteries still a fire hazard. I remember when they even became issues for Boeing Dreamliner planes:

https://en.m.wikipedia.org/wiki/2013_Boeing_787_Dreamliner_g...

Sure, just like large fuel tanks are a fire hazard. We have to take the correct precautions. In the case of batteries that is make things with the right margin of safety, and be careful how we charge. (probably more, I'm not a battery expert)

Battery chemistry is another one. There are chemistries[1] that are much less, if at all, prone to go up in flames on their own.

Of course there's usually always a tradeoff with other properties, like weight or cost.

[1]: For example https://en.wikipedia.org/wiki/Lithium_iron_phosphate_battery...

Batteries are a much greater hazard than gas fires. Not even close. To start, you can actually put gas fires out with a reasonable amount of water before its fuel source is burnt up.

Typically battery fires with no obvious cause are due to a fault in the manufacturing process. Big batteries are more vulnerable simply because there are more cells to potentially have a defect.

Current iteration of EV trucks is just dumb. Completely useless for any real truck things due to the abysmal range and exorbitant cost.

It's ok for you to say quietly in your head "I'm not the intended audience for this product" and move on. You don't need to embarrass yourself like this.

Much more embarrassing to be the intended audience for such a ruse.

Ruse. Most people don't buy truck because they need a truck, they just like trucks. People who only drive their trucks in the city are not being tricked, they just buy what they want.

I would argue that it's not what they like but rather what advertisers have told them they need to prove they're tough or masculine or important. I keep getting adverts on Youtube from Jeep/Chrysler and they're some of the most fragile ego appealing cringe ads I've ever seen.

Oh give me a break. You're so brainwashed that you don't see the marketing bs EV manufacturers pull is the same thing and you're out here touting their products FOR THEM.

You need PROVE you're saving the planet by mining precious metals out of Africa with slaves. But, hey you've got a "green" vehicle so who cares.

Who's the actual audience? Elites that want to pretend they're saving the planet with an oversized vehicle they get in after they just got done flying their massive personal jet around?

Why are you embarrassed by reality?

Reply on news.ycombinator.com