Hacker News Reader

"Replacing a tiny part" is trivializing the work necessary to do this, in my opinion.

eMMC flash is typically a ball-grid array part and it's soldered down hard to the motherboard of the processor. Almost every mobile device has the same setup and it's tricky and expensive to rework this chip.

You need to carefully desolder it off the board, reball the part with the correct masking stencil, then reapply it without torching the rest of the board, missing a solder connection, or zapping any other component with static electricity.

You can see a similar process done on an iPhone here: https://www.youtube.com/watch?v=y4M9uAZlbK4

You can understand why Tesla would rather replace/refurbish the board than try to field-repair it.

This is the right answer

Replace the board, then you can refurbish it in a specialized lab

Same reason why genius bars won't replace components, just boards

Luckily Tesla isn't very space or money limited with this pretty ciritcal component. An SD card would make life easier for everyone, as the logs could be easily read out and the booting OS changed - yay for actually owning the device you paid for. But wait: That's not something Tesla wants.

eMMC is literally the equivalent of an SD card soldered down to the PCB. It’s more mechanically stable and also way faster.

In my experience, removable SD wouldn’t hold up to an application like this.

The actual logs Tesla seem interested in (including the stills of things they are interested in) get written to a removable SD Card before getting uploaded to the Tesla mother-ship.

You are not wrong that a soldered eMMC (compared to a eMMC held by a ZIF or a removable SD Card) would be more mechanically stable and potentially faster (A fast uhs-ii sd card compared to a cheap slower eMMC for example), just saying that Tesla already use removable SD cards in production cars today (which doesn't brick your media unit if it dies or dislodges).

Storing the OS on a removable SD Card is (imo) a bad idea, but if they are really interested in these logs they already have a easier to replace removable storage "drive" they could write the logs to instead of burning up the eMMC.

An SD card might work fine if it is mounted properly. The slot found on a Raspberry Pi would not at all be sufficient.

If it were held in and held against some robust contact pins it likely would be fine, even under automotive conditions. There are SIM card slots that are used on heavy vehicle computers and controllers (the ones that run the engine and so on) that are robust enough to hold a SIM card in place for years of excessive daily vibration.

Whenever I drop my phone I see the "SimProcessor" service spinning up...

I suspect that designing reliable spring contacts isn't as easy as it seems...

Yeah, phones aren't designed to have robust SIM slots, they're designed to be thin, so they use very low profile SIM cradles that are not robust to impact.

The trade-offs for a high vibration environment are very different.

Or just add an SD card reader on the board and output the logs there if they need it so much, and keep the rest on the EMMC.

They have been replacing the whole media unit including the screen (and charging customers $2k-5k for the priv) where they could replace the processor module board (which is "just" a add-in board inside the media unit) keeping the rest of the media unit and lowering costs.

Sure refurbish the processor module and replace the eMMC at a proper rework station but replacing the whole media unit esp when the failed part is on a removable board just seems like a huge waste to me esp when the customer is footing the bill.

It feels like they should just turn off syslog in production builds, release it in a firmware update and offer a recall for everyone affected by broken board.

So it's not fixable with software update?

The under lying issue can not be fixed with software. the eMMC is simply running out of write cycles. BUT they can mitigate the issue by not writing the syslog to the eMMC allowing the remaining write cycles to last much longer (This change could be done with a software update).

On the newer versions of the Media Unit instead of disabling writing the syslog to the eMMC they added more storage to the processor boards which isn't going to solve the problem just push it from appearing to further down the road.

The problem is that there is significant write amplification as the used space on the partition approaches the total space of the partition.

To say that the choice to use eMMC is the fault is to say that ANY use of flash storage is a faulty decision.

If there is sufficient space on the drive to mitigate write amplification, then the media will last a very long time indeed. Longer than the drivetrain, certainly.

The problem is that Tesla doesn't delete logs it has collected from the cars. All Tesla vehicles (as I understand it) upload their logs more or less continually. Currently, the firmware on the cars doesn't delete those logs once they've been uploaded, presumably so that they're available when it is time to service the vehicle.

The disk filling up due to bad log file hygiene, coupled with the fact that the storage is flash-based, together cause the issue.

Most eMMC cards use really primitive wear levelling algorithms.

A simple firmware update to the eMMC chip could make it use a wear levelling algorithm that really does spread writes evenly over the whole chip. Current implementations simply use a sector till it goes bad and then using a spare till that goes bad, etc. After 1% or so go bad, the whole device rejects writes entirely.

With a decent implementation (like ssd's mostly do), you will never be able to kill a card in a lifetime.

Wow, this is exactly the tutorials that you get on "How to get your Rasberry Pi to run stable for extended periods of time". Exact same failure mode, exact same fix proposed.

> Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards.

Do you know that they toss the old media units? Are you sure they don't swap the entire media unit to simplify service procedures, and have have the old units inspected and refurbished (i.e. eMMC module replaced) for future repairs?

That sounds way more efficient than training all of the shop workers to diagnose and swap out individual parts of the media unit.

An SD card has a much different set of problems, including physical contact issues. eMMC is an acceptable tool for the job, it's just complete negligence to not use a batch logging technique if you even need full syslog data.

They use an SD card for the car logging, according to the Rich Rebuilds video, so it's doable. It just needs to not brick the car when it fails.

That they use it in other applications doesn't make it automatically the best idea. SD cards are fairly unreliable compared to the other options, and as Rich's friend points out, smartphone manufacturers use eMMC and don't have issues because they understand wear cycles.

It is a bit baffling that a company which enables “autopilot” OTA does not do this as well...

If this anonymous coward's characterization of Tesla is true and accurate, it's not surprising https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...

Why did your choice of words include “anonymous coward”?

On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in.

The linked reddit post is from a user who claims to have worked for Tesla in the past, and is (presumably) not using their "real" reddit account to post it.

I don't read it as an insult.

> On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in.

Gotcha. That makes sense. Slashdot was never my thing. Thanks for clarifying; I had read it as bafflingly unjustified aggression.

The wording makes it sound like you're calling the parent comment an anonymous coward. I also suspect a lot of people didn't catch the Slashdot reference.

Maybe he was hit / smashed over the head with an OPEN SOURCE CD?

Wow. Thanks for that link. Some amazing inside stuff. My favorite anecdote:

most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do shit like add farting unicorns

Problems with an SD card is that it will wear out even faster than an eMMC due to lack of FTL, and also friction fit instead of soldered connections means the part is susceptible to wiggling out of the socket and also corrosion between the contactors.

Not sure of the benefits of an SSD over an eMMC. Guessing the former will have a more sophisticated FTL and other things.

Multiple companies make high endurance SD cards for things like dash cams and automobiles. Take, for example, WD https://www.westerndigital.com/products/embedded-removable-f...

Didn't know that, thanks.

But there's still the problem of a friction fit connection in an environment with a lot of vibration and contaminants. Seems like these would be good for use cases where the part needs to be removable and also have high write cycles, like a dash cam or data logger that has no other interface to a PC.

Also the eMMC interface exposes quite a bit of the underlying hardware (see mmc-utils on Linux). With SD cards I think you're generally stuck with some kind of SMART interface.

> Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards.

Am I the only one thinking about the recycling and cost implications of doing this?

Well, if you have some thoughts on it, I am curious what the other approach would look like. What are your thoughts?

I'm thinking they would have all the labor of taking it out, and all the labor of putting it in, but then much additional labor and repair time incurred by doing the component level repair instead of just grabbing another radio off the shelf. Recycling would be possible on the unit later and it could be installed as a refurb unit in another vehicle.