Burned-Out Flash Trips Up Older Teslas
eetimes.com
eetimes.com
Maybe "Made by Tesla" is the new "Made in China".
Not sure why people consider breaking best practices without understanding them in the first place is a good idea...
Meanwhile companies like Garmin and Magellan manage to build GPS devices that work just fine even though they spend their whole lives in the passenger compartment. They did this since the beginning, I never heard that you should stay away from this or that manufacturer of GPS units because the touchscreen fails.
The first iPhone didn't have 3G while other phones had it. In order to do something new, you need to cut some corners. It's not like Tesla shipped non-automotive breaks.
Source: https://www.thedrive.com/tech/27989/teslas-screen-saga-shows...
I’m guessing it didn’t effect their brand much because other companies like Tom Tom were aggressively targeting the entry-level market segment, and then the iPhone came out the same year.
The number people who had these higher-end units in hot southern states was likely just too small to significantly tarnish Garmin’s brand goodwill.
Soon enough most people switched to using their phone once the software was good enough (circa 2009 IIRC).
Edit: Thanks for the suggestions!
I've heard very good things about the Nissan LEAF, Hyundai Ioniq, Hyundai Kona, and Kia Niro. It might be a coincidence but all of these cars are packaged more like traditional cars, i.e. popping the hood allows you to access many of the components.
On the true luxury end you also have the Jaguar i-Pace. If you have the 80k EUR to spend that is. There are a lot of traditional car makers coming up with electric cars now. With the added benefit that in some cases they can be built at the same factory like their ICE brethren. That would allow them to drive costs down quickly if the demand is there and economies of scale kick in.
The repair on your Tesla will cost more than a motor or transmission replacement on a conventional vehicle.
Tesla owners should file a class action lawsuit to force Tesla to extend their warranty to cover this issue. Tesla will try to fight it and make you pay out of pocket.
Sure they do. Take a look at the FICM failures with the Ford 6.0 diesel. In the gasoline world you'll come across failed injector or ignition drivers if you look hard enough. Failures due to environmental causes (like the wiring harness wicking hydraulic fluid into the control unit itself) are a thing as well.
It's not engine related, but BMW's late 90s-early 2000s five series uses a computer to control its lights. The failure mode is that your lights stay on as long as the battery is hooked up (fail safe). Unfortunately the IC that BMW used was discontinued by Infineon. I think a surface mount version is still available, but ultimately yeah these things wear out and you're largely at the mercy of the OEM.
Injector drivers and ignition drivers are part of the control unit.
The ECU is a computer, a packaged bundle of chips which plugs in to wires and does nothing but communicate to other components via those wires. That's what an electronic control unit means.
The driven devices, amplifiers etc. are under the control of the control unit. They are not part of the control unit. They're located in areas subject to much more heat and mechanical stress, and in most cases (ignition coils excepted) are mechanical in function.
Because...
The driven devices, amplifiers etc. are under the control of the control unit. They are not part of the control unit. They're located in areas subject to much more heat and mechanical stress, and in most cases (ignition coils excepted) are mechanical in function.
I'm not talking about the driven devices.
Here's a diesel example in video form:
https://www.youtube.com/watch?v=4k-G1dMgFUo
Here's one in gasoline form:
(But at least an EV doesn't have issues like VWs "maintenance free timing chain" skipping a link or two after 70kkm/50kmi)
Why is it Tesla's responsibility to fix the car out of warranty?
All products and services in Australia need to be fit for purpose. There's no strict definition for this but it basically means if you are selling a "premium" product, your warranty lasts for however long is reasonable in the eyes of the courts and not what the manufacturer says. It is very pro-consumer.
It would be reasonable to assume that a $40k-$100k car would last 10+ years. Obviously this doesn't cover consumables but bad memory cards would definitely be covered.
If you're saying this part is intentionally designed to fail shortly after the warranty expires, then I understand your point. Is that what you mean?
EDIT: The way I see it is if I buy a car with a 3 year warranty and something fails after 4 years, it's not the manufacturer's responsibility to fix it because the warranty has expired. Why do you think it is?
Ford for example did, with failing intake manifolds and the issue where the engine would eject spark plugs on Modular V8 engines.
Tesla in particular is screwing over the people it needs the most, by making boneheaded component choices.
I completely agree Tesla cheaped out on this part, I just wasn't convinced they are responsible for fixing it. I'm still not sure this justifies a class action lawsuit (I don't really feel strongly one way or the other), but I at least understand why Tesla might be held accountable for their actions.
You're talking about whether or not people should be upset. I'm talking about whether or not Tesla is legally responsible. These are orthogonal topics.
Yes, people should be upset with Tesla for a shitty design. I don't think anyone is arguing that. I don't think that automatically means Tesla should get sued for it.
Then again, I've never tried to sue an automaker. So maybe I'm just ignorant of how this is supposed to work.
But if Tesla cheaps out on critical items, thus dooming a car to that scenario, and a consumer can’t know that, it introduces a different legal dynamic.
(Disclaimer: IANAL.)
Because us tech people have been successful at hand-waving away product defects as mere planned obsolesce; we think about it as an option less and less. Point: Intel and all of the speculative execution crap, and specifically TSX (or the lack there of... but marketed as "Secure!" regardless).
Yeah it pisses me off. If you make an under-delivering pile of shit and make no attempt to right the wrong, fuck you and your company.
So for a variety of reasons, I'd argue they have a moral responsibility to fix it, INAL, so it's unknown to me if they have a legal or regulatory responsibility too.
Depends on the regulatory environment. In the main though people can expect cars to last for a good long while with maintenance, so a part that was guaranteed to fail well before then no matter how good care the consumer took of it would be a regulatory no no.
https://en.m.wikipedia.org/wiki/Implied_warranty
One might argue that this flash memory was not the correct choice for the purpose it was being used for, and as an expert, it’s was Tesla’s duty to select a part up to spec for the purpose it was being used for.
The normal expected lifetime of a new car today is apparently 8 years or 150,000 miles. So I would expect the car to last 8 years on average. If a car is going to last significantly less than that on average (e.g. assuming this means the car needs a fix every 4 years), this should be disclosed in the marketing.
"years" of warranty is a silly concept anyway. If I buy a new car then store it in a perfect environment unused for 20 years (assuming we maintain anything that isn't stable like oil), it should still work as if new. If it doesn't, it clearly wasn't sold fit for purpose.
A corrolary of this is that when it is not spread over a high number of units, in a niche product, it will be very expensive so we will see even less niche products as software takes over more functions.
The mechanical parts of a well maintained ICE car rarely die for no reason. Electronics on the other hand...
And for those of you whom know their history, this has all the hallmark signs of anti-trust.
Enjoy replacing literally every part of your car if you expect it to last close to a decade.
When I was considering a Model3 they harped on how reliable it would be. I still passed.
Only the gearbox (automatic) will cost you not less than 8000€.
ECU fails? Add 5000€
I had to dump an 09 VW GTI (I'm in the US) about four years ago with only 60K miles on it. The ABS computer failed; a $4k repair.
As a side note I traded it for a new 2016 Jetta Sport; which was the worst car I ever had, and the last VW I'll ever buy.
I traded the Jetta for a 2019 Hyundai Kona which I like but you can't drive it with the forward collision system engaged. I had always shut that off from day 1.... except a month ago I didn't and that sucker SLAMMED on the brakes when I was lane merging during a busy time--I was Not being aggressive just assertive. Only by luck no one slammed into the back of me. I can't imagine Anyone trusting their lives to a self-driving car.. it makes no logical sense to do that!
Replacing the entire ECU feels like an admission like Apple and increasingly Lenovo (thinkpad), they have driven premium to the wrong place. "please give us soldered on memory and soldered on SSD" is not actually what we said, on our feedback forms to Apple, or IBM (X1 Carbon).
Nobody asked Tesla to shave $1 on the ECU assembly and the incremental cost of recall and fix, along with goodwill now outweighs the cost saving in any sense I can understand it. If the authorities demand he fix for free, it becomes another shit show of the TSLA share price. This wasn't a good choice against all the other choices.
I don't know why you are lumping Lenovo ThinkPad in there. Their units are still highly repairable.
Even their thinnest model (X1 Carbon) still has the replaceable SSD. It accepts any standard NVME M.2 drive - https://www.windowscentral.com/how-upgrade-ssd-lenovo-thinkp...
https://www.statista.com/statistics/261877/average-age-of-pa...
It's true they don't look great underneath...
Stop buying bad cars.
Conversely, there are many engines on there noted for their reliability, Ford Modular V8, for example, GM LS1, Ford Model T engine, VW Air-cooled Engine.
Don't forget, just because a typical owner may sell a car after 5-10 years, doesn't mean it goes away into the aether :).
[FWIW, I have a 2004 Subaru WRX; I make the dealership rounds every year looking for replacements, and cannot find anything that handles better in that price range - in fact, by all accounts modern WRXs are a softer drive]
A long tail on only one side (no car has a negative age) means that more than half would be older than the average. You were thinking of the median, not the average.
It's really mostly the macbooks that are terrible in this regard.
What's not replaceable or too expensive about a $400 or so nVidia Tegra board in a $100k car?
> Nobody asked Tesla to shave $1 on the ECU assembly
Are you suggesting it would have cost Tesla $1 per car to go from an industry standard board used in many cars to a custom solution with all parts tested in-house?
But to your main point, no. The actual cost per board of getting a design done for replaceable parts would have been higher. Once they hit volume the amortization would kick in. But for the earliest, yes,expensive. Strangely in the world of eg routers, the first gen often has more hackabilty, more removable parts and it's the later ones which shrink down, gets the bill of materials down.
Seriously, they should comp all the replacements for the life of the car, just for p.r. value.
This particular issue may be unique to Tesla, but obviously other car manufacturers have a whole host of issues which their customers deal with and can result in service bills even staggeringly higher than this.
My last Infiniti G35 has an issue with the Bose head unit, where a single resistor would come loose and the radio would lose power when you hit a bump. The problem was the whole center console stack of buttons all were powered through the radio and so you would also lose the ability to change the temperature or turn on the defogger.
Cost was $2400 to replace the unit, and they would typically fail again ever few years. The problem was extremely widespread and despite a campaign to get Infiniti to extend the warranty, they never did.
I ended up putting in an after-market unit but I had to replace the module with all the climate control buttons as well with one from the Japanese market version of the same car, because that one could work independently from the radio.
I think Tesla were the first, but everyone does it nowadays.
https://developer.nvidia.com/embedded/community/lifecycle
Tesla needs to have stock relative to replacement rate purchased before nVidia's last ship date. So if they used eMMC built in to a part like this then they should be raising problems for their parts logistics.
It's very likely the hardware designers didn't expect the software living on that flash to iterate that many times.
In too many ways, Tesla was and still is a hardware startup.
I'd have no issue with this stuff bricking if Tesla replaced it free (even more galling that they won't on a 50k+ vehicle), you'll have a hard time convincing anyone that a ECU should last less than a decade under normal operation, Nor should you need to replace the ECU before you replace the battery pack on an electric car.
If storage space was alone an issue, then they shouldn't have extended the features of those firmware loads so much - as far as I know the owner has no choice on if you update or not.
They probably plug in their dongle for diagnostics and hit « yes » if it prompts them for an update.
are there such connectors suitable for automotive environments ? you don't want your flash to fall off while driving
Finding (or customizing) the right one is the hard part...
That also applies to other expensive consumer durable goods.
I wonder if Tesla maybe skirted this (like "oh this computer isn't in the car it's an attachable add-on).
Source: I interned at an electronics component company back in uni as a chip tester. Mostly same chips to everyone but ones rated for automotive or space tended to get a lot of extra testing (since failures on electronics are bimodal, running tests in extreme environments just for a bit catches a lot of the bad apples).
And it's already caused more issues: https://www.thedrive.com/tech/27989/teslas-screen-saga-shows...
This + the horror stories I read about getting replacement parts when you damaged the car is what keeps me from buying a model 3.
Look at, for example, what Sandy Munro says about the electronics quality specifically on the Model 3, and you will appreciate they have come a long way since then.
There is no automotive company I would trust more when it comes to quality of the electronics. This is a group which taped out their own custom silicon for their APv3 hardware, designed by famed chip designer Jim Keller, and will field upgrade it into existing cars. What other manufacturer sells hardware upgrades to their late model year vehicles?
He had negative things to say about the quality of the rest of their electrical components, noting that based on the cost of the components Tesla should have an 18% markup...because Tesla is using consumer grade parts, not automotive grade parts.
What other manufacturer sells hardware upgrades to their late model year vehicles?
Almost every car sold to consumers in the US today allows you to replace most parts of the car, which includes upgrading nearly every component, if you so desire. Sometimes the automakers sell upgrade parts; usually they leave it to third parties (i.e., the "aftermarket"). It's actually not a good thing that Tesla is the only source of upgrade parts for their vehicles.
That is simply false. Here he is talking about the Model 3:
https://www.youtube.com/watch?v=aVnRQRdePp4&t=10m11s
Their margins are better than their competitors because Tesla is vertically integrated, allowing them to do things like reduce the number of harnesses by 40%, and design solutions like the "SuperBottle" which Munro once said is his favorite part of the Model 3:
https://www.youtube.com/watch?v=ACdvsfZH-4w
And there's a world of difference between after-market upgrades (of which there are plenty for Tesla) and the manufacturer taking their latest generation technology and manufacturing and designing it so that they can slot it into late-model cars, all while constantly shipping firmware upgrades to their entire fleet which make the cars better in every possible way from speed, handling, range, efficiency, user experience, safety, entertainment, etc.
Whoa Whoa Whoa ;-)
As somebody who has 9 ThinkPads around his house, including the controversial T25, don't forget Lenovo completely bought and owns all Thinkpad design production branding etc for the last decade.
Would IBM have maintained their priorities if they owned the product today? Not sure. But Applezation of the ThinkPads has happened strictly under Lenovo :-<
The P series is still pretty good; X1 was never about modularity but strict portability; it's the devolution of T series that bothers me personally the most - the T420s was barely thicker than modern T490, but had swappable battery and hard drive. Not "user replaceable", literally push a button and swap it out.
[Disclaimer: IBM employee, but in a completely unrelated part and a ThinkPad fan before my IBM employment; I work in software consulting on large ERP projects and haven't seen an IBM hardware person in my 15 years of service:]
The main firmware should have been on a separate physical device if data was write heavy regardless of “enough space” being left. It would have cost a few dollars to make this issue go away.
The more I hear about the engineering the longer I will avoid buying an electric vehicle. Iterating big problems away on a tangible chunk of engineering rather than a piece of software is stupid.
[0] https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...
You can look up the nVidia Tegra board on Wikipedia and see for yourself that the "shitty short sighted engineering" board is used in many gasoline cars.
Tesla's use case is unique because they stuff enormous firmware images into the on-board NVM and write to it constantly with very heavy logging. Of course your Flash/EMMC will wear out quickly if you constantly write to it while your device is on - that is the short-sighted part.
The board isn't necessarily a problem, but how Tesla chose to use it sounds like a very predictable problem.
> As a result I've taken down my howtos. I'm sorry about this but there is some kind of political problem which I don't understand.
https://www.diyelectriccar.com/forums/showthread.php/howto-t...
Pity the site is not available on waybackmachine.
But if it was an SOIC... just lift the power pin, power the chip externally and find some other points to tap into the clock and data lines and dump away. Then solder the legs of the new one on one side and connect VCC on the other side.
Its still possible to do this. You do wear level between the blocks containing the firmware and the logs.
In any case, if 90% of the space is filled with static firmware files then wear-leveling becomes difficult. Probably best to turn it off and let the physical blocks with the logs fail while keeping the firmware blocks intact, or just stop writing logs after a certain point.
https://en.wikipedia.org/wiki/Wear_leveling#Static_wear_leve...
No. You do normal, dynamic wear leveling. When you notice that the block you want to write to is much more worn than the rest-- say 100 erases extra-- you move a firmware block to it instead and claim that for use. Then that block sits near-quiescent for a long time until it's time to put it back into the rotation of blocks that get actively written. Static wear leveling can just be a tiny, tiny share of write amplification-- a factor of 0.01 or even less-- while keeping all the blocks in play.
You only "lose" from static wear leveling if you end up moving something right before it becomes active again. Whatever your strategy, there's always a degenerate write pattern which will cause useless write amplification, but even relatively simple schemes are robust against basically everything except deliberately malicious write patterns, and you can bound this worst case.
Incidentally, one way flash fails is by dielectric breakdown increasing leakage causing worn-but-idle-blocks to slowly accumulate error. Moving / occasionally scrubbing static, important data actually helps.
There are Ford Model T cars that still drive. They were made 92 to 111 years ago.
With undocumented and cryptographically signed interfaces between components, it will be impossible to keep a Tesla running that long. The best you could hope for is to graft the body panels of a Tesla onto the innards of a newer car, merely dressing up the newer car to cosmetically look like a Tesla. Such a hack would not be have vehicle title as a Tesla, would not drive like a Tesla, and would not have an interface like a Tesla.
There is already some development of open source ECUs:
https://www.google.com/search?q=open+soure+ECU
For ICE based cars, you can expect something similar to happen for electric cars.
It would be more expensive for Apple because people would buy cheap models and later upgrade, instead of buying expensive models or buy new machines early.
No, the best you can do is root the car and fix the software as you like.
https://teslamotorsclub.com/tmc/threads/let-the-hacking-begi...
Alternatively there may be holes that can be used--such as a 'boundary scan' of one of the chips yielding the key. I don't have a spare Tesla sitting around to test this theory, however (grin)
2.There's a difference (8-fold) between Gb (gigabit) and GB (gigabyte). Memory chip capacities (and data transfer rates) are often expressed in bits (per second), but memory device (RAM stick, storage,...) capacities and file sizes are always in bytes.
Easiest hardware fix would be to replace the part with a larger one which would only be a BOM change. But that doesn't help the units already in the field. I'm sure they've figured something out by now.
>Tesla’s firmware size over the years went from a paltry 300Mb to a full 1Gb over an estimated four year period, which consumes much of the storage space in that Flash. Add on the constant vehicle log updates, and the failure rate of those chips increased exponentially.
Does it really increase exponentially or does 'exponentially' just mean 'a lot' now? Genuinely curious. Also I think I'm allowed to be pedantic when the source is an electrical engineer.
Most game consoles and TVs are unlikely to do enough logging to wear out their storage. But if they did, especially if they cost as much as a car, it probably would make the news.
Cars tend to last a decade or two. They shouldn't stop working because of something as silly as excessive logging.
Phones hit bad batteries before anything else and issues related to them get reported fairly often.
> game consoles,
I think Nintendo had a console that hit max firmware updates, instead of breaking it just meant they couldn't interfere with people rooting it anymore. There was much rejoicing.
> What makes Tesla special
An order of magnitude higher price compared to anything else you have listed. Maybe they should try to sell cars for the price of a new smart phone with the goal of selling a new one every two years so nobody would notice these long term issues.
It simply doesn't happen very much (especially not in game consoles and TVs, which don't do much writing).
> Are there really no other car manufacturers using eMMC btw?
I assume there are, but they're not writing loads of logs, or, if they are, specced their flash storage appropriately.
I mean, exponential over what? Over time? The failure rate (more accurately the cumulative failure count) will probably grow logistically, but would have done so without the firmware size increase. Over free space reduction? That should be linear. More accurately, the amount of data you can turn over before wearing out the flash should have a linear relationship to the amount of unallocated storage available.
> The MCUv1 units supplied for the Tesla Model S and X, up until 2018, feature an Nvidia Tegra Arm-based SoC with 8Gb of eMMC integrated into the same board, which is hosted on a mainboard.
Whether Tesla are willing to foot the bill outside of warranty, however, remains to be seen.
There could be a software update to slow or disable the log-like writing or at least warn when the reserved blocks were running low.
"Rather expensive"?
Just to give you an idea: https://www.nvidia.com/en-us/autonomous-machines/embedded-sy...
I know it's hard to include replaceable storage due to the vibrations etc. but hopefully the new Teslas will at least start using something like XFMEXPRESS for semi-permanent storage.
Or, alternatively, hire people who know what they're doing and take their advice? Flash wear isn't exactly an obscure issue, and it would certainly be on the radar of, say, Bosch (who make a lot of this stuff for the industry).