The code went through early release tests successfully; the problem came with how it was more broadly released.
They should have had further staging of the rollout (randomizing when it is offered to users).
They should have had further staging of the rollout (randomizing when it is offered to users).
Lots of ways to screw this up, especially in automotive where you're likely to be dealing with TI and their (in)secure boot.
I've solved this problem god only knows how many times now and I've rarely found an automotive board that doesn't introduce fun, new edge cases. OTA can't exceed x kilobytes of memory, the processor isn't fast enough to verify signatures and write the image in < x seconds, can't write the image to flash unless the signature is verified, but the image doesn't fit in RAM, the server delivering the update is 3+ networks away from the device receiving the update, etc.
Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardware, the programmers (and their managers) are the problem.
The rovers on Mars, the Voyager, these problems have been solved for a long time. The compute in a Tesla these days can probably run Crysis.
You can do this OTA to a Raspberry Pi running Nerves via remote SSH and it works really well. The Nerves runtime utilizes A/B partitions for OTA updates.
For a vehicle that costs $100k+ it shouldn't be hard to double or triple the budget for onboard compute considering it is vital to the operation of the entire vehicle.
A shockingly large part of my job is telling both that one team won't be getting what they want and to work it out among themselves. Rinse and repeat between dozens of boards because the relevant teams don't talk to each other and none of them read the "design requirements to ensure we don't have to tell you no" doc either. One time they didn't even tell us there was a board until the end of December, when delivery was scheduled for Feb.
The test vehicles accept test/prod signed versions
Regular vehicles only accept prod signed versions
They are otherwise identical.
The test vehicles were sent test signed versions
The prod vehicles were sent the exact same update, signed with test.
This would not be uncommon since the test vehicles probably occasionally run test releases for debugging.
Further, the update is probably multiple signed pieces, and the only part accidentally signed with test was likely infotainment software.
Or something like this.
It's hard to believe they wouldn't test sending badly signed updates, so i have to imagine it's a particularly weird badly signed update.
In other words, i would not assume they are idiots.
You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.
OTA is generally developed by tier 1, so this is probably a bug in the tier 1's code. (Samsung, Panasonic, Sony, etc are common tier 1s in this space.)
IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key.
While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time, only at runtime.
So, OTA update comes in, containing a bundle of software for different subsystems. It's sent to different subsystems. Then, those subsystems check integrity at startup, but one subsystem's bootloader isn't happy because the firmware looks to be invalid.
You can only prevent this if the OTA knows how to do equivalent verification for every subsystem in the car that checks integrity. (And, of course, even if you do this, there's other ways you can go wrong that aren't specific to integrity checks).
It seems silly to off the cuff assert that it is a bad process or only done by those who have no idea what they are doing.
Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
> Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
As for this, the entertainment system can control basically every feature of the car and is often the primary or only way to accomplish certain things. Even in much much dumber cars the infotainment is still part of the CAN bus and is able to interact with the rest of the vehicle.
Sorry, but things no longer work that way, and never will again. This is a good thing, as long as processes are improved to avoid situations like this one.
How do you figure. I can’t think of a single thing in my vehicle that could be improved by software. When I buy a car I’d prefer it was done.
My biggest issue with modern cars, and it seems this is spreading to other embedded systems than cars, is treating those as software: connect them to the web and run OTA updates everytime you need to fix a bug. That requires some form of inzernet cinnection, which requires regular security updates, which require OTA update capability. All that because, bluntly, software devs cannot be bothered with just finishing software running on non-connected hardware that just works, not can they be bothered following hardware development in case hardware is the main component, as in cars. And no, no car manufacturer is a software company, nor phone maker like Apple.
The proper way of fixing software bigs in automotive used to be, again thank you Tesla for breaking something that just worked, to recall the affected cars to a workshop to conduct the software update. Honestly, the only thing on a car that should be updated by the owner is maps for the GPS unit.
The less software in my car the better.
Sure, but if you are rolling out to 1% of users per hour, you detect the problem in a couple of hours and much fewer than 2% of users will have applied the update. This is a relatively small support problem.
While if you roll out to everyone at once, you'll detect the problem sooner (within an hour) but have 10x as many affected.