Hacker News Reader

Comment by rollcat | Hacker News Reader

A/B boot is old, good, and proven, but what do you do when you have dozens of separate microcontrollers, each governing a semi-independent subsystem, all running their own software that needs its own updates? It would make sense from safety/reliability PoV when a crash/bug in a slightly less critical system (like adjusting mirrors) does not affect a slightly more critical system (like indicators). What do you do when updating only one of the components fails? Do you rollback everything? Do you continue with the rest?

Just to be clear, I'm firmly in the "less smart = more reliable" camp, but even a SoC resembles a distributed system nowadays, and this architecture has its merits. The GPU on my M1 Mac crashed, rebooted itself, and all I saw was 1-2s of glitched graphics - no application noticed. Of course ideally GPUs wouldn't crash, but the same could be said of cars.

View on HN

I don't think "more smart = less reliable" is inevitable.

The goal should be designing it in a way that failure of the more smart features does not spill to critical functionality. Or that you can use it in a degraded state sort of like fly by wire planes do.

I have no car design experience, but here is my 5-minute idea:

For starters, you build a basic car. It drives, it has basic indicators, it has ABS, etc. It has some APIs that can be connected to but there is limited ability to break the basic car through those APIs.

Then you build your smart car around it by adding to the basic car. You add on more computers, electronics, etc. But any time you rip off the smart car functionality, you still have the basic car working.

If there is a need to update software, this should really only be needed for the smart car part. But even if it fails completely, the basic car underneath should be functional and you should be able to use the car albeit without infotainment and navigation and other "stuff". You should be able to do whatever you need to do, call your mechanic and limp there to get them to sort the problem out for you.

That's not how vehicles are designed for many reasons. For one, there's no option where critical components aren't designed with updateability in mind. It's simply a fact of life that code has bugs, and the only way to fix those bugs is updates. It's valid to feel that that those updates should applied at a dealership where the system can be tested afterwards, but that has a lot of implications of its own.

You simply do not understand the point of extracting "basic car" from "smart car".

The point isn't that the software will have no bugs.

The point is that the basic car will have much less software and much more basic software and that it is much easier to test and ensure this software is in working condition.

Also, it is absolutely not true that the basic software has to be updateable. With care, basic functionality can absolutely be completely finished.

Imagine you have been in the market with ABS controller for a decade or more. It has really simple functionality, it should be possible to just bake it and never touch it again. Or maybe a controller that takes care of basic battery protection. Or motor ESC. When the components are simple enough and well defined enough, it should be possible to "finish" the software to a state where it never again needs to be updated. Or updated only in exceptional circumstances.

I have a very deep appreciation for simpler automotive components. Please don't take anything I'm saying as discounting those very real benefits.

What I'm saying is that no practical amount of testing and verification can eliminate the need for updates. Let me give some practical examples I've seen: a bug in the silicon vendor BSP that prevented programs from accessing half of the physical memory. In another case the vendor provided optimistic battery curves that caused safety issues in certain conditions, so changes were needed after units were already in the field. In another case, a factory was taking the stuff we were giving them and using it to build weird Frankenstein images from various binary blobs, so nothing that had been produced for those weeks corresponded to anything that had been tested.

Cars have been built for decades without needing any updates at all. Microcontrollers were first put in cars in the 1970s, for fuel injection control, and later for ABS and SRS. No one ever got a software update for their ABS controller in the 1990s. They've been happily running non-updateable code for many decades now, and all of a sudden you think it's normal to update embedded software? No, it really isn't. If there was a real problem, there was a manufacturer recall and they replaced parts.

Yes, this fatalistic "all software inevitably has bugs" attitude is corrosive and self-reinforcing. How about we replace it with "the more complex the software, the faster it's built, the more likely it will have bugs". At least then, there's a solution. Simplify. Slow down. Test. Remove software. Don't use software when it's not required.

The current free-for-all where we must put complex software into everything is killing us.

Those systems had bugs like anything else, they were just never addressed if the customer didn't bring their vehicle back to a dealership. Have you dealt with code from the 70s-90s before? The codebases I've worked with from that time period were rife with bugs and quality issues we wouldn't consider remotely acceptable today.

I am struggling to understand what it is you're trying to communicate. The original point is that it's quite clear that motor vehicles can physically operate without software, and so therefore smart automotive systems should use graceful degradation in order to avoid situations such as in the OP.

Are you disagreeing that a motor vehicle can operate without software? Or are you disagreeing that a vehicle should continue to operate when smart features are unavailable?

My point is that software is essentially required to build the "basic car" they propose, and all software components potentially need updates. There are valid arguments both ways whether they only happen at dealerships or also OTA or some third way, but they need to happen regardless.

One example where software is practically required is the backup camera. Would you prefer a series of lenses and mirrors?

None of that should be taken to imply that a vehicle should stop operating under any remotely normal circumstances. The situation pictured should not happen to customers.

I wasn’t part of the original convo but I also think you’re missing the point. The backup camera is not part of a “basic car” and a series of mirrors is how we’ve backed up and parked for decades before backup cameras were invented. Seriously imagine the most basic car you can imagine. I mean as little to no software as possible. That’s the basic car. Any other addition, anything else that someone might slap into the car, should not affect the basic cars functionality if turned off or otherwise impaired. This is obviously possible and the example of cars are with basic systems not NEEDING an update and still driving fine is evidence of this.

Perhaps this is a difference of definition, but when I think of "basic car" I understand that to mean the minimum vehicle that can legally operate on the road. Brakes, steering, lights, mirrors, airbags, etc. Backup cameras, AEB, and emissions controls, are all part of that now for better or worse.

As far as NHTSA is concerned, backup cameras are just as mandatory as working brakes. You can't build a vehicle that doesn't have them, and most (all?) states prohibit operating vehicles without these features. Tesla, Mercedes, Nissan, Toyota, Honda, and Subaru have all had safety recalls because of issues with software failing to display that image properly.

Older vehicles don't comply with these newer regulations and are only allowed to operate because of grandfather clauses.

There is no NHTSA standard I'm aware of that requires a vehicle to become inoperable if the backup camera loses functionality. There are no states I'm aware of that require a vehicle to become inoperable if the backup camera loses functionality.

Please provide the C.F.R. link or state statute link you're talking about.

You're missing how FMVSS works. They're all mandatory unless they're one of the listed exemptions to the make inoperative prohibition (part 595) or otherwise stated. States regulate that generally by some variation of "operating an unsafe vehicle" rules, e.g. VC 24002 in CA. Those rules deliberately take a wide view of what unsafe means, but broadly include things that would prevent passing inspections like failing to meet FMVSS. Note that I'm not talking about the actual chance of getting dinged for operating a vehicle this way, because we all know there's effectively no real enforcement here.

Cali. Veh. Code 24002 does not require a vehicle to become inoperable if the backup camera fails. It does not reference backup cameras at all in the text, and there is no case law to support that interpretation.

You have been given plenty of opportunities to make your case and have refused to do so. I think it's pretty clear by now that you're just being disingenuous.

Please enlighten me. What are the vehicle codes that specify which component failures render a vehicle unfit for operation? Let's say something obvious like brakes or lights.

New York: https://www.nysenate.gov/legislation/laws/VAT/T3A9

California: https://casetext.com/statute/california-codes/california-veh...

Illinois: Chapter 12 at https://www.ilga.gov/legislation/ilcs/ilcs3.asp?ActID=1815&C...

Michigan: https://www.legislature.mi.gov/documents/mcl/pdf/mcl-300-194...

Your turn.

You don't need a software-free car (with all the complicated mechanical contraptions this then requires, like carburetors) to have a basic car that doesn't need updates. You just have to write software properly. This has already been done! Cars in the 1980s and 1990s all had ECUs running software, and they never needed updates just to keep the engine running. Later (late 80s and through the 90s) they added ABS brake controllers and SRS airbag controllers, all running software. They didn't need updates either.

The idea that computers need software updates is just insane, and clearly fallacious. Cars worked fine without regular software updates for decades.

> My point is that software is essentially required to build the "basic car" they propose, and all software components potentially need updates.

Both of these are obviously false. Automobiles have existed longer than automotive computers, and lots of software continues to function without updates. I suspect you're aware of both facts.

> With care, basic functionality can absolutely be completely finished.

Yes, but not in SW.

"Even the perfect program still has bugs".

This sounds like the concept of progressive enhancement.

So this is mostly a solved problem at other OEMs, and I do not know how Ford does it.

I know that other OEMs do variant testing, have complex SILS (software in the loop) test systems so that all potential failure scenarios are tested in software prior to update. The downside is that updates are slow to release then, with some other OEMs only putting out an update once or twice a year, but they avoid this scenario.

Software wise, the industry is moving towards defined standardized interfaces for sensors that are versioned. Example: https://blackberry.qnx.com/en/ultimate-guides/software-defin...

Back to the topic though, all major domain ECUs will have an A/B partition. Usually one of these is the OTA master, and has the capability to update the other microcontrollers via UDS. For safety critical microcontrollers that do not have a A/B partition, about half will not support OTA (this is just a thing for small MCUs), and the other half are flashable completely. So something like this is rare and should only happen on a secure boot failure or some other catastrophic scenario, which ideally your SILS system will have tested (SILS won't test all scenarios but will definitely test all failure cases).

Larger OEMs may even have HILS (hardware in the loop systems) where these things are also tested on physical hardware prior to launch, but with software defined vehicles this is slowly going away.

Regarding M1 Macs: Apple bundles all the firmware with the OS and initializes every auxiliary core by uploading the corresponding firmware into its RAM on system boot (as far as I understand it). Not sure if that's something that can be done with a car (e.g. what if a controller loses power physically and reboots from a clean slate?), but that's one way of keeping firmware in sync with the OS release.

Reply on news.ycombinator.com