A recent software update was not successful. Your vehicle cannot be driven | Hacker News Reader

A recent software update was not successful. Your vehicle cannot be driven

twitter.com

273 pointslopkeny12ko179 comments

https://nitter.net/danluu/status/1739387245034139692

Some 20 years ago I was asked to prepare OS distribution with OTA upgrade capability for some PoS system.

I think I prepared two system partitions and a custom MBR which would alternate the system partition on every start (uses one bit of non-volatile memory to alternate selection of partitions on each boot).

When the system starts, it does self-test. If it fails the self-test it just restarts. If it succeeds -- it writes the result on its partition and then looks at the other partition. If the other partition did not succeed the test, it reports the fact back to HQ and overwrites it with the current system.

The upgrade procedure is extremely simple -- the current system writes to the other partition, then restarts. If the upgrade is interrupted at any point in time, the other system will know it was interrupted and will just restart immediately. In any case, after two restarts we are back to the system that initiated the upgrade and if it checks the other system did not start and self-tested successfully, it can redo the upgrade as many time as we want and you still have a working system.

Could it be done better? Probably it could. It took me one day to think up and implement.

I would think Ford could do it better (Edited: yes, not a Tesla)

That is very much standard.

I wouldn't know what the standard is. This seems like something my kid could figure out. It is a shame that multi-billion dollar company that makes consumer products costing hundreds of thousands of dollars can't.

Yep TiVos did something similar

This is a Ford Mach-E, not a Tesla.

My router does that. It’s the wrt3200 and has two partitions and if it fails at booting three times in a row it will boot the old partition. I know because i have bricked and unbricked it this morning whole updating openwrt ^_^”

A/B boot is old, good, and proven, but what do you do when you have dozens of separate microcontrollers, each governing a semi-independent subsystem, all running their own software that needs its own updates? It would make sense from safety/reliability PoV when a crash/bug in a slightly less critical system (like adjusting mirrors) does not affect a slightly more critical system (like indicators). What do you do when updating only one of the components fails? Do you rollback everything? Do you continue with the rest?

Just to be clear, I'm firmly in the "less smart = more reliable" camp, but even a SoC resembles a distributed system nowadays, and this architecture has its merits. The GPU on my M1 Mac crashed, rebooted itself, and all I saw was 1-2s of glitched graphics - no application noticed. Of course ideally GPUs wouldn't crash, but the same could be said of cars.

I don't think "more smart = less reliable" is inevitable.

The goal should be designing it in a way that failure of the more smart features does not spill to critical functionality. Or that you can use it in a degraded state sort of like fly by wire planes do.

I have no car design experience, but here is my 5-minute idea:

For starters, you build a basic car. It drives, it has basic indicators, it has ABS, etc. It has some APIs that can be connected to but there is limited ability to break the basic car through those APIs.

Then you build your smart car around it by adding to the basic car. You add on more computers, electronics, etc. But any time you rip off the smart car functionality, you still have the basic car working.

If there is a need to update software, this should really only be needed for the smart car part. But even if it fails completely, the basic car underneath should be functional and you should be able to use the car albeit without infotainment and navigation and other "stuff". You should be able to do whatever you need to do, call your mechanic and limp there to get them to sort the problem out for you.

That's not how vehicles are designed for many reasons. For one, there's no option where critical components aren't designed with updateability in mind. It's simply a fact of life that code has bugs, and the only way to fix those bugs is updates. It's valid to feel that that those updates should applied at a dealership where the system can be tested afterwards, but that has a lot of implications of its own.

You simply do not understand the point of extracting "basic car" from "smart car".

The point isn't that the software will have no bugs.

The point is that the basic car will have much less software and much more basic software and that it is much easier to test and ensure this software is in working condition.

Also, it is absolutely not true that the basic software has to be updateable. With care, basic functionality can absolutely be completely finished.

Imagine you have been in the market with ABS controller for a decade or more. It has really simple functionality, it should be possible to just bake it and never touch it again. Or maybe a controller that takes care of basic battery protection. Or motor ESC. When the components are simple enough and well defined enough, it should be possible to "finish" the software to a state where it never again needs to be updated. Or updated only in exceptional circumstances.

I have a very deep appreciation for simpler automotive components. Please don't take anything I'm saying as discounting those very real benefits.

What I'm saying is that no practical amount of testing and verification can eliminate the need for updates. Let me give some practical examples I've seen: a bug in the silicon vendor BSP that prevented programs from accessing half of the physical memory. In another case the vendor provided optimistic battery curves that caused safety issues in certain conditions, so changes were needed after units were already in the field. In another case, a factory was taking the stuff we were giving them and using it to build weird Frankenstein images from various binary blobs, so nothing that had been produced for those weeks corresponded to anything that had been tested.

Cars have been built for decades without needing any updates at all. Microcontrollers were first put in cars in the 1970s, for fuel injection control, and later for ABS and SRS. No one ever got a software update for their ABS controller in the 1990s. They've been happily running non-updateable code for many decades now, and all of a sudden you think it's normal to update embedded software? No, it really isn't. If there was a real problem, there was a manufacturer recall and they replaced parts.

Yes, this fatalistic "all software inevitably has bugs" attitude is corrosive and self-reinforcing. How about we replace it with "the more complex the software, the faster it's built, the more likely it will have bugs". At least then, there's a solution. Simplify. Slow down. Test. Remove software. Don't use software when it's not required.

The current free-for-all where we must put complex software into everything is killing us.

Those systems had bugs like anything else, they were just never addressed if the customer didn't bring their vehicle back to a dealership. Have you dealt with code from the 70s-90s before? The codebases I've worked with from that time period were rife with bugs and quality issues we wouldn't consider remotely acceptable today.

I am struggling to understand what it is you're trying to communicate. The original point is that it's quite clear that motor vehicles can physically operate without software, and so therefore smart automotive systems should use graceful degradation in order to avoid situations such as in the OP.

Are you disagreeing that a motor vehicle can operate without software? Or are you disagreeing that a vehicle should continue to operate when smart features are unavailable?

My point is that software is essentially required to build the "basic car" they propose, and all software components potentially need updates. There are valid arguments both ways whether they only happen at dealerships or also OTA or some third way, but they need to happen regardless.

One example where software is practically required is the backup camera. Would you prefer a series of lenses and mirrors?

None of that should be taken to imply that a vehicle should stop operating under any remotely normal circumstances. The situation pictured should not happen to customers.

I wasn’t part of the original convo but I also think you’re missing the point. The backup camera is not part of a “basic car” and a series of mirrors is how we’ve backed up and parked for decades before backup cameras were invented. Seriously imagine the most basic car you can imagine. I mean as little to no software as possible. That’s the basic car. Any other addition, anything else that someone might slap into the car, should not affect the basic cars functionality if turned off or otherwise impaired. This is obviously possible and the example of cars are with basic systems not NEEDING an update and still driving fine is evidence of this.

Perhaps this is a difference of definition, but when I think of "basic car" I understand that to mean the minimum vehicle that can legally operate on the road. Brakes, steering, lights, mirrors, airbags, etc. Backup cameras, AEB, and emissions controls, are all part of that now for better or worse.

As far as NHTSA is concerned, backup cameras are just as mandatory as working brakes. You can't build a vehicle that doesn't have them, and most (all?) states prohibit operating vehicles without these features. Tesla, Mercedes, Nissan, Toyota, Honda, and Subaru have all had safety recalls because of issues with software failing to display that image properly.

Older vehicles don't comply with these newer regulations and are only allowed to operate because of grandfather clauses.

There is no NHTSA standard I'm aware of that requires a vehicle to become inoperable if the backup camera loses functionality. There are no states I'm aware of that require a vehicle to become inoperable if the backup camera loses functionality.

Please provide the C.F.R. link or state statute link you're talking about.

You're missing how FMVSS works. They're all mandatory unless they're one of the listed exemptions to the make inoperative prohibition (part 595) or otherwise stated. States regulate that generally by some variation of "operating an unsafe vehicle" rules, e.g. VC 24002 in CA. Those rules deliberately take a wide view of what unsafe means, but broadly include things that would prevent passing inspections like failing to meet FMVSS. Note that I'm not talking about the actual chance of getting dinged for operating a vehicle this way, because we all know there's effectively no real enforcement here.

Cali. Veh. Code 24002 does not require a vehicle to become inoperable if the backup camera fails. It does not reference backup cameras at all in the text, and there is no case law to support that interpretation.

You have been given plenty of opportunities to make your case and have refused to do so. I think it's pretty clear by now that you're just being disingenuous.

Please enlighten me. What are the vehicle codes that specify which component failures render a vehicle unfit for operation? Let's say something obvious like brakes or lights.

New York: https://www.nysenate.gov/legislation/laws/VAT/T3A9

California: https://casetext.com/statute/california-codes/california-veh...

Illinois: Chapter 12 at https://www.ilga.gov/legislation/ilcs/ilcs3.asp?ActID=1815&C...

Michigan: https://www.legislature.mi.gov/documents/mcl/pdf/mcl-300-194...

Your turn.

You don't need a software-free car (with all the complicated mechanical contraptions this then requires, like carburetors) to have a basic car that doesn't need updates. You just have to write software properly. This has already been done! Cars in the 1980s and 1990s all had ECUs running software, and they never needed updates just to keep the engine running. Later (late 80s and through the 90s) they added ABS brake controllers and SRS airbag controllers, all running software. They didn't need updates either.

The idea that computers need software updates is just insane, and clearly fallacious. Cars worked fine without regular software updates for decades.

> My point is that software is essentially required to build the "basic car" they propose, and all software components potentially need updates.

Both of these are obviously false. Automobiles have existed longer than automotive computers, and lots of software continues to function without updates. I suspect you're aware of both facts.

> With care, basic functionality can absolutely be completely finished.

Yes, but not in SW.

"Even the perfect program still has bugs".

This sounds like the concept of progressive enhancement.

So this is mostly a solved problem at other OEMs, and I do not know how Ford does it.

I know that other OEMs do variant testing, have complex SILS (software in the loop) test systems so that all potential failure scenarios are tested in software prior to update. The downside is that updates are slow to release then, with some other OEMs only putting out an update once or twice a year, but they avoid this scenario.

Software wise, the industry is moving towards defined standardized interfaces for sensors that are versioned. Example: https://blackberry.qnx.com/en/ultimate-guides/software-defin...

Back to the topic though, all major domain ECUs will have an A/B partition. Usually one of these is the OTA master, and has the capability to update the other microcontrollers via UDS. For safety critical microcontrollers that do not have a A/B partition, about half will not support OTA (this is just a thing for small MCUs), and the other half are flashable completely. So something like this is rare and should only happen on a secure boot failure or some other catastrophic scenario, which ideally your SILS system will have tested (SILS won't test all scenarios but will definitely test all failure cases).

Larger OEMs may even have HILS (hardware in the loop systems) where these things are also tested on physical hardware prior to launch, but with software defined vehicles this is slowly going away.

Regarding M1 Macs: Apple bundles all the firmware with the OS and initializes every auxiliary core by uploading the corresponding firmware into its RAM on system boot (as far as I understand it). Not sure if that's something that can be done with a car (e.g. what if a controller loses power physically and reboots from a clean slate?), but that's one way of keeping firmware in sync with the OS release.

You speak as though this is an ancient hack, but this is exactly how it still works in a lot of modern devices. https://source.android.com/docs/core/ota/ab/ab_implement

It’s a bit troubling that automakers (and, well, anyone who is building things that have mechanisms for software updates) aren’t making software updates failsafe. If you have multiple copies of the image on storage, and you take care not to touch the currently-running image, then a failed software update to the alternate image shouldn’t have any impact. Think of it like blue-green deployments on the cloud; and it’s what TiVo was doing with software updates to their DVRs all the way back in the late 1990s.

My router has dual firmware partitions, the fact a car doesn’t is inexcusable

It should be mandated, but corporations dominate legislation and regulations.

"You'll drive up costs for the hard-working American consumer"

"We'd have to cut salaries or fire factory workers"

"We wouldn't be able to compete against that car company from Nothereistan!"

etc.

Absolutely. One fact I recall from many years ago (so I don't know how true it is now) is that auto makers were shipping 7 year old silicon (and thus board support packages for their software). The guys working on chips and CPUs were writing software and it wouldn't hit the road for 7 years.

This was why infotainment systems lagged the state of the art for so long - they were apparently shipping components that would have belonged in a smartphone from more than half a decade before.

In this case, I believe we had a few phones get A/B updates by Android 7, which was late 2016. Giving it a year or so to filter to more CPU chipsets, that would mean (if the old rule held true), we would expect to see A/B hitting new cars in the next year or two. This assumes auto makers are using enough of the Android BSP to get the bootloader with A/B (which isn't a given), and then that they use it!

Anyone with any experience of auto software know if the teams doing software are experienced and understand the platform side of things, or if it's a traditional front-end heavy "crunch release" type affair? I've seen some pretty poor quality infotainment systems in the past, so my assumption is inexperienced devs with limited platforms knowledge, focused on "get it working to ship it", but would be interesting to hear from anyone with experience of how this really goes down.

It was a great advantage, both in marketing and the real usage, when the motherboards started to support the Dual BIOS (GIGABYTE term, AFAIR). 2003 I think?

I'm sure the infotainment system does, but that is not all that is being updated; you have many more processors in ECU, assistance systems and so on that are also sometimes getting fed new firmware over various wonky buses as part of an upgrade. In some instances there might even be multiple levels involved in the firmware distribution.

So? Everything that can update firmware should do the same thing.

And then what? A/B update isn't magic dust, if some sub component ends up with an incompatible firmware because it had to boot to the alternate partition, you might still have an incompatible overall system.

(Not like the subsubsupplier of that module even specced the extra flash memory for A/B update, hah)

Retain the known good firmware for each controller on master disk A, if the update fails, reflash all components.

The real question is why would they be doing OTA upgrades of such critical systems.

This is more commonly known as A/B partitioning and as you say, is standard practice in embedded devices. It is indeed disturbing if automakers have not adopted this, nor a rollback mechanism.

Tesla is. Failed updates are non-destructive, they also have an RTOS gateway isolating entertainment software from the software needed to drive the vehicle.

https://www.pentestpartners.com/security-blog/reverse-engine...

And this is their decade old architecture. If you watch some cybertruck videos you’ll see it is much more elegant now.

Tesla is very very good at software engineering. They still need to find the balance in their software and process engineering, though.

You are talking about the company that wrote logs to the fully read-write filesystem of their infotainment system and killed the EMMC from raw flash cycles within as few as 2 years. Recalled 10 years later.

Yes, this happened to me, although it took 10 years. Free replacement, and the car still worked fine.

It's still a major embedded engineering oversight on their part. We had fallback developed for such cases in $5 embedded devices.

Calling out Tesla as a model for excellent software is insulting to SW that's actually quality.

What Tesla SW has and others lack, is a great "wow" factor, which consumers who don't get to see how the sausage is made, mistakenly associate with 'quality' because it behaves like their iPad, but don't know all the skeletons in that closet.

Yeah, it's also a mistake they made 10 years ago.

Tesla is very good at automotive software engineering.

>Yeah, it's also a mistake they made 10 years ago.

You're making it sound like that mistake was an "I spilled my coffee on you boo-boo" and not a terrible engineering decision that should not have passed anyone with actual experience in the embedded industry, let alone a team. If this is the kind of engineering oversights that ends up in Tesla SW, only God knows what else they have lurking in there that hasn't yet reared its ugly head.

>Tesla is very good at automotive software engineering.

Citation needed. Repeating something doesn't make it true.

You don’t appear to be arguing in good faith. It was a minor issue they had 10 years ago that was fixed.

Tesla obviously has excellent software, but I’m interested in what you’d consider to be excellent software, and which companies would meet your absurd and non-sensical expectations.

Please name some companies that meet your standards.

Yeah, they have some truly breathtaking software controlling the automatic wipers. It's like a true ai, probably competitive with a 6 month or so old child. The automatic high beams are up there with the pinnacle of software engineering as well.

I've heard a lot about how bad they are, but I've just never noticed either doing anything but their job.

This isn't a tesla-specific thing or even something they innovated. The earliest cars with data networks had this.

It's also not something Tesla does better; numerous security researchers have bypassed that gateway.

Since you bring up Tesla: used to be that Tesla control units would fail after a certain number of drive time hours because they did so much logging to the flash chips they'd wear them out.

In a vehicle which is highly connected and basically should never lose power most of its life, they were writing logs to flash memory. Anyone who has done embedded work can tell you how stupid an idea that is.

Yes, that happened to my car. But one could still drive their car after the failure and there was a free recall.

https://www.tesla.com/support/8gb-emmc-recall-frequently-ask...

Genuine question, in that scenario what should you write the logs to if you wanted to keep the files locally on the car?

I understand EMMC/Flash is a bad idea, but what is the better approach, a HDD/SSD?

It's really interesting that updates can't revert, or enable some limp mode, or disable the offending functionality.

Anyone on the 'net back in the 1990s will remember the image macro, email forwards, and usenet posts about "If cars were made by software companies". It looks like they're finally coming true.

Except the price part. We have an inverse Miley Cyrus situation on our hands.

We should have a law for this. In most countries, new vehicles need to be tested by govt agencies before they are allowed to hit the road. The same should hold after software updates.

Ideally, the update should be done by independent agencies, so they can make sure the software is tested and they can properly inform the user (something which we already know companies don't always do).

The way it is now, it is all too easy for a company to decide to push a quick fix for a problem that really needs a more careful approach (but which might cost the company money).

You want an agency to validate every single software updates to vehicles?

At the very least, I want the agency to keep a hold on the update for $N days.

That way, I am more convinced that the software engineers of the car manufacturer won't push any quick fixes that might cause bigger problems later.

In the case of urgent problems, the correct response is to ground the fleet. This will cost the manufacturer money, but I'm a software engineer for long enough to know that a quick fix won't be a safer solution.

> At the very least, I want the agency to keep a hold on the update for $N days.

What is the difference between this and "No Deploy Fridays"?

I actually think this isn't a bad idea. I really don't want an automaker treating consumer updates like a developer repeatedly pushing nearly empty commits to github to get ci green.

At the very least, if it touches anything that impacts the driving… maybe?

It’s a 3 ton (+) machine with people paying attention less each passing day. It should be taken very seriously.

A scalable way, and which I believe is in place, is having rules and then periodic audits validates that the controls and processes provide a high degree of assurance of compliance with the rules and regulations.

This seems far better than having an agency ad hoc review every single specific software update to vehicles - would people at the agency review the code? Do they write automation? Do they do manual verification?

I see a lot of people asking "How could this possibly happen?" It turns out that there's a common failure mode that makes it past almost every unit test suite: a lack of disk space.

Running out of disk space pops up time and again. I've often wondered why software fails so often in this situation. But it's because the basic components of the system start to break -- things like "Call with temporary file" stop working, because creating a tempfile fails. And usually you're writing that code precisely in a spot where it assumes it can't fail.

This just happened on my PS5. I thought I'd play a bit of Armored Core 6 over Christmas, so I sat down and was greeted by "Can't start game." There was a pending update which couldn't be installed. So I skipped update; same error. But now the update was skipped, so it wouldn't even try to reinstall it. I ended up having to reinstall the whole game. Never seen a PS5 game completely brick itself like that.

Disk space woes are also one of the primary reasons websites go down. Ever forget about a server for two years and suddenly see your service break? It's almost always because the process ran out of file handles (resource leak) or because it ran out of disk space (hello imagemagick tempfiles).

Of course, the bug here could be completely different, but disk space errors are so common that it's as good a guess as any. Computers don't do well when you're out of disk space. MacOS freezes and bluescreens instead of going to sleep, because it can't save the hibernation file to disk. PS5 media recording stops working, because the hour-long recording backlog no longer has room to write the video to disk. This software update obviously passed their unit tests, but failed in this particular case; disk space is one of the few variables that they probably didn't test for.

I wish it were easier to write unit tests for all possible out-of-space situations. There are just so many. Even if it technically doesn't break, it often craps up the experience so much that the user ends up wondering why the software is so bad.

I had a rock solid app that I wrote die on me yesterday because it tried to access a folder and Windows returned "Drive is corrupt," which of course I hadn't handled because that will never happen.

Out of drive space is the worst, though. Like you say, most things don't handle it, they just plough headlong into an install and then die, leaving the whole drive in a horrible unknown mess of files where you might or might not be able to retry the operation, and you might or might not have used up the last few bytes on the drive and now for some reason when you try to reboot it can't even boot the system up because it can't write some temp file it needs.

Could also be out of memory errors which are also tricky. Linux's OoM killer just picks something at random and terminates it. If you're lucky is something your app doesn't need but if you're unlucky and you're not paying close attention to syslogs, you're in for a fun time trying to figure out why your app just stopped for no logged reason.

Btrfs have (had?) interesting failure mode for this scenario, because this is Copy On Write file system it need to allocate new block to modify metadata, when you are out of space, you cannot do that anymore. You might end up in scenario where your filesystem is full but you cannot delete anything because this required allocating new metadata block and your drive is full

Out of disk space errors is why we need larger disks.

I get annoyed at people who put in the smallest possible disk, when a disk 4x larger would cost another $10.

Back in the 1980s I was in the room with a customer who I had written an inspection system for, using a hand-held computer. After downloading the data from the handheld, it would upload a new inspection route.

While demoing the transfer, He asked "What happens if I disconnect this right now?".... I didn't know, and said so.

I week later it was bulletproof... you could unplug it any time you wanted, and never lose data.

That was the days of MS-DOS.

So, here we are, 30+ years later, and computers are nowhere near as constrained as they were back then.

I don't understand how Ford could even allow a screen like that to exist. It's a big red flag saying "Never buy a Ford again, they're unreliable"

As someone who has been considering an EV, I'm noping out pretty hard right now and thinking about buying an old and reliable Honda Accord. A few more of these posts and I'll be driving a Volkswagen bug with a garage full of spare parts ;)

There are EVs without this problem.

Which?

It's not an EV thing. All this same tech is moving into ICE cars too. I know someone with a Genesis GV80 that just randomly threw itself into limp mode and wouldnt go over 20mph. Genesis support gave him the magic button combination to reboot the onboard computers and the car was totally fine. This car has all the same tech as any EV. It gets updates. Has all the sensors, screens, safety tech, advanced cruise (he says it drives itself)... Hell even my mom's Subaru Outback has all this same tech, an onboard modem, updates, etc.

Every passing news cycle, I am getting more and more reasons to repair and stick with my current vehicle. Perhaps I would be better off paying a mechanic to teach me how to do the basic maintenance and repairs.

Can’t recommend highly enough: https://www.howacarworks.com/

Problems can be found just as easily with the wrong year and make of any ICE vehicle. In those vehicles, software updates are generally only done by a mechanic, sometimes loaded from a cd or DVD loaded into the car's stereo.

Except it's not going to ever leave your car inoperable...

Probably, although I don’t think I’ve ever looked nor would the media have a reason to.

EVs get attention as disruptors

Very lame.

My immediate question upon seeing the headline was: Which EV brand are we talking about? Evidently it's not Tesla, because if it had been, the headline would have mentioned it, no doubt. It's a Ford, in case you're wondering. Shame on Ford!

Ford and all other automakers are learning the old-fashioned way that making truly great user-facing software is... harder, slower, and costlier than they expected.

Ford makes bad cars in general, so why would we expect them to make decent UI, let alone truly great?

Does anyone have an original source for this? The picture is reposted from a different social media influencer's tweet https://twitter.com/githii/status/1738289618921951296, but with less compression artifacts and without that influencer's story.

It's not the same photo, but here's one with the same error from from July. https://www.f150lightningforum.com/forum/threads/priority-up... (about halfway down)

How could they waste the opportunity of undivided driver attention not to display any ads?! Show them nearest Starbucks where they can relax, or Lidl where they can buy some Coke. C'mon automotive, hammer in the final nail to your coffin.

saying "for the tow truck operator" is horrifying language to put in a 'your car is bricked' spot

For decades software engineers have moaned “why can’t software engineering be a real engineering discipline like civil engineering”. The travails of the legacy car makers, compared to Tesla or Rivian, show conclusively mechanical engineering is easier.

Or maybe there's just a stronger regulatory environment around mechanical engineering. If you sell cars whose fronts fall off, you're going to get in trouble. If your software updates fail, your users are mostly limited to complaining at you on twitter.

Mechanical engineer is in no way easier. However, it is easier for a better mechanical engineer to spot issues in your design just by eying it, whereas software is for all practical purposes a black box unless you spend 100s of hours reading the code.

You take a software engineer's point of view. The people who build cars, especially at established car companies are mechanical engineers however. And they design consumer technology, they don't design military vehicles or ambulances, so their main goal is not reliability, never has been.

Cars have constantly gotten harder to repair, more brittle, and if a repair is needed, you often have to, instead of being able to swap out a $5 component, replace larger components that the $5 component is part of. This is by design as replacement parts is a high margin revenue source for the car manufacturers.

Why should the software side of things be any different?

>>> why can’t software engineering be a real engineering discipline like civil engineering

Programming pays better than civil engineering or mechanical engineering.

The software team pays better than the hardware team, and programming in support of hardware pays like hardware. When asked why, the answer is: "Software is more valuable because it's closer to revenue." And: "Software has no cost after it has been written once." No manger or engineer can explain what this phrase means, but it's taken as conventional wisdom.

"Software has no cost after it has been written once."

well not if you're running a server.

I've done both. Software engineering is way easier. In traditional engineering you dont have "move fast and break things" and if you get sloppy with edge cases at best it could cost $$$ at worst people die.

Cars have had computers controlling them for decades and didnt have these problems. What's happening now is auto makers are adopting the silicon valley way of doing things which worked ok for webapps but is horrible for hard goods.

The same thing happened to our Tesla Model X a few years back. Tesla even wanted us to pay $200 flat fee to fix the problem they created and that we couldn’t opt out off. I may have used impolite language until they waived that. And that’s not even addressing the massive time sink it was.

My Tesla had an update fail, and it didn't break anything, I just got a new update package the next day.

I also replaced my own headlight and had to run a software update to initialize the new part. I called a service center and had a software update in less than an hour. Try that at Audi.

Tesla knows what it’s doing. Not these guys.

That's the point I'm making. Fail-safe and good testing.

They know so well that there was a report yesterday showing how tesla’s a low quality vehicle. Behind the dacia.

The report regarding TUV inspections?

Do you do understand that failing those inspections doesn’t necessarily have anything to do with the car, right? Tires, wiper blades, or even rust on rotors (due to only using regenerative braking) will cause a fail.

Having owned Mercedes, Audi, Toyota, Porsche, Range Rover, my Teslas have been the most reliable vehicles I’ve ever owned.

Not a single mechanical issue over 4 cars and hundreds of thousands of miles.

This is the way.

Also the scrutiny Tesla has received at the start of it's operations might be quite a bit more than the other makers.

I'm hearing Kia/Hyundai's needing a replacement battery are having to wait months if not up to a year, while new ones sell away.

Buying a used EV will become like buying a used ICE car - there are ways to do it well, and ignoring either is at your peril... and no guarantee a brand new one will be reliable, nor the warranty performant when needed.

Back when it was in news cycles that Tesla's stock couldn't possibly be worth more than legacy automakers' because it was only a matter of time until they came for Tesla's lunch, I couldn't imagine they'd fumble it this badly.

Lord knows Tesla's never had any problems with updates before…

https://teslamotorsclub.com/tmc/threads/my-car-is-bricked-by...

https://old.reddit.com/r/TeslaLounge/comments/112oqln/new_te...

https://teslamotorsclub.com/tmc/threads/failed-software-upda...

https://old.reddit.com/r/TeslaLounge/comments/qt0ekf/2021_mo...

An Audi doesn’t need a software update to replace a light from my experience.

Also, my experience with Audi and BMW are that community tools exists to let you flash modules and change settings all on your own. Try that on a Tesla.

Tesla is way ahead on software imo, but I don’t want my car to be like modern smartphones where they’re dependent on the manufacturer to maintain and keep working.

Mine did - had to code the module to the car.

This is usually because the modules are generic and don’t know how to behave, for example, a light fixture needs to know if it’s US or Germany and work accordingly. You could’ve done it yourself with the proper software and knowledge is my point and most things don’t require this.

I tried to do it myself, but the module was running newer software than the vehicle, and required the other vehicle modules to be flashed to the same software version.

I replaced two of my lights in my car and I had to re-adjust the carpet piece that covers it. That's all I had to do, no software updates, no service centers, nothing.

Are we going in the future to have to call a service center for a software update when we replace a floor mat or a windshield wiper?

What car do you drive? Go replace the headlights in a modern vehicle and see what happens. You have to code the module to the vehicle.

Not a Tesla, this is for sure. As long as there are any cars with physical controls and replaceable lightbulbs on the market, those are the ones I'll be driving. Fortunately, this is still very much the case, especially if a car for you is a means of transportation, not the expression of your ego.

Doesn’t really seem like you even know what we’re discussing.

You don’t need to do any coding when replacing bulbs (assuming the car has bulbs) - we’re talking about the entire headlight, which has modules that require coding. This has been true as far back as 1999, when my BMW needed its HID module flashed.

As for bulbs themselves, even a base corolla now has LED headlights that you won’t be replacing yourself.

I suppose one could consider a Toyota Corolla to be an “expression of your ego”, but I’m not sure who would.

Do car companies outsource their software to fiverr or something? Every single car I rent lately is a buggy piece of shit.

Ugh. I rented a Ford Explorer recently, and somehow they managed to screw up Applr Carplay. I'm in a place I don't know very well, navigating using Apple Maps via Carplay, and my GPS location goes for a wander. Just wanders off so that the navigation gets way confused trying to route me to my designation, because the current location is miles and miles off from where we are. So I unplug my phone and suddenly my GPS location is back to the correct location. HowTF did that even happen?

carplay just seems like crap on all cars I've used (including the new subaru I own)

I don't even try it anymore and just bring a vanilla charger and pipe audio to the car via bluetooth (which generally works without issue on every car I've tried) and the phone for actual navigation

lol even that was broken on the rental Ford Explorer - audio from Siri and music worked, so I know the Bluetooth connection was fine, but the GPS navigation audio didn't speak.

Fwiw there are dozens or perhaps 100s of microcontrollers on cars, usually sharing a small number of comms networks. Not all have the luxury of flash space for dual-boot. It's possible an error on one can affect a large number of others. It should be better, but it's more complex than a single firmware image.

> Fwiw there are dozens or perhaps 100s of microcontrollers on cars, usually sharing a small number of comms networks. Not all have the luxury of flash space for dual-boot.

The auto maker needs to use micro controllers with larger flash memories, or write simpler code.

Does the owner get compensation for this bug? For example, if you need a rent a car for the day, will Ford reimburse the cost? If no, can you sue in small claims court?

I can hear the dejected software engineer that wrote or plugged in that error message after layers of bureaucracy

Not at all. Dejection and uncooperativeness render one a bad team player. The recruitment pipelines aggressively screen against these. Software engineers rarely care or have authority to be ethical gatekeepers.

I’m pretty good at hiding it tbh

Bricking a device like a car is not on , even my router and PC has a recovery option if a firmware update goes south.

Don't they know about A and B images ???.

They cheaped out to ensure their flash memory is adequate for restoring from a bad update.

Cars are so much safer with all the technology we have in them, this is just a bad implementation.

A lot of the actual mechanical/engine related software really does help a tremendous amount. Even something like ABS can be improved with more sensors to help sense exactly what's going on when you need to stop to prevent yourself careening through a barrier and down a cliff in bad weather.

Ever since using computers and waiting for PC/Windows to boot up, I've dreaded software coming to general utility hardware. Unfortunately the ills and lack of quality software is permeating things that doesn't need it. At the very least, separate the core function of driving from everything else (entertainment, navigation, etc).

Let me try to play devil's advocate.

This screen is actually result of alternative boot in dual boot image, but it's not A/B system but system/recovery.

This is valid solution when there is not enough space to fit second full os in available storage. Recovery allows for repeating update of major system.

Another potential issue is that car is composed of multiple connected systems and one "main" computer, it will try to upgrade everything else to the one expected version that was tested with this release. Something failed in secondary module that does not support a/b systems (3rd party?)

Another possibility, update is composed of update and verification of secondary modules and verification is not passing, because update is passing there is not really good way to recover now, because maybe flash is now damaged.

edit: usually all components in car communicate using canbus (including software updates) this usually works in this way: You first switch device into flashing mode by sending speciall command, it will then jump to special mini application that will write new bytes to flash and finally you finalize to reboot. In case something in flashing went wrong and device is crash looping or stuck in infinite loop or in any other way is not able to finish boot, you cannot retry anymore because there is no way to put it again into flash mode as its not able to receive canbus messages anymore. If that is critical component the safest thing to do is prevent car from driving

The car owner's original thread is https://www.reddit.com/r/MachE/comments/18m3ne2/what/

Software and hardware are prone to bugs , but that should not hinder its ability to perform the core operation (driving) of the product unless something major (brake failure) is detected.

https://en.wikipedia.org/wiki/Sturgeons_law

Well I think 90s Honda's are gonna start going up in value.

As a huge fan of '90's Hondas, I agree.

What brand of car? So I know what to avoid?

Screenshot says Ford.

Ford

In the centre over the phone number is "Ford Customer Relationship Center". So I guess it's a Ford.

Looks like the Ford Mach E screen

Found On Road Deactivated

It is a Ford. Either an f150 lightning or a mustang mach e.

https://cars.usnews.com/cars-trucks/ford/mustang-mach-e/phot...

Looks like the Mach-E

I'm sure it's the Chevy Blazer, or at least same problem as. https://www.thedrive.com/news/the-chevy-blazer-ev-launch-has...

Software ate the world.

What kind of car is it?

FORD- fix or return daily ?

I wish someone would make a truly dumb car. Sure you can have microcontrollers for emissions, safety, etc but beyond that give me a 1970s user interface and make the car as independent of technology as possible.

100% this, an electric car could be so reliable and simple to work on and repair if we wanted it to be.

Nissan Versa. It’s also one of the cheapest cars available.

Enjoy

That's why I have kept my '97 dodge dakota all this time... Like you said, dumb, but also using standard components like headlamp bulbs for repair-ability.

Absolutely this. Been hoping that this will become a cool trend someday. Perhaps when teen-age Gen-Alpha or Gen-Beta kids seek to differentiate themselves from their Algorithm-controlled over-sharing IG/Tiktok parents.

My eldest daughter knows her way around cars, and she refuses to buy any car that has any sort of computer in it.

She knows how to change this, repair that, fit whatever and is very happy doing that. But she also knows that as a female if she goes into any garage for a 'diagnostic' she will be overcharged. In fact even for something normal with a 'non-computer' car she will be overcharged.

Cars without computers are like gold.

any car builtin the last 100 years have electrical components. Any car built in the last 50 have "computers".

Even the camshaft-crankshaft system could be considered an analog computer that renders (in realtime) the positions of the valves based on the phase of the pistons. Any car with an internal combustion engine has "computers."

As a fellow “analog” automotive enthusiast, I’m curious how she defines computer in this situation?

Given that ECUs / microprocessors have been standard on cars since the late 60s / early 70s.

For me it’s the network connections and updates that scare me. The software on my ‘99 Benz hasn’t changed in 24 years, and I don’t need it too.

I’ll add a small point to this. The computers my old cars do have, I’m quite thankful for. Being able to purchase a ~100-200 buck code reader and easily plug it in and figure out what my warning lights are trying to tell me has been a great educational tool and saved me a ton in maintenance.

Advanced software can be used for good. It's sad that the same advanced systems that can perform extremely accurate self-diagnostics that empower even the most amateur mechanic to do their own work on extremely complex ICE setups for years and years will also be the component most likely to render the vehicle a 6000-lb brick made of animal skin and rare earth metals

I suspect she may be referring to the stereotypical carbureted V8 + manual / hydraulic auto combination that's still highly popular amongst US enthusiasts, which certainly has no ECU at all.

Microprocessors didn't exist until the early 70s. Late 70s / early 80s is when they started becoming common in cars.

Heh, thanks for the info, I have a bit of a gap of knowledge about that era of vehicles outside of a type II fun experience with an older Ranchero.

I see VW advertising a car with an ECU in 1968, but likely just transistors.

https://bollingermotors.com/b1-b2/

My 2012 Dacia Sandero is as dumb as you can be nowadays I think.

Great news! [1]

[1] https://www.youtube.com/watch?v=A7-7Ps8EWnk

It would not be economically feasible to build. CANBUS has revolutionized cars; now cars look more like a human body, with power and data being distributed to major points and then branching out. Before, everything ran from the fusebox. That meant a LOT of wiring, among other things. Copper is expensive and heavy. Car makers had a lot of incentive to save weight and cost so the wiring was as thin as they could get away with, so stuff never worked quite right.

It would be dangerous, because various driver warning systems and such would not be possible. System that are having a significant impact on per-mile safety for vehicle occupants (road safety for vulnerable road users has plunged.)

It would not be economically viable to operate, because engines are efficient as they are thanks to things like variable valve timing, direct injection, and so on. Some ECUs can sense how good the combustion was on a particular firing cycle via resistance across the spark plug after the ignition event. None of that would be possible with a "dumb" car.

It would also not be economically feasible to repair, because the diagnostic time would be crazy. Trying to diagnose "dumb" cars from the 80's and 90's was a nightmare. You could spend hours just trying to figure out what was wrong in a maze of wiring, relays, resistive packs, analog sensors, vacuum lines, etc. Dealerships are staffed by low-level techs, people on their way to something else or starting their own shop. Car companies couldn't possibly train them on all the intricacies of analog "dumb" car technology anymore.

Now? I plug in a OBD2 wireless module, tap "scan" on my phone, drink a cup of tea while it systematically talks to every module in the car, and I not only get fault codes, they've likely got time and mileage stamps, an indication of whether they were intermittent or continuous.

I can trigger a test on a component.

I can see what the ECU or body control module THINKS is going on versus what's actually going on.

I can tell the body control module to adapt to a new analog sensor.

all without needing to so much as pop the hood, remove a single screw, or upholstery clip.

It's a fucking dream.

The shittiness in "smart" cars is because legislation in the US hasn't kept pace with technology, and that is because legislation and regulation in the US is dominated by corporate interests.

I know motorcycles are basically just an occasionally-useful game of russian roulette but every fucking automotive news cycle has me thinking about how they're basically the only way to buy just a motor and some wheels anymore.

You'd better check twice. Some had dial-out networking for years, like BMW.

oh, well, shit, there goes the g310gs I had being eyeing for noodling around the state.

Great idea for another country, I the US there are just too many laws.

I know there's no market for people like me, but I would never buy an internet connected car. It just seems so fucking insane to me that people are willing to accept relying on something that can be bricked remotely. Same goes for any sort of involuntary OTA updates to anything, and software as a service in general. The thing that matters to me is that my tools still work tomorrow when I need them to, and if they break it'd better be my fucking fault. We live in fucking clown world. Thank god you can still perma disable windows update, having a cell phone that breaks itself every few months is bad enough.

You'll change your mind the first time you turn on the heat from across the parking lot while sipping coffee in the ski lodge, or need to check the charge status while eating lunch, or point stuff out on the satellite imagery on the live map.

I know it's easy to imagine that the product you think is "a car" is finished, but product categories evolve. People want their devices to do different things today than they did 20 years ago, and next year will be more changes. Connectivity isn't going away, the market has spoken.

All this is possible without integrating the software into the vehicle itself, making it brickable.

Reliability and repairability going down is not "evolution" in any sense.

And the market hasn't really said anything when those things have been pushed into consumers without much choice. Saying "the market has spoken" in this case is like saying the market has spoken when someone claims that slapping a fruit logo is enough to sell computers. Turns out the choice of which car to buy is non-binary, it's not "internet connected" vs "non internet connected".

I feel it's not completely impossible to design a system that is properly partitioned, such that an update failure to the seat heating software doesn't brick the car. Updates to core systems should be rare in the extreme.

I would say part of the problem is that they want the option of updating seat heating software, rather than actually properly testing the up and down buttons and the thermostat.

That's an extreme example, but: definitely.

Having an interface to turn seat heating on/off if simple stuff. Having the upgradeability, DRM, the subscription thing, and circumvention protection is significantly more complicated.

> All this is possible without integrating the software into the vehicle itself, making it brickable.

Sure. And Ford messed up. But the request upthread wasn't for a vehicle with robust software update semantics, it was for a vehicle without connectivity. That's not going to sell well.

Tesla does it well with their real time gateway mechanism isolating entertainment software from crucial vehicle driving software.

>the market has spoken.

In the sense that a market is made up of buyers and sellers, and the sellers have decided to offer no other option yes.

I think the issue is more of the design and ownership of these things. No one wants an update bricking their car. I doubt a lot of people here would object to a discrete subsystem where you could poll charge status, turn on heating etc.

You can buy many vehicles today dating all the way back to the turn of the last century. Here's some listings for steam cars if you insist on no electronics whatsoever https://www.steamcarnetwork.com/for-sale

Whatever your line is in the sand for tech integration, you have many great options from before that point.

Except you're proposing a false dichotomy of no tech, and complete tech integration.

Would you also propose a calculator as a valid alternative to a locked down tablet/phone/pc that you can't install your own software on?

I don't understand where you are trying to go with this - steam cars have the same primary feature as any other vehicle - longitudinal and lateral control to traverse asphalt bands across the country. They can be retrofitted with any modern convenience feature as that stuff does not need to be integrated into the propulsion system. The same can be said of any other vehicle made between then and now. Of course, nobody has retrofit an early 1900s vehicle with 12 impact sensors and multistage airbags, and no modern manufacturer has offered a steam powerplant, because nobody actually wants this. Products are sold on features, not principles. In the US, you can (and people do) build just about anything and drive it legally on the road, so if not a single person has created (whatever it is several people in this thread think should exist), it would be preposterous to think a manufacturer would do it.

This answer is like Microsoft saying the 1990s that "abacus exist, so we're not really a monopoly".

People want modern cars, modern TVs and modern computers. They don't want a steam engine. But modern doesn't mean "crap that the manufacturer can brick remotely".

Sure, it's the market that has spoken..

https://foundation.mozilla.org/en/blog/privacy-nightmare-on-...

My car does this and more, so perhaps I chose my words unwisely. (I live in a van I have pretty much full remote control of save actual driving, I'm out right now and I just turned on my espresso machine so it's preheated when I get home). I don't mind connectivity, I mind having to trust other idiots, I'm happy to trust this idiot. I want to be able to approve every change. It's MY car.

I want to have the final say about what happens with the things I own. I want to be able to rely on them.

You'll enjoy this, should start right where he talks about his experience buying a car:

https://youtu.be/hlrv7DIHllE?t=1203

I'm fine with having both. A dumb ICE car for emergencies and an electric car with bells and whistles for daily use

Reply on news.ycombinator.com