Safe and reliable production changes, and how Rivian recently got this wrong | Hacker News Reader

Safe and reliable production changes, and how Rivian recently got this wrong

blog.substrate.tools

60 pointskelp45 comments

> In this post I’ll be discussing a recent over-the-air (OTA) software update to Rivian vehicles that went badly. It is speculative; I have no insider knowledge of Rivian’s software, systems or practices.

and then it goes into canary testing, "pre flight tests" and rollback

Those are very common practices and he's clear about saying "that probably means..." and "This seems like...". Seems like fine writing to me.

sure. my point is that the title doesn't say a thing about what it actually is about, and what the article is about indeed.

To be fair, Volvo did the same thing recently, so it's not just weird American startups that do this - Volvo never released the numbers on how many cars were affected, but I'm a member of several facebook groups for Volvo owners and it was just like an onslaught, people were posting daily warning others about not applying the latest software patch or it had a good chance of bricking your car. Absolutely no idea how that got released into the wild.

I hadn't heard about the Volvo one! I had a 2022 Volvo C40 before I got my Rivian R1T.

When I first got the Volvo the GPS and LTE connection would periodically stop working for a day or two. They pushed a fix for it. Later they added CarPlay, which wasn't there when I got the car. Good updates. But not as frequent at Rivian.

Was Volvo able to fix it with another OTA or did people have to go in for service?

Imagine being rich enough to buy high end cars every year and worry about OTA updates. What a world.

Beats being so poor you can't afford a car, or the only car you can afford is slowly rusting itself away or one no-longer-mass-produced part away from making a repair out of your financial reach. I'd take bad OTA updates anyday.

GP is referencing the "having the being stolen from stolen".

Nevermind that the wealthy ain't fretting for one second over OTA updates. Imagine a personal vehicle budget where the Porsche is technically your daily driver, but you never drive it because you're driven everywhere.

A /personal IT team/.

A boy can still dream!

Seems like this batch of half baked cars receiving OTA updates are more likely to be the ones that are in repairable in the not so distant future, whereas parts for a 1990s toyota gmc or ford anything can be found cheaply and installed by any ambitious teenager.

> had a 2022 Volvo C40

I tried to buy one, but the dealer in Montana was such a pain about it! How did you like it as a car?

I liked it a lot. Not great range, but new ones improved that and I still road tripped it all up and down the west coast. Sometimes regret selling it.

> With the botched 2023.42 update Rivian explained that they pushed the wrong build with the wrong certificates to customer vehicles. This made me immediately think they probably don’t have a canary fleet of vehicles that they roll out to first.

Sounds to me like they certainly could have a canary fleet, but if they do, they didn’t have sufficient process controls to only allow updates to the public after having gone through canary testing.

Repeating what you said in my own words to ensure I understand it.

I think you're suggesting they might have a canary fleet, but there wasn't anything/enough that preventing a mistake from bypassing the carry fleet before going to production?

Could be!

Exactly.

Software is such a powerful tool that I understand motor vehicles having as much code in them as they do. What I don't want is for that software to be shoddy or for it to spy on me. I also want complete control over whether or when it changes, and I want to understand the nature of and reason for the updates, just as I do for my Linux laptop on which I use apt-listchanges before accepting upgrades.

For example:

apt-listchanges: Changelogs

---------------------------

bind9 (1:9.16.48-1) bullseye-security; urgency=high

  * New upstream version 9.16.48
   - CVE-2023-4408: Parsing large DNS messages may cause excessive CPU
     load
   - CVE-2023-5517: Querying RFC 1918 reverse zones may cause an assertion
     failure when "nxdomain-redirect" is enabled

...

glibc (2.31-13+deb11u8) bullseye; urgency=medium

  * debian/patches/any/local-qsort-memory-corruption.patch: Fix a memory
    corruption in qsort() when using nontransitive comparison functions.

...

imagemagick (8:6.9.11.60+dfsg-1.3+deb11u2) bullseye; urgency=medium

  * Fix CVE-2021-3574: memory leak was found in TIFF coder
  * Fix CVE-2021-4219: a special crafted file could lead to a DOS.
  * Fix CVE-2021-20241 / CVE-2021-20243: divide by zero in
    some coders (Closes: #1013282)

And so forth. If something makes me raise an eyebrow I can go look at the source code to see what's up. I also like for upstream maintainers and other members of the community being able to do that same. Having that process in place helps keep everyone honest. Why not have this for my car's computers too?

For the install I would rather download a signed image onto a USB drive and flash from that versus letting my car communicate with the mothership indiscriminately. I also want to downgrade at any time with a previous known-good image when there's something about the update that I don't like. For example, if it sends my car's console unit into a bootloop.

I've also often thought about what an open source car software stack might look like, but with different motivations. I'd love to be able to see more diagnostics about what the car is actually doing and to add 3rd party extensions.

For me, I don't want to have to tinker too much, but I want to be able to. I think the ideal would be something like SteamOS on Steam Deck where you can get into the system, and you can change or add things. But the default is just having it all take care of for you.

That said, cars have all sorts of regulations about how certain things work. I have no idea how any of the above ideas would interact with those regulations.

Imo the current continuous update while letting customers beta test new updates starts to fall apart as the cost of the hardware increases.

Bricking am expensive smart phone is infuriating, but bricking an expensive household appliance or even more expensive automobile is a non starter.

The signed image on USB seemed to be the norm from maybe 2010-2020 but it seems cellular connectivity has gotten too cheap and telemetry too valuable...

In the case of Rivian they have been pushing very meaningful improvements on a roughly monthly basis via OTA.

I got my R1T in June 2023 and since here are a few things they've improved, just off the top of my head, not bothering to look it up:

1. Significant improvement to ride quality via different / better suspension tuning.

2. Ability to schedule warming the cabin and pre-condition the battery

3. Completely redesigned the UX for setting drive modes and suspension height (for the better IMO)

4. Added a ton of car info, like battery temp, motor temp, and other info like altitude, various angles the vehicle is at (for off-roading), degrees the front wheels are turned

5. Added additional settings for ride softness / firmness (I got this update yesterday and haven't tried it yet)

When an update is ready I get a notification in the car and from the Rivian app on my phone. I can just hit apply and it installs it.

IMO a USB install would be a substantially worse experience and it would be much less likely that customers would actually install it.

But, for the type of person who just wants the car to stay the same as it was the day they bought it, and never change, it's not the vehicle for them. Personally I really like that it's continually improving and I don't have to go in for service or even go out to the truck to do an update.

> But, for the type of person who just wants the car to stay the same as it was the day they bought it, and never change, it's not the vehicle for them.

I never said I didn't want updates. What I said is that I want to understand what the updates are and then choose to upgrade or downgrade when and how I see fit. Or better yet make the updates OSS and then let me do my own builds with the features and functionality I prefer as they are developed.

One thing that is right is that a Rivian is not for me, for a lot of additional reasons.

I wasn't trying to suggest what you personally want or don't want. Just that I could see how some people do not want their car interface to change, or even ride quality to change.

It's not that I don't want improvements, I modify my cars for exactly that reason, but I want reliability. Improvement to the ride quality shouldn't be a manufacturers after-thought. UX adjustments are nice, adding further visibility to system features, great. OTA updates on systems impacting car functionality or safety, no. These things should be tested thoroughly enough before release to not require periodic updating. They should be stable and tested enough that an difficult to apply update is a reasonable cost. These are not the systems to fail and fix on repeat.

Software in cars and OTA is the stupidest thing in recent years. Like the damn laptop riddled with mediocre software wasn't already frustrating enough, let's fuck up cars too.

Same things with phones. A phone should just make calls, it doesn't need software.

After the deluge of spam calls I receive daily I'm somewhat inclined to opt for the opposite direction. No more phonecalls and just using apps for communication.

That's so much better. I only communicate with people who have the rich man's apps. No unintended negative side-effects can possibly be found there!

At this point I would like to be there. Telecom companies don't seem to care about spam calls much.

Phone has become much more than a caller. It's become a cybernetic extension of our self, and are even used to validate our identity.

To be fair cars as currently designed are a pretty stupid idea to begin with. Let’s just waste energy carrying around 5000lbs of car at ridiculous speeds to move around a 200lb person.

In this case it's even what, 7000 or 8500lbs car? Which is even crazier when you consider road wear scales on the 3rd or 4th power of weight.

(Edit: 4th power, damn) https://en.m.wikipedia.org/wiki/Fourth_power_law

So a 8500 car does 64x as much road wear as my small sedan.

Wrong

https://twitter.com/ajisuzu1/status/1681123111364620294?s=46

TLDR; From a road wear perspective there is no real difference between a heavy EV and a lightweight smaller ICE.

Edit: Not sure why I get downvoted so heavily. It is just a fact that the weight difference between an EV and comparable ICE has no measurable difference to road wear.

People like the above poster just like to touch on the fourth power law but not how the calculations actually work.

ESAL is part of that calculation. A 5 axel semi has a ESAL of 2.35, a dumptruck ~4, a 3.5ton vehicle .004, a 3ton vehicle .002. When we are talking about the difference in hundreds of pounds between EV and ICE, there is no wear difference.

No one can read that thread.

TLDR paved roads are generally designed to handle large trucks and construction equipment. On such roads passenger vehicles (even heavy electric vehicles) have a negligible impact on pavement life.

The difference in road wear between a 2k lbs. vehicle and an 8k lbs vehicle is too small to matter.

Okay that makes sense, basically amdahl's law.

I guess it'll be interesting when we are trying to support electric medium duty or heavier trucks, like WA is trying to do. Guess they'll be subject to Class 7 & 8 weight anyways, because if you try to make a currently-medium-duty truck into an EV it's way over the limit.

I mostly just have doubts about our current revenue model scaling for it (since it's heavily reliant upon gas tax and the truck weight $$ amounts don't match up), and the general lack of lighter EVs in the US. Something will have to change there

I'd be totally happy in the city with a 2-2500 lbs BYD Seagull or whatever. But that vehicle doesn't exist in the US.

A vehicle the size of a Seagull is practically a non-starter in the US in terms of mass-market appeal. Most US consumers think of the Chevy Bolt as too small of a car, and that's like 20" longer than a Seagull.

Which is really a shame :/

I want a $10k small car for just intracity trips. Easier to park, small battery charges fast enough at home even on 120V 20A. Cheaper to insure.

This is a terrible argument against cars. You could say the same for a train, let's waste energy carrying around 1,500,000 pounds of train to move around 120,000 pounds of passengers.

You need lot more lbs of cars to move 120k lbs of passengers, that's the argument in favor of trains.

Yeah about twice as much. But it's still a bad argument because pounds don't really matter.

Yes, infrastructure is more more important, and railway infrastructure is much more sustainable to maintain in long term.

Unrelated gibberish

If you live at a place with decent public transport, you'll ditch cars first thing. That's the biggest argument against cars.

I agree there are better arguments. My point was that the specific argument I was replying to was bad.

Except your example makes it clear that a train is a way better deal

No because it's not a problem for things to be heavy.

Scooters and motorcycles are much more efficient in this regard, but uptake has been very limited in Western countries when compared to the pervasive use in Southeast Asia.

Reply on news.ycombinator.com