Fiat Chrysler pushed a UConnect update that causes constant reboots | Hacker News Reader

Fiat Chrysler pushed a UConnect update that causes constant reboots

jalopnik.com

65 pointsrbanffy60 comments

They later posted on their Twitter feed that the problem was caused by SiriusXM, and as a stopgap fix they'd be disabling that. But this mostly sounds like a massive failure to follow software-industry best practices. In particular:

* They released the update a Friday (bad; you need staff ready to respond to problems

* They released the update to everyone at once (bad; you want to deploy to a small subset first, monitor that subset for problems, then deploy to the rest);

* They didn't have a rollback option ready

* They didn't isolate software components well enough (SiriusXM shouldn't be able to crash the whole OS)

* They didn't have good instrumentation - it took until Monday for them to find out there was a problem, and two days after that to find out which component was causing it.

The center-console software is mostly not safety critical, but there is some possibility for a problem there to be a safety issue, which makes these issues quite concerning.

SiriusXM may be responsible for transmitting the OS update for their hardware stack over the satellite, but the chances are vanishingly small that SiriusXM is responsible for designing, testing, and queuing for delivery OS updates to a car manufacturer's hardware stack.

Blaming SiriusXM for the issue masks that, but they can be called to question more accurately as a result by the press:

* Did Fiat-Chrysler negotiate phased-deployment and day-of-week criteria with SiriusXM, or was SiriusXM alone responsible for those choices?

* Can SiriusXM release OTA firmware updates for the hardware stack in these vehicles without the cooperation and signoff of Fiat?

* Why did the firmware's watchdog process fail to detect the crash-reboot loop (or is no watchdog present), and do they intend to correct that failing in the future so that such crashes result in an automatic firmware rollback in the future?

FCA ISOs are bundled from various vendors. The QNX OS from Harman, some of the uconnect apps and the data connectivity is from Sprint.

* FCA is an automotive company, I presume their s/w release management dont follow best practices.

* SiriusXM cannot push OTA updates. It has to be provisioned from FCA and pushed through their uconnect DRM process to the headunit.

* There is a watchdog in FCA UConnect but for the apps. Im not sure they have it for the OS. I know for sure that the watchdog restarts daemon apps when they crash.

> The QNX OS from Harman

Is it true that Panasonic makes some head units for FCA? Are all UConnect head units Harman?

I think the 6.5” screensizes and MY13 or older ones were from Panasonic. Harman took over with the new touch enabled screen models. The consumer facing apps run on J2ME LWUIT on top of JamaicaVM in QNX.

Mitsubishi made a handful of UConnect head units (specifically the RHB and RBZ models) in the early 2010s as well.

The fact that they didn't do 5%/10%/25%/50%/100% rollout over a week or two is just pure amateur hour. It's like OTA 101 for this exact reason.

People always say the big 3 are going to catch up with Tesla but if anything this shows they have a heck of a long way to go.

> People always say the big 3 are going to catch up with Tesla...

Who are "people?" I don't own a Tesla, but from what I've read, they are not great at manufacturing, and that's something that takes awhile to fix. My guess is that companies with decades of experience in building physical things, but little in software, have a decent chance.

Agreed, like Blockbuster

Or Netflix. They got a whole lot worse as they started failing at shipping DVDs. Then BitTorrent ate their lunch.

EDIT: Unfortunately, Tesla is in the business of moving humans around, and they are physical things with mass and volume.

I don't believe Blockbuster manufactured anything.

Your list of mistakes seem to me like (to borrow from Fight Club) a list of unpolished brass on the Titanic. There is no right way to implement a software system integral to a car that auto-updates over the air at the whim of the manufacturer. And I'm uneasy about recommending for a safety-critical system any "best practices" from the industry that regularly brings us blue screens of death (or modern equivalents), gaping security holes, megabytes-stuffed webpages full of coin miners and ad tracking, etc.

A staged roll out (with a/b test manner with equal control/experiment arms) and monitoring of health metrics would totally catch this.

A staged roll out would mean only 5% of customers' cars lost their radio, navigation, climate control and rear-view camera.

I'll admit that's better than 100%, but it's still garbage. Where was the pre-release testing?

Why not start with 1 car, 10 cars, 0.1% and beyond.

My point is prerelease testing can only catch so much. you can think of many bugs that affected only people with certain settings etc. It is hard to replicate all the conditions appearing in the wild. This could be as simple as "last software/firmware version", or as complex as "car stalls as it makes a u turn". Hard to test all these.

It's things like this that make me enjoy older cars from the 80's and 90's.

I used to own an old Datsun truck. If anything went wrong, I knew that I'd most likely be able to fix it myself, or if I couldn't my mechanic could. There was no software to break, no security vulnerabilities that would cause the brakes to fail, no DRM requiring Genuine Datsun© lightbulbs for the lights to work.

Sure, the thing was a deathtrap, but I owned everything in that truck.

Things are a bit more locked down in 90's cars with fuel injection, as the injection system is a bit more specialised than a carb and they have an ECU. But even the ECU on 90's cars is fairly open.

Obviously there are a couple of fundamental flaws with older cars. The first is that older vehicles are a lot less safe than modern cars, the second being that they are a lot less reliable because they have usually done a lot of miles. They're also less fuel efficient.

I don't want or need an entertainment system in my car, all I need is a aux in, and I'm happy. I don't need my car to be connected to the internet, I don't need GPS, or the weather forecast, or any other dumb tech gimmick. I have a phone for all of that, and when my phone gets old, I can replace it. I can't replace the entertainment system in a modern car.

Luckily, there are still a lot of lower end cars that don't have all these "features". But what I would like is a higher end car, but without all the gimmicks, this is especially a problem with 4wd vehicles, they have too many electronic things in them. You can completely flood an old diesel Toyota Landcruiser or Hilux, and as long as your air intake is above the water, it will keep driving. You can't do that in a modern one.

I own a Toyota Celica Supra from 1985. I love the car to pieces - not only is it amazing to drive (absolutely no electronic interference between the driver and wheels), it is maintainable. I can access everything in the engine bay, the radio is an aftermarket JVC which Just. Works, I have aftermarket parking sensors fitted, and absolutely no internet connection.

The great thing about early fuel-injection vehicles is that they're phenomenally reliable. I have never heard of an ECU failing. The simple 8-bit microcontroller powering the Supra will just keep going and going and going. The brakes are purely hydraulic (no antilock, which okay, I accept is a bit of a problem when you drive cars that do have ABS). From the cylinder block to the wheels, it's purely mechanical transmission (stick-shift, clutch limited-slip diff). If something goes wrong, it's pretty obvious where the problem is, rather than chasing error codes that turn out to be a busted sensor buried deep in the engine and the thing it's monitoring is working fine. It has a beautiful, fancy electronic dashboard with VFDs for speed, fuel and temp, and LEDs for the rev counter, and it all works.

I wouldn't say the car is necessarily less safe than a modern vehicle, provided you respect its limitations. I am an alert and responsive driver, and I know how hard I can hit the brakes before the wheels lock up. I keep the tyres inflated, the brake fluid fresh and all the electrical components in working order. There's even a reassuring scenario that I hope never to experience - a driver of one of these cars once fell asleep on cruise control and crashed into a concrete barrier at 70MPH. The car was obliterated. The driver walked away, every bone in his hand broken, but considering the nature of the crash, it was incredible to think a 30-year-old car could protect the driver. The long engine bay acts as a huge crumple zone. The space-economy nature of modern cars seems to require a lot of creativity to allow for crumple zones to be squashed around the engine.

The reliability is also very subjective. Japanese engines and transmissions will long outlive the car they're fitted to. 300,000 miles and beyond is not unreachable. Bodywork suffers more, I have had to have lots of rust repaired on this car, and I'm struggling to source a few parts, but other than that, this 33-year-old car is fully working and starts first turn of the key. And on a long highway journey, I can get a good 500 miles out of the 60-litre fuel tank.

My family owned a brand new Land Rover Freelander, which was a pathetic off-roader - as you mention, once water gets into the electronics, it's game over. This happened once in a major flood and the car was never the same again. The car stalled, but eventually restarted, but never developed the same power. We eventually sold it for a Hilux Surf, much older, but the truck was considerably more rugged.

I have a real dislike for cars I'm not in full control of - my winter car is a Subaru Outback, and that has some annoying electronics. I'm constantly arguing with the automatic climate controls, which do what they like, and the electronic auto-box is sluggish and unresponsive in the extreme. It's a tough car, and a capable off-roader as I found out, but constantly second-guessing the driver is something I can't stand. I'm just thankful the car doesn't have OTA updates - I can imagine Subaru pushing new 'profiles' to the A/C and completely breaking it!

....and this is why I don't ever want to own any car with a cellular radio and reflashable firmware. Why on earth would I want an automaker to have any control over my car after I've bought it, just because they happen to have manufactured it? No way. When the day comes that my only reasonable options are all vehicles manufactured after this noxious trend became universal, "how easy is it to disable the modem" will be one of my search criteria.

Hopefully we'll still have that option. With the right-to-repair lawsuits ongoing a loss might mean disabling the modem is against the EULA.

For people in the EU a EULA isn't legal anyway - unless you need to explicitly agree with it before you can buy the car.

There is every chance that the car will say “warning, the firmware is 11 months old and must be updated or the car will not be usable in 30days”

With more and more features being “intelligent” (from driver aid to autonomous features) and the software being more and more complex, cars will ship with huge bug backlogs and critical issues will be discovered along the way (as well as changes to traffic laws, street signs and other things that must be patched in for the car to actually function).

Manufacturers can’t risk having drivers drive around in old versions of the car. It hurts their brand. This (together with a few other other reasons) is why I think car “ownership” will soon be a thing of the past. Already we hear news of models that will never actually be possible to own.

Yes, you are describing the nightmare scenario I want to avoid.

This is a bigger problem than the article indicates because it disables the rear-view camera.

The feds consider rear-view cameras essential safety equipment now, which is why they're required on every car made since 2015.

(Which is what makes the Honda ads so funny when they tout rear-view cameras as some great novelty they're giving you for free, when they're required to have them by law.)

May 2018 is the actual requirement date.

https://www.autotrader.com/car-news/new-backup-camera-rule-c...

But yeah, I'm with you on decoupling the camera system from the IVI. It's going to have to be treated more like a transmission or engine control computer.

This one confuses me. What's so safe about rear cameras? I guess if a little kid (or other very short person) is standing behind your car when you're reversing, but that's a failure of common-sense safety measures already.

Little kids don't observe common-sense safety measures, and can dart out behind a car before a parent even realizes what's happening.

Also the cameras usually have fairly wide-angle lenses. In situations like backing out of a spot in a parking lot, the camera often can spot vehicles or pedestrians that you otherwise would not see if you were just looking out your rear window.

Wider field of view with fewer blind spots (because it's at the back of the car, not trying to look past the structure of the car from the front).

I'd also wonder if people are more likely to do a thorough check if its easier.

Newer cars have very poor visibility to the rear and sides of the car due to rollover safety requirements, the rear view camera gives you back most of the visibility you had before the new safety requirements were enacted.

I would seriously consider getting one of their cars (read: Hellcat!), but only if I could remove Uconnect and the associated computers/'entertainment' system altogether, or at least disconnect/disable the antennas, because there's no way I'm trusting my life to (random car manufacturer)'s idea of security. Not even Tesla.

To those who say this is a conspiracy theory: what happens when my newfangled car decides to kill me, or just takes over by accident? Unfortunately too many insane and unthinkable conspiracy theories have been proven true in the last few years. Can you imagine if every Intel processor had a flaw that allowed it to leak memory on demand, from in the browser? To wit: Chrysler/Jeep Cherokee, Toyota Prius, Audi, BMW, etc. People actually died due to sudden Prius acceleration and Toyota wouldn't provide any significant information, even under subpoena.

So, for me, I kinda like the idea of driving 2006 or older vehicles. Why I have to drive around my mobile phone/tracking device is beyond me anyway. (That's why I carry one.)

Most manufacturers are installing firewalls in 2018+ vehicles, so even if someone hacked the headunit, even in drive by wire systems it shouldn't be able to control your accelerator, transmission, parking brake, steering, etc. With the past few years of vehicle hacking and the automakers being embarrassed, they have started to take Bus security extremely seriously.

In my opinion, with full display gauge clusters becoming popular, the entertainment and vehicle operation busses should be completely separate (including unique J1962 interfaces for each). Any software driven driver options should be controlled via the gauge cluster display instead of the entertainment system. That way, it's nearly impossible for a head unit hack to affect vehicle safety and control ECUs.

> Most manufacturers are installing firewalls in 2018+ vehicles, ... With the past few years of vehicle hacking and the automakers being embarrassed, they have started to take Bus security extremely seriously.

Unless their software source code is available for public review, I absolutely do not trust that this 'firewall' will do its job 100% of the time.

I agree, which is why I think they should be on completely separate busses. That said, they at least seem to be taking security seriously... finally.

As a counter point: What if your old 2006 car decides to die while you're going 60+ on the highway because it's old, and you die because it doesn't have as many safety features as newer cars do?

I mean, I get where you're coming from, I share some of the same worries you do, but the reality is that there is already hard evidence that newer cars are tons safer then older cars. It's really not clear to me that you can conclude that the chance of your new car deciding to kill you because of a software update/issue is a higher risk then the one you're taking by driving an old car. All of the examples you've given are more or less one-off events with newer cars, and they are serious, but people die every day in crashes with old cars that they could have survived or could have been prevented completely with a newer car with better safety features.

>What if your old 2006 car decides to die while you're going 60+ on the highway because it's old, and you die because it doesn't have as many safety features as newer cars do?

Is there some middle ground? My 2007 Toyota Tundra drives fine, and has few software components (certainly nothing that is updating OTA). Is the 2018 really that much safer than 2007? Even the body style was only updated a couple of years ago.

> Is the 2018 really that much safer than 2007?

Yes. The raw stats show that newer cars half around a 50% reduction in fatal accidents. You can find the stats here[0], but the relevant information is below (Sorry the formatting isn't great. You can find this exact chart from the linked page, the important point is that the numbers are about half of what they were):

    Occupant deaths per million registered passenger vehicles 1-3 years old, 1978-2016
    Year	Drivers 	All occupants
    Cars 	Pickups 	SUVs 	All passenger vehicles 	Cars 	Pickups 	SUVs 	All passenger vehicles

    2006	77	101	49	73	112	137	74	106
    2007	70	95	44	67	104	128	69	99
    2008	65	87	35	61	92	114	52	85
    2009	57	63	25	49	82	83	36	69
    2010	49	64	20	43	71	80	29	61
    2011	43	49	17	37	62	66	24	52
    2012	42	44	16	35	61	56	23	49
    2013	41	39	19	34	58	52	26	48
    2014	37	38	18	32	53	47	24	44
    2015	42	40	20	35	62	52	29	51
    2016	43	39	21	35	63	48	32	51

A lot of things are simple, like standard back-up cameras. Newer cars also have stronger frames and more air bags (Especially side airbags, which your car almost definitely lacks).

One feature in particular, ESC[1][2], is probably not on your car but is mandatory for all cars 2012 and later, and IIHS estimated in 2006 that if the feature was standard on every car it would prevent as much as 1/3 of all fatal crashes each year (And of course, the above charts don't disagree with this estimate).

[0] http://www.iihs.org/iihs/topics/t/general-statistics/fatalit...

[1] http://www.iihs.org/iihs/news/desktopnews/electronic-stabili...

[2] https://en.wikipedia.org/wiki/Electronic_stability_control#E...

"50% reduction" sounds impressive, but it starts to lose meaning when cars are already so safe. These are pretty low numbers, out of a million. My annual survival chances go from 99.993% to 99.997%, which is great on a population scale but essentially meaningless on a personal one.

For age groups 25-64, car accidents aren't even the most deadly type of accidental injury: accidental poisoning has twice the fatality rate [1]. If this were a software system and I proposed buying a newer car, my manager would be annoyed at me for micro-optimizing the wrong thing.

[1]: https://www.cdc.gov/injury/images/lc-charts/leading_causes_o...

I haven't entered a dealership since 2010 and part of me was holding onto hope that most of these "upgrades" were optional. This thread is confirming my fear that I'll never be able to find a new car worth buying again no matter how much money I bring to the table.

I bet they didn't realize that deploying software updates to an embedded computer/OS like this effectively makes them an operating system vendor, and that they now need the same operations expertise as Microsoft, Apple, Google, etc. I hope they take lessons from the software industry and don't spend years relearning update best-practices.

The software industry is still relearning update best-practises, just look at Android fragmentation and how a mere 1.1% of users are running the latest version [1], or the entire Internet of Things [2]. If Google cannot manage to keep Android users relatively up to date (And yes, I am blaming Google), what are the odds that a random automaker will fare better.

---

[1] https://developer.android.com/about/dashboards

[2] https://twitter.com/internetofshit

I wonder it they will be able to fix this OTA, or will require a trip to the dealer. How can you perform a firmware update on a system rebooting every 45 seconds?

Only other possibility is a download to a USB stick. Not sure if they have this option though.

I bought a new Accord two months ago. Where the tachometer is is actually a display that can be cycled; it started flaking out immediately when I did anything with the audio. Took a week to put new parts in.

Now CarPlay stopped working. I fear another week to repair/replace.

Not quite as serious as UConnect issue, but I totally feel people when they say they'd rather have less technology.

My Fiat 500e's Remote/Status iPhone application hasn't worked in months, through at least 2 iOS updates. The app starts up, then crashes immediately. Are the Fiat Chrysler folks not paying enough to retain good devs?

Anyone else excited for these systems to start driving cars?! It's gonna be faaaaaan-tastic.

If you were to choose a car solely because of its infotainment system, what would you go with? Thought about that as was struggling with a conference call from my car this morning. Being able to do stuff like listen to podcasts, navigating via Google Maps, etc, as easily as possible might actually be my top car buying criterion.

Get a car with CarPlay or Android Auto

Except for the obvious instant issue, the UConnect infotainment system is actually pretty nice, IMO.

And it can come attached to a Dodge Charger or Challenger, both of which have a kind of primitive, hypnotic appeal to me.

BMW, honestly

To elaborate, they understand actually having physical buttons you can macro to functions, and doing things like starting your first new playlist with a pre-mapped button. It's better in that tactile sense than carplay or anything because you don't have to futz with a screen once it's set up, and has a good interface for the steering wheel controls

Car companies seem exceptionally bad at both software and hardware. Not only does the tech lag years behind the industry, it's horrendously designed. Like, your media network should be airgapped from your critical systems, but in practice they are not. This allows people to make your car slam on its breaks using nothing more than radio waves.

Url changed from https://www.theverge.com/2018/2/13/17007332/fiat-chrysler-uc..., which points to this.

Thanks.

These are the companies that we trust to build us "safer than human drivers" self-driving cars, yes?

I can tell you for sure that the way they make software for engine, transmission, safety break, lane assist etc. is very different than the way they make software for anything not safety related. The ultimate luxury they provide is time.

This likely qualifies under “lemon laws” to permit new car owners the ability to rescind their purchase; I’m surprised no one has done so yet.

EDIT: I don't think it's something worth doing for its own sake, but someone pursuing such an approach might be able to compel a more coherent reply from Fiat-Chrysler than a reporter alone.

How?

Lemon Laws require a persistent unreasonable issue to invoke.

3 days of "entire nav and sound system now completely inoperable" out of, say, 30 days of lemon law window, would theoretically qualify as both persistent and unreasonable in a new car — if the selling dealership says they are unable to resolve the issue, and they are unable to provide a timeline for resolution of the issue, when asked.

This would be less arguable if the manufacturer simply shipped the same nav+audio in all cars worldwide, but since they usually reserve this sort of thing for "fancy expensive upgrade options", selling a fancy option that goes dead within a week and stays dead without a timeline for repair is quite in-scope for Lemon concerns.

This wouldn't qualify under the California Lemon Law - it may very well take a dealer more than three days to diagnose the issue, for example, in this case, the diagnosis is occurring at the manufacturer, and a fix being worked out.

It would be another thing entirely, if the issue kept occurring, repeatedly over months, and the manufacturer was unwilling or unable to repair it.

My parents went through a lemon law case on a Corvette related to the electronic system and alarm. The car would either set off alarm or electronic system wouldn't start the car. It took them 5 tows, hours of phone calls and a few trips to the dealer before GM was willing to accept.

Lemon law exists but its not as simple as my car rebooted last few days, take it back.

As of May 2018 all new cars sold in the USA are required to have video backup cameras.

If a bricked IVI unit causes that camera to not be available, I wonder if that's enough to qualify as return issue.

Reply on news.ycombinator.com