Automotive Grade Linux | Hacker News Reader

Automotive Grade Linux

automotivelinux.org

230 pointsknown204 comments

I like Linux and used it for over 22 years now, I did some kernel development. I worked for Samsung on TizenOS and I had some discussion with Intel on IVI (their offshoot for automotive). I then went to work for Intel. In my opinion one thing Linux is never going to be is "automotive grade".

There is conflicting set of requirements when developing Linux and in the end it is somewhere on the spectrum of compromise between performance, security, stability, features and development cost.

And I think this is right, Linux is real software for real world and it would be wrong and wasteful for general purpose OS to prioritize one requirement to the exclusion of others.

Unfortunately, an automotive grade software absolutely requires focus on stability and you just can't get it if you also want performance, features and want to do it cheap.

I am also of the opinion that Linux is just way too complex to ever be stable enough for controlling your car. Think about Android. You get an OS tightly managed by Google, running on purpose-built hardware (I have Pixel 2 XL) and it still regularly crashes on me and does otherwise funky things. I have no illusions car industry is going to be able to do any better.

Now, I have nothing against using Linux for everything else in a car (infotainment, navigation, etc.) but I would not trust one that will have anything to do with basic controls.

This is basically what I've heard from automotive engineers when I've asked about "Android for your car".

Paraphrasing: "You can remote jailbreak Android - do we really want to trust every vehicle in the country to use it for their control systems?"

Except that many brands use Android to power their infotainments these days. People who you talked to obviously didn't quite know what they're talking about - it's like saying embedded Linux on a computer can be "remote hacked" because there's a GNOME userspace exploit somewhere in the world.

I think the person had a qualm about specifically using Android vs an embedded Linux or other embedded software for auto control systems, and the prevalence of Android remote hacks was their concern - they were happy about the state of Android auto and infotainment consoles.

I think they are knowledgeable on the subject but I may be misquoting them or not including some nuance about the security concern.

> Paraphrasing: "You can remote jailbreak Android - do we really want to trust every vehicle in the country to use it for their control systems?"

FSB, GRU, Mossad, MI6, NSA, CIA, GCHQ, DGSE, CSIS, ASD, CSE, BND, NIS, PSIA, GCSB, ISI, RAW, MOIS, MSS, MIT, GIP, CISEN, NIA, RGB, and all other intelligence agencies in the world: "Yes, that is a splendid direction."

On a serious note, with the automotive industry's historical track record with timely updating their car software and doing so for as long as a car is kept running (with the possible exception of Tesla), I'm not sure any OS will be sufficient. Without some legal definition of orphaned automotive software being something like X unfixed CERT vulnerabilities over Y years, and legally-mandated regulation that open sources the ability to load alternative software, that security drift will always be the Achilles heel of any gear that uses software, not just cars.

This is one of the reasons I've become a lot more selective with hardware gear. Any gear that lets me use open source software immediately goes to the top of the evaluation list above closed source vendor software, because I've been stranded way too many times by hardware with closed source software the vendor abandons for a newer model.

Can you not jailbreak automotive software?

> Paraphrasing: "You can remote jailbreak Android - do we really want to trust every vehicle in the country to use it for their control systems?"

We already trust every vehicle in the country to use jailbroken software in their control systems.

Mechanics regularly upload modified manufacturer and third party firmware.

Car enthusiasts modify and trade patched binaries for their cars' ECUs on forums like phone jailbreaking enthusiasts do on XDA-Developers.

The existence of remote attacks was perhaps the most important concern for them on using Android specifically, perhaps I should have phrased it as "remote hijacking/escalation".

Could you please explain why we need a full fat OS for a car anyway? I understand the importance of the ECU in modern cars, is there something that Linux can do in a car that the ECU can't?

You're confusing ECU and IVI, the ECU doesn't drive anything in a "digital cockpit" car, all the displays are handled by the IVI.

I have worked with Linux on tight embedded systems that require 100% up-time in safety critical environments.

I disagree, it's fine for automotive use, but it will be a stripped down fork optimized for it. Just like ucLinux and other special built distros.

With multicore processors becoming the norm, a combination of isocpus, some low latency kernel options, and some virtualization are all that's needed imo.

I get sub-microsecond response times on my servers which, afaik, is a few orders of magnitude better than what's needed for control systems.

Basically, on a modern ARM chipset, the requirements of a linux distro suited for cars aren't particularly exotic these days. All the logical bits are there, it just needs firmware/drivers.

Real time has nothing to do with response times or performance. It has to do with predictability and ability to reason about how much time an operation will take (most of the time it is enough to be able to have some kind of hard limit on how much time something will take).

This means that you need all your operations that you call from critical realtime code to not suddenly start doing some housekeeping.

On Linux, even as simple operation as dereferencing a pointer can cause context to be switched to kernel to suddenly start doing its memory management stuff (not really Linux fault, that's how x86 architecture is built).

So while a machine might be able to respond "orders of magnitude" better than needed for control systems it is the one response in X thousand or million that is not meeting the limit that is deal breaker here.

I have implemented an embedded controller for coin acceptor. If the driver does not respond within X milliseconds the coin passes the gate and there is one more angry customer.

I have implemented a framework for algorithmic trading for one of largest brokerage houses on Warsaw Stock Exchange. You have XX microseconds to respond to market on event and if you don't there could be financial consequences.

Both apps were implemented on Linux. In both cases I had to work my ass off to track down multitude of potential causes of unpredictability and I think I did pretty good job but I cannot give guarantee that it will always work within time limit. If there was somebody's life at stake I would refuse to do it.

Did you know that x86 CPUs stop from time to time to do their housekeeping, like checking temperature to decide whether to throttle or not? You can turn it off (at the cost of the server potentially overheating) but how many more potential sources of problems are there?

Yes, I am extremely familiar with high frequency trading, I've been doing it for about 9 years now on most of the highest volume exchanges in the world.

Satisfying hard real time requirements is hard, but not impossible. When testing, set a fault/trace point on long response times, debug, rinse, repeat.

The point is that Linux (or QNX or whatever other RTOS) have way more resources, libraries, and commoditization than a dedicated microcontroller/FPGA and are more widely tested than some proprietary OS.

And those Intel hardware faults you mention, aside from being irrelevant to the kind of architectures a car would use, are on the order of microseconds, well within the operating requirements of a car (or coin system).

What is coin in this context CoolGuySteve ? (HFT ?)

He meant algorithmic trading which is close (but not exactly the same) as high frequency trading. HFT is obsessed with latency. Algorithmic trading is mostly about automating wishes of a trader because if he can execute his strategy faster he gets a bit of an additional edge.

The coin I mentioned is completely different area. This was for a fixed odds betting terminal. The company ported their software from RTOS to Linux but they had issue with occasionally missing events generated by their custom hardware board controlling the device. This being betting terminal caused grief to customers and unhappy customers == no income.

I was contracted to solve this issue for them which I figured out would be to write a proper kernel driver for their board, restructuring of their existing code and to tune their Linux kernel/configuration.

The device they used as coin acceptor was very cheap (one figures for a device that returned its cost in couple days of operation...) and it had no controller, just analog output. The output would show the current value of something related to the induced current in the coin while the coin is falling through a piece of tubing inside an energized coil. The software has to probe the value in regular intervals while the coin is passing the sensor, and then classify as one of the possible coins or an unknown object based on duration, strength and profile of the signal. This relates to the dimensions and materials used to build the coin.

The precision at which you perform the measurements greatly influences precision of detection of various types of coins and it allows tighter tolerances which helps prevents fraud.

Once the coin passes the detector it has couple centimeters before it arrives at a gate which sorts the coins to their possible destinations.

The software has to periodically (multiple times a millisecond) monitor the value of the analog value, and then has couple milliseconds to decide what the coin is and send command to the gate.

There is not much to it unless you have some other interrupt completely disrupting the process and this is the part they could not figure out how to solve. The calculations are not complex and there is plenty of time to perform them.

I suggested they can easily implement it on their board and return buffer of all values to their Linux but they told me they have huge amount of these machines in the field and they don't want new design for their controller.

> With multicore processors becoming the norm, a combination of isocpus, some low latency kernel options, and some virtualization are all that's needed imo.

Have you worked in vehicle embedded systems or are you just pulling requirements (or lack thereof) out of your hat?

I'm not an expert, but I would guess that they do require deterministic behaviour, which, AFAIK, Linux doesn't offer. What happens if for some weird reason the scheduler starves the critical process for 40 ms in some uncommon situation that only happens 0.00001% of the time? How can you prove that such a thing won't ever happen?

His point is that there are ways to keep the OS from ever fooling with certain cores, after startup. At the extreme, you put an exo-, para-, iso-, whatsit-kernel on them, but just starting a process that does all its kernel activity at startup, then never does another system call after, is more common.

Usually that involves mapping device registers into the process address space via a custom ioctl, and then just polling in a tight loop.

That's not really enough. As an example ANY operation involving filesystem or memory access has no defined limit on how long it can take.

Your filesystem can decide to do some housekeeping while you do write and as simple thing as dereferencing a pointer can cause cause context switch to Linux kernel to do the TLB thing. You might want to allocate some memory and then Linux might decide to move a bunch of memory around before it gives the pointer back to you.

What I mean is, to be able to guarantee execution of a piece of code is not a simple task for the faint hearted. You need to forget IO. You need to forget switching context to Linux (basically forget any syscalls). You need to forget allocating any memory after program startup. You need to use really huge pages. Forget being green, because changing CPU frequency or sleeping can cause unexpected latency in the middle of some operation.

Also doing this inside VM is neigh impossible.

If you are not doing any system calls, it means you are perforce not doing file system operations.

Memory is normally pre-allocated and mlocked into place at startup. TLB misses can happen, but those are filled in hardware, just like regular cache misses, not by kernel activity (and often from cache). The "TLB shootdowns" you read about happen when memory is unmapped, so you don't. The kernel can block you if you write to memory mapping a file backed by a physical disk, so you don't.

And, yes, the core is locked in C0 state, drawing maximum power full time.

It is certainly possible to do I/O, by operating directly on the device registers that I noted were mapped into process memory. And, in fact, this is a routine method of conducting such operations. You can read about it by searching "kernel bypass".

There is usually no need or desire to do it in a VM, but kernel-bypass libraries are quite capable of mapping their registers into a VM's memory, so somebody must be doing it.

Commonly, input from network interfaces is dumped into one or more big-ass ring buffers mapped onto /dev/hugepages; and other, regular processes, running on other cores, map those read-only and pick over it for interesting bits, and maybe do file system operations of their own.

Honda has been shipping Android powered entertainment systems with direct access to the CAN bus (which may even include vehicle control, based on geohot's work) for several years now. Just as on phones, as long as this using being used for timing critical operations like an ECU and doesn't have any sort of remote network access it doesn't seem to be a major source of issues.

Direct access to an isolated CAN bus used only for vehicle infotainment

You missed his reference to comma.ai/autopilot

He could be right - I was operating under an assumption that it's the same CAN bus access they're using, it's entirely possible it's only an isolated segment.
I did not confirm this, purely an assumption on my part as it does appear to have access to HVAC, lock settings, as well as fuel information which makes me suspect it's not so isolated.

Linux is used for SpaceX' Falcon 9 and Crew Dragon. If it can be used successfully in space flights, why shouldn't it be used in cars?

To the OP's point about infotainment being okay, I wonder if the core functional components of space flight (RCS, etc), are controlled through linux. It would surprise me if it wasn't used for high level things (displays), but not the "keep the rocket flying on course" stuff. I can't really find anything about what's controlled by that Linux box in the Falcon 9.

It is being used for high level things, so is electron.

Manual flight controls are ran on Linux via electron.

Yes they are. iirc, they run a modified kernel that is closer to realtime, and everything is a single thread for determinism. Code runs on a dual core processor, with one copy on each core, to detect bitflips. Three such modules are connected together in most places.
I believe this is the setup they use for the main stabilization computer on each stage, as well as the engine controllers for each engine.

I only saw it used on the recent crew mission, but wasn't Linux basically used as an infotainment system there as well? The presenter even explicitly mentioned there was a backup system in case the fancy user interface (which was HTML+JS based) crapped out on them.

The space is a much more controlled environment and has low uptime compared to millions of cars every day

There is a huge difference here. Cars clock billions of hours of operation in an uncontrolled environment. These must deal with decades of use, abuse, neglect, sabotage, third party parts, etc. Anything that can happen WILL happen in a car.
Compared to this, rockets are tightly controlled environment and running for a very short time. There is also incredible amount of redundancy and supervision in a rocket but you will not be able to have double of every critical component in a car and then have separate controller to check if both are returning matching results and which might be faulty and you will not have a team of specialists obsessing every time something did not work exactly right.

Why wouldn't it be possible to have a similar system in a car as in a rocket? The manufacturer could easily rack up 3 somewhat cheap compute nodes to validate the outputs. This really shouldn't be much more than $10k extra - for the end consumer. And the fact that a lot of cars sell for $100k or more this cost should be negligible for self-driving capabilities.

I can't decide if you're serious or trolling ...

Every production car is built down to a price, even expensive luxury cars with large margins. All car segments are extremely competitive and any efficiency you can gain over your competitors can be put into more features, higher quality, lower price, or more profit. Over-engineering isn't that much of a marketing point or Saab would still be around.
That said, mass produced electronics are cheap - even automotive grade ones. It wouldn't add that much to the BOM, it would be mostly design cost. I just can't see the justification without it being required by regulation.

I'd love for somebody with knowledge to comment re: the SpaceX flight control platform. There's Linux in the UI to be sure, but I'd strongly suspect there's a hard-realtime OS that's doing the hardware-level communication to the flight hardware.

There was a recent reddit AMA with some SpaceX engineers that confirmed that they run a buildroot system with RT patches for most things, and a handful of microcontrollers for the timing critical stuff.

Because rockets have tight tolerance requirements, but very low reliability requirements. Consider this case [1] where missile software leaked memory because it would blow up before it ran out of memory. It would hardly be reasonable to state: "If missiles can leak memory, then why shouldn't that be okay for a car." even though missiles are probably more advanced, complicated, and more difficult to make than a car. This is because the "features" are largely orthogonal from the "reliability" of the features which is really a measure of how consistently the features as promised are delivered.
In the rocket case, you need very tight tolerances, but it does not matter much if it fails 1 in 1000 times since the total-system-failure rate is on the order of 1 in 100. A single 1 in 1000 failure mode does not massively increase total-system-failure rate. However, in the car case the tolerances are worse, but the requirement to consistently deliver on those tolerances is vastly more important. Consider Ford Pinto which sold 3,000,000 cars and had 27 deaths attributed to the fuel tank design defect. A 1 in 100,000 failure mode is viewed as an unconscionable deathtrap, so adding a single 1 in 1000 failure mode would make your system 100x deadlier than the most egregious example of an unsafe system.
The prevailing rate of system errors tells you what sort of errors a quality assurance process can detect. If you have 1 in 1000 system errors, then it means you can probably detect things on that order otherwise your system error rate would be dominated by those higher frequency errors. However, you probably can not or do not bother to detect 1 in 1,000,000,000 errors since they would either happen so infrequently compared to other errors or not be worth solving since they have such a minor impact on overall reliability.
So, in summary, reliability is different from tolerances, techniques used on low reliability systems are not necessarily applicable to high reliability systems, rockets are on the order of 10,000-1,000,000 less reliable than cars, so technology used on rockets does not necessarily apply to high reliability industries.
[1] https://groups.google.com/forum/message/raw?msg=comp.lang.ad...

> Now, I have nothing against using Linux for everything else in a car (infotainment, navigation, etc.)
Which is what AGL is primarily about. There is not a single set of requirements for something to be "automotive grade", neither in hardware nor in software.

Infotainment and navigation don't require "automotive grade" label. "Automotive grade" is a special label for things that meet additional stringent requirements to be able to be used in harsh environment of the car. You can run navigation on an Android phone and it will be perfectly happy without "automotive grade" label.
Go to any electronic parts manufacturer and you will find separate sets of parts, ones for normal use and then ones for "automotive", that have improved resiliency to temperature, vibration, etc.
For software, this tends to be used to describe components meeting special additional requirements with regards to stability. You wouldn't want your ABS controller to decide to break at most inopportune moment. There are special sets of standards for coding that kind of software (see MISRA) that are not required but highly suggested.
For software that has no influence on safety, the tag "automotive grade" would have no meaning.

Eh, automotive requirements are more than just for safety-critical pieces - reliability matters for non-safety functions too, and all automotive Linuxes I've seen had similar pieces build in that direction - which from my understanding AGL attempts to standardize.
Then there's things that are less critical than the hardware keeping the car on the road, but more important than the radio, and Linux is creeping in there too. Be it parts of the instrument cluster, or non-required warning systems, or visualizations of surrounding traffic, or ...
Just because the requirements placed on these components are not as strict than e.g. for engine controllers they still are additional requirements specific to the field, so I don't think the label is totally misused. Although I admit that the confusion it apparently causes every time it is mentioned points to it being a bad name nevertheless, and "automotive linux platform" would be appropriately buzzword-compliant too.

> Think about Android. You get an OS tightly managed by Google, running on purpose-built hardware (I have Pixel 2 XL) and it still regularly crashes on me and does otherwise funky things. I have no illusions car industry is going to be able to do any better.
Android regularly crashes in part because it's very complex, but also in part because there's really not much of a problem if it crashes occasionally. Cars have much stricter requirements.
The car industry have more or less settled on a process, "functional safety certification" (FuSa for short); if you can verify that you've followed the process, then you have a reasonable defense in court if the software then crashes.
> Now, I have nothing against using Linux for everything else in a car (infotainment, navigation, etc.) but I would not trust one that will have anything to do with basic controls.
Indeed; you run Linux inside a safety-certified hypervisor, and run your controls in specialized, safety-certified RTOS guests.
ARM has settled on Xen as the hypervisor for their automotive reference stack, and there are currently efforts to achieve safety certification for Xen:
https://www.youtube.com/watch?v=boh4nqPAk50

For payment terminal, pinpad security there is concept of Hardware Security Module. This is separated processing unit in the terminal that is managed by the main application in a very limited way, so even if the security of the main, feature rich app is broken the cryptographic material and processing logic stays safe.
Another solution used in a terminal is that, technically, it is required that the pinpad terminal (keyboard + screen) must be controlled by that separate computer. This is achieved by galvanically switching keyboard and screen to perform PIN entry and then switching ownership back to the main application which then can fetch encrypted PIN block and perform rest of the transaction, never having access to either plaintext cryptographic material or PIN.
Now, this could be solutions used for the car security but it would still require separation so strong and applications so different that they could just as well be different OS-es with no commonality.

> must be controlled by that separate computer
Right, the "simple straightforward" solution for all of this is to have a separate SoC for each control unit you need.
But this has lots of things that are suboptimal:
1. Less flexibility. You need to know ahead of time exactly how many SoCs you need. If at the 11th hour you decide you need another unit (or need one fewer), you have to go back and redesign your entire electronics layout.
2. Less efficiency. If your unit typically only uses a small portion of its compute power, it's still taking up space, taking up weight, and drawing a basic amount of "idle" power.
With virtualization, your RT guest OSes typically still want their own dedicated core. But it's a lot more efficient to put a 6-core chip on a system that only needs 4 cores than to design 6 separate SoCs onto a system that in the end only needs 4 of them.

"Android regularly crashes in part because it's very complex, but also in part because there's really not much of a problem if it crashes occasionally. Cars have much stricter requirements."
I don't remember a single Android crash or freezes in recent memory (various Samsung Galaxy S-series). Not that my iPhone crashes either.

I can't remember it freezing so hard it required a restart, but I've had iPadOS 14 do its "respring" black screen with a spinner crash in the last couple of days.

That is based on unix.

Parent comment noted "Not that my iPhone crashes either," so I figured it was worth mentioning that sometimes they do

iPadOS 14 is still in beta. Crashes on beta aren't representative of released product.

Whoops, meant 13 for my iPad. Mixed it up with a spare 6S where I have the iPhone beta.
Haven't seen the beta crash but I'd attribute that to not using it on either of my main iThings.

Automotive Grade Linux is pretty much a Linux distribution for infotainment units. It's not about running Linux on ECUs.

Seriously, what does your phone running 100s of apps like TikTok, Browers, Facebook and everything else have to do with a closed system like car infotainments?
It's like saying that Linux will be crashing while driving SCADA systems because you've seen your KDE desktop crash a few times.
C'mon you guys are supposed to be engineers, think a bit -_-

An OS that crashes when pushed moderately hard does not bode confidence for something that must have very near to 100% uptime.

Usually it's not the kernel that crashes, but the desktop environment or some other user-space software.
I can't remember the last time I encountered a genuine kernel panic on my desktop. I don't imagine that these "Linux powered cars" will be running KDE Plasma, Gnome, or anything else remotely as complex as a modern DE.

True. Android isn't such an OS though. As proven by the fact that so many vendors use it to power their hardware - including existing car manufacturers and other hardware builders.

Automotive grade Linux is not controlling the car.
AGL targets infotainment systems and other aspects. It can have access to more important data but it's not controlling the car.

There was a bunch of attempts at this and I even have passing experience working with Intel guys on their IVI project which was integrated into AGL (https://wiki.tizen.org/IVI). I don't know what makes the people think this will work this time when it did not previously. I just know that it previously was restricted to infotainment even in its name and mission statement and now that does not seem to be true any longer.
"Automotive Grade Linux is a collaborative open source project that is bringing together automakers, suppliers and technology companies to accelerate the development and adoption of a fully open software stack for the connected car. With Linux at its core, AGL is developing an open platform from the ground up that can serve as the de facto industry standard to enable rapid development of new features and technologies."
See, nowhere it mentions this is for infotainment only.

At a high level that seems true for now, but the site does say this:
"Although initially focused on In-Vehicle-Infotainment (IVI), AGL is the only organization planning to address all software in the vehicle, including instrument cluster, heads up display, telematics, advanced driver assistance systems (ADAS) and autonomous driving." [1]
That's getting pretty damn far into controlling the car.
[1] https://www.automotivelinux.org/subaru-outback/

If they add everything other than the last part I think its fine. I think something used for self driving should be on its own separate hardware unrelated to any other hardware used for any other purpose within a car in order to fully isolate itself at the hardware level so your infotaiment system doesnt crash the whole system rendering your car undrivable for a moment.

Telematics can be dangerous also. Things downstream might be using that data for things they shouldn't...safety systems, etc.
Also, couldn't ADAS be an issue? Sending the driver lagged lane encroachment info, for example.

> Also, couldn't ADAS be an issue? Sending the driver lagged lane encroachment info, for example.
"Lagged by how much?" is the qualifier that determines if that is a problem. 100ms wouldn't be a problem, 200 could cause issues but likely wouldn't result in crashes.

That's orthogonal to the OS, no? You can (and should) have separate hardware for separate functions while running AGL on the lot.

ADAS can break, accelerate and steer the car.

Of those only ADAS and autonomous driving sound particularly worrisome to me. Sure, a frozen instrument cluster would be inconvenient, but I'm more worried about the microcontrollers that control my breaks.

Aren't all self-driving systems complex enough to require a full OS like linux? And those systems are very much "controlling the car".
Of course it makes sense to have a special-purpose microcontroller that handles real-time tasks related to engine and brake control. But those devices lack mmus or even caches so running linux on them is not terribly interesting.

sort of. like linux is tricky because linux implies a set of compromise and automouse requires a different set

What specifically does linux compromise that prevents its use in automotive control? Obviously specific components should be implemented using PLCs, FPGAs, or baremetal/FreeRTOS based microcontrollers but for an overall control system for an automotive vehicle, nothing specific stands out to me as preventing it from functioning properly.
Please feel free to point out all the many ways I am wrong as I imagine I am missing something here.

All the things that you would choose a rtos or bare metal for.

>Now, I have nothing against using Linux for everything else in a car (infotainment, navigation, etc.) but I would not trust one that will have anything to do with basic controls.
Comma AI has been shipping self driving software based on Android for 4 years now. It's rock solid and completely safe, with millions of miles driven [0]. The idea of everything in automotive requiring formally verified aerospace grade software is just nuts. It's better to have flexible, adaptable software whose failure modes are fully covered than to rely on an illusion of perfection that will never actually exist.
[0] https://comma.ai/

When you're talking about AI then your software suite looks very different. The actual running of the vehicles engine and subsystems is something that is very well known, and can be handled by embedded software/hardware quite easily. It's not that I don't think more complicated stacks can handle it, it's that I don't think they're needed. That said, no matter which way you skin it, if you increase complexity then you increase risk.

>That said, no matter which way you skin it, if you increase complexity then you increase risk.
This isn't really true, for eg. if you add redundancy technically you increase the complexity but also reduce risk (point being complexity will also be introduced when making the system safer, simplification just makes analysing and screening for bugs more effective, but there are other risk factors)

> It's rock solid and completely safe
It sounds... Unbreakable!

> It's rock solid and completely safe
What is your source on this? I don't think even GeoHot says that.

25 million miles is completely inadequate testing to make any such statement. The prevailing traffic fatality rate in the US is 1 in 125 million miles. 25 million miles means that the information is extremely insufficient to make any statement about comparative safety. In fact, making any such statement would be mathematical malpractice and criminally irresponsible. This also ignores the fact that unless those miles were all driven without any human correction you are actually testing the safety of a combined human+software solution which had better be better than just a human alone otherwise the system actually reduces safety.

>you are actually testing the safety of a combined human+software solution which had better be better than just a human alone otherwise the system actually reduces safety.
That's the entire point. Comma AI is a level 2 system. It is not intended to, nor will it ever be fully automated. It is a tool to augment and enhance human driving capabilities. Full autonomy at Level 3 and above is a 20+ year pipe dream at this point. Anyone advertising such a system (Tesla...) is just fraudulent.

I looked in to a bunch of options when deciding on a new ECU for my car, ultimately the idea of leaving the reliability of my vehicle up to a more complex stack of software, with dependencies, updates and bugs within the layers beneath the ECU software itself, didn't feel like the right choice.
I also really love that aftermarket ECU manufacturers have been able to find a market, and I wanted to support that. Almost all of the ECU companies I've been in contact with were started by one or two engineers as a commercial project for enthusiasts. You just don't see bread and butter circuitry and embedded software companies solving real problems like that getting a fair go in the market these days. It's a niche where talent makes your reputation, and it's not a space big enough to get "disrupted" by the Venture Capital bubble.

As someone who develops mostly bare metal embedded systems, not everything that is simple is also golden and stable, especially compared to the Linux kernel.
Honestly I have never heard of a case where a OS too complex to be the cause of catastrophe. Embedded systems on the other hand...
I think the maturity of kernels of popular OS are much more battle tested than a standalone system can ever be at this point. Sure, maybe don't give essential controls to any complex (technical) system without the driver being able to correct. Just look at Boeing and MCAS, probably not based on Linux.
I think the automotive industry currently has the problem that it too often relied on proprietary systems and buses. An open system could handle the diversity of suppliers very common for most manufacturers. It can supply a common base to develop against. I don't think it is fair to compare the stability of Linux to Android, they are worlds apart.
Teslas OS is based on Linux, so we have a case were it does indeed work at least for supplemental functions. And I fear alternative choices could be the death of other manufacturers because their software would fall behind.

> Just look at Boeing and MCAS, probably not based on Linux.
This just reinforces your general point, but it wouldn't matter if the 737 ran on Linux. The MCAS behavior is part of the "business logic". No underlying OS would have prevented that issue. Just like the safety of the Linux network stack doesn't prevent grandma from E-Mailing her CC details to a scammer.

Boeing’s management has done a good job recasting what were deliberate business decisions that drastically reduced safety into bugs and technical errors, at least in the public eye.
It doesn’t seem the FAA is being too strict on them in this either, but I suspect the Europeans and Canadians will not go as easy on Boeing as the FAA has.

I have seen software for a unit in a car which had no safety relevant tasks, which was written using a custom, homegrown rtos with half of a standard library reimplemented from scratch for no reason at all.
Their supplier did all of this on purpose, to create lock-in, which worked - now they charge by line of code.
Any old Linux kernel haphazardly patched together to run on the target would be better than this garbage.
Do airplanes still run an emulator for a chip that is no longer produced because nobody wants to port the software? That was a fun story I still hope is an urban myth.

Not an airplane, but I have a few HP calculators (50G, 12C, 15C) that run an emulator for a chip that is no longer produced because nobody wants to port the software. They still make the 12C.
I wouldn't trust them to port it without introducing weird bugs after seeing stuff like the 35S anyway.

A lot if not the majority or PoS terminals around here run som DOS software, usually on top of an emulator. Sometimes you can even see a DOSBox window on top of a Windows desktop.

Just a few months ago I caught a glimpse of DOS in a multi-national home improvement store. Turns out all of their computers are fairly recent ThinkCentre machines, all running Windows 7 that is locked down to 3 programs: calc.exe, Internet Explorer 10 and some proprietary DOS emulator they pay something like 50k€ annually for, running a TUI client for their inventory and order system.
I don't even know how to sort those by level of stupidity...

If we are talking credit card terminals then this will depend a lot on implementation. You can have whatever you want on a terminal as long as it meets PCI DSS/PED requirements which mostly are concerned with safety and integrity of the transaction. Safety means your PIN and private issuer data (data your bank put on your card) stays safe and integrity means nobody can meddle with the data or the flow of the transaction (so the amount you have approved on the display is the amount of transaction in your bank is the amount printed on your printout, etc.)

The only thing I'd argue is that from what I've seen of car software, it is generally horrible and a battle-tested kernel, even if seriously stripped down, would be a marked improvement.
I think a lot of people working with these closed systems try to perhaps impress those not in the field by cloaking it in mystery, but a lot of these systems are honestly dumpster fires. Not left-pad level, sure, but nothing at the level where Linux wouldn't do.

From what I've seen Tesla doesn't use Linux for the critical parts (i.e. everything directly to the CAN bus). The gateway between the vehicle part and the rest is using FreeRTOS.
The rest is indeed Linux. I think it is a customized Ubuntu. The "rest" includes the autopilot...

You & upstream ("Imilcim") are both right, but both your examples are flawed (Android is not really Linux, MCAS is a logical design flaw).
Linux can be very stable (most of the time it is).
I think that in this context "Quality Assurance" (QA) is king, independently from what it tests.
Linux has a lot of stuff included in the kernel, and that stuff is actively used by a lot of people in a lot of different (potentially unexpected) ways => if QA can certify both custom-written-SW and Linux (for the needed usage) I would then tend to trust Linux more than a SW written by dedicated teams in companies (because of the possibility that QA could miss some tests which might have been indirectly caught by the Linux community/users).

Safety-critical systems have specific processes that need to be followed and it's definitely not as easy as using a simple OS, but it does help of course to keep things simple.
And that's the problem with Linux: it's huge and it's always changing so that one cannot make any guarantees about anything except that it's broken in known and unknown ways. Saying that it's battle tested is not as useful as one would think, since any small kernel config change would result in a different configuration with different behavior. Abstract Linux is battle tested, but the config you want to use is probably not that well tested and even if it is, the whole development process is so chaotic that there's no guarantee that somebody of sufficient skill will report even the majority of encountered bugs and that those will be fixed.

I would have agreed with you at one point, but most of the automotive companies are diving into this. They want less computers and would rather invest in hardware partitioning if needed.
BWM has said point blank all of the software in the car will be running Linux.
Adopting Linux on BMW - The Long Road to Integrate Linux as Mainline Platform - Helio Chissini de Castro, BMW https://www.youtube.com/watch?v=gNr-XmWM-eM

"and it still regularly crashes on me". Your experience differs vastly from mine.
Not to say that Android is perfect nor bullet-proof nor never-ever crashes buuuut.... It tends to be an App that crashes the OS rather than the uderlaying OS just doing a BSOD.

Most people can't tell the difference between app and OS crashes. With all the misinformation out there about it, who can blame them.

This top-level post is mostly incorrect. A linux distribution is incredibly flexible and stable if optimized for such. It is a simple task to build in exactly what is needed from the ground up from stable "LTS" sources, and nothing else. To configure for near-realtime responsiveness rather than throughput is trivial as well. That's why it scales from embedded devices to PCs to supercomputers. See: https://news.ycombinator.com/item?id=24260635
For hard realtime requirements, QNX is/was often used.
Splitting off all non-critical operations to isolated nodes is best practice.

Would an OS like QNX or VxWorks be more suitable for a car?

Parts.

It's very easy to choose these trade offs yourself by modifying the kernel and creating your own version. Linux gets far higher real world use, meaning that even if there are errors, they will be patched upstream and you can integrate those into your patch. Code re-use helps us in this respect.
Barring a few small details, it's possible to configure linux to be a deterministic-scheduled RTOS or whatever you want, really. I think it's a good idea to use a (hardened) version of linux for these systems, and other rigorous design principles will handle the rest (for example, testing, signed binaries, program determinism, fault tolerance and redundancy).

Intel is investing heavily into Linux for more serious stuff in automotive, like this cluster instrument that is in production already:
https://projectacrn.org/acrn-project-chery-exeed-launch/
For deterministic high frequency real time, like controlling an internal combustion engine or the damper control of the chassis, for heaven's sake use a deterministic cpu running a real time is.
However the parameterization of that real time control, based off sensor fusion, object detection, trajectory planning, all high data volume highly dynamic processes, that can happily live on Linux.
Linux thrives in high performance volume data processing in networked environments.
The challenge is to not show but to rationalize and "explain" it's dependability. It works. How can we explain why this reliability is not random, but explicably dependable?
That's what https://elisa.tech is all about, and my gut feeling is, that Linux will move from infotainment to certifyable functional safety within the next year or two.

"Reference applications including media player, tuner, navigation, web browser, Bluetooth, WiFi, HVAC control, audio mixer and vehicle controls"
It is nice to have a alternative to Google's car spy software, but I am suspicious of any ECU containing malloc. If ECUs gets so complicated that they require Linux maybe there just is a fundamental design issue.
(EDIT: ECU as in Electric control unit, not only the Engine control unit.)

ECU can mean both engine control unit and electric control unit.
The former is a safety critical hard real-time system, the latter can be anything including the widely open "infotainment" system.
My understanding is that AGL currently mostly targets infotainment systems.

Ye I know ECUs are more than the engine's ECU. I am fine with the car stereo running Linux and getting access to read only CAN data, or at-least with some hardware upstream CAN-filter for what it can send on CAN.
Given the auto-update fad that breaks my stuff all the time on my office Win 10 computer (both from MS and the company IT department). I'm not very keen on having over the air updates for ECUs or the Infotainment system. At-least not automatic.

I did over the air updates of every ECU in a car 20 years ago. This was just a demo at the research lab of the car manufacturer but this has been possible for a fairly long time.

From my time in the industry, AGL is not even close to considering targeting Engine Control Units. The primary target is infotainment systems.
At the time I never heard anyone - not infotainment guys, not ECU, not the Linux geeks - seriously proposing to put Linux there.

To be fair they don't seem to want that or use AGL as an ECU OS, ye. However looking at the demo cases they seem to at least to speed control interface with AGL. The quoted list point of vehicle control can get quite broad if car companies are not conservative enough.

It is stated there. It is intended for _rapid development of new features_. So it is intended more for prototyping to test new feature and prove points.

Nah man python powered ECU is the way it's going to go.
The engine stalling whenever the GC runs is a bonus feature.

I know you're joking, but I bet it'd be possible to make a python-powered ECU these days. It's not that hard to make non-allocating code in the first place, but even then the amount of data is so small that a GC cycle could be done in a ms or so at most.
An engine running at 3000 RPM is only doing 50 rotations per second, so you have a 20 ms budget to calculate whatever you want, hit whatever spark plug timing you need and then GC.

Engines can turn faster than 3000rpm, even in the general case. Car engines are 4 strokes, so you're probably interested in collecting data and issuing control signals through those 4 strokes. I believe my small car redlines at 7,000rpm, so that leaves roughly 2ms per stroke (for all 4 cylinders).
Also, the ECU does more than just the cylinder values, it's monitoring other systems as well, such as the transmission, traction control, &c which also have hard deadlines.
I don't think there is enough time to garbage collect, and as other gave stated, there should be little to no need to allocate in this type of code anyway.

> It's not that hard to make non-allocating code in the first place
It sure is in cpython. `a = 1` may need to allocate/reallocate the dict holding the local/global variables. If an integer exceeds 256 or goes below -5 you're getting an allocation for the object - regardless of whether that number has been seen before or not. Of course you can forget about floating point and function calls - maybe even function calls to C functions.

Trivial ECU's run at 500Hz, better ones at 1kHz, the very best (Formula 1) at 10KHz. (fixed typos)
There's for sure no dynamic memory, no recursion, only fixed memory slots.

Rurban wrote: "the very best (Formula 1) at 10 KHz".
how much of the above is a typo and how much is real?
I'm pretty sure 'KHz' is a typo which maybe should be kHz as in the start of the sentence. But even then, why are the very best running 50x slower than the trivial ones? Or did you mean MHz?
Do you have a link to an example of an F1 ECU with the clock rate specified?

Oops, big typos. Factor 1000 slower, fixed in parent. Thanks
Sorry, no links afaik. Internal knowledge.

To me it looks like the frequency of the control loop, not the cpu core frequency was meant? 1kHz is typical for i.e. steering or abs control loops.

Sure. The cpu's are normal Intel 4GHz. The control loop is 10Khz. In space the CPU's would be radiation hardened and throttled, somethink like 250Mhz. Still enough.
The bottleneck is not the CPU, nor the model running in the loop, but the IO devices and driver's costing latency. In real-time latency kills you. Which turns out to be bended cables, missing resistors, broken filters, or such. A simple cycle-miss in real-time means reboot or worse.

With a (conservative) 10k RPM limit, you get a 6ms per rotation. But I would imagine the control must run many times per rotation, so it has a chance to do something different depending on the angle, like a physical camshaft would do. Say you have to re-calculate everything at every 1 degree of rotation, that would give you a 60kHz calculation rate or about 16 microseconds per iteration. I guess that's much more than even the best case in OP's example; I wonder what the compromises are then. Surely they don't punt on the granularity and (say) only do something every 10 degrees? Or worse?

About every microcontroller has hardware timer devices that can be set to trigger a signal change with N-nanosecond resolution (N from say 5-50), and even interrupt the core when it is done. The only sharply timing-sensitive bit would be ignition firing, so maybe control those with a hardware timer.
Thing is, you can use the whole repeating schedule, parameterized with adjustments to instantaneous RPM, for many cycles before there is need to check for e.g. throttle input changes.
In between, it is a dance -- shoot some fuel into 1, wait a bit, stop shooting fuel, wait a bit, start closing a valve, wait a bit, schedule a spark, wait a bit, ...
All the settings are chosen, whenever input changes, by table lookup. So tuning means fooling with table entries.

> An engine running at 3000 RPM is only doing 50 rotations per second, so you have a 20 ms budget to calculate whatever you want, hit whatever spark plug timing you need and then GC.
The engine should still work at redline, so lets go for a conservative 6000 RPM, or 10ms budget. The spark plugs don't fire at the same time either, but (as far as I know) every second rotation in a staggered manner. So with a 4 piston engine you'd have 5ms (one firing every 180 deg) to do calculations, check for failure conditions and run the gc, with the latter one needing to finish in time.
It might be possible to do a proof of concept, but I don't think its that easy.

It's even worse considering all of the conditions being controlled for and the precision expected. Spark timing needs to fall within a degree of rotation. Modern cars get direct injection of fuel into the cylinders, which needs to be provided at a precise volume at somewhat less precise timing. Continuously variable valve timing and lift control cylinder conditions for improved burn efficiency while preventing an environment that encourages preignition, at the same time providing as much volumetric flow as possible. All of this needs to account for airflow, altitude, intake air temperature, variations in fuel quality, hiccups in fuel supply, engine temperature, over the entire range of possible engine load and speeds, and transitions throughout all of these states.
It's a seriously hard optimization problem, one that's not yet up to the task of providing reliable engine management quickly enough to also provide throttle-by-wire response in general production cars that isn't as spongy as an old Mercedes diesel sedan. I mean, if Mazda could provide throttle response for the new throttle-by-wire Miata that was remotely similar to a 30 year old model, I think they would.

Don't give Hacker News any ideas, or we'll end up with an Electron-based ECU (as-a-service).

It doesn't say how tight the control is, however. There might be a RTOS-based embedded system between the Linux system and the actual vehicle.

Why not Real Time Linux?
https://wiki.linuxfoundation.org/realtime/start

ECUs use "proper" RTOS, where everything is fully static.

I'm not sure how you get from "use on vehicle controls" to "they want to put it in the ECU"?

I understand the difference between “taking control inputs and sending them to the ECU,” but you have to be pretty naive to assume no one goes from “vehicle control” to “engine control unit.”

But if the commenters immediate reaction is "that's a bad idea", then it probably makes sense to assume that's probably not what they mean with the word?

15 years ago, I'd have laughed if someone predicted we'd see speedometer and entertainment system on single-screen dashboard, as it was 'obvious' there had to be an air gap between them.
Just because something once seemed like a bad idea to me, doesn't mean it won't be done - perhaps successfully!

For Mazda, Linux is used in the infotainment system. There is an entire community built around tweaking that infotainment system, and somebody has bundled those into a package called Mazda All In One Tweaks. They've done some pretty impressive stuff, such as adding Android Auto support before Mazda officially did. https://github.com/Trevelopment/MZD-AIO/

Even more, the Mazda's UI (like many other car infotainment stacks) is actually written in JavaScript. The infotainment boots into Opera browser and renders whole UI as a webapp.
Otherwise it's a rather nicely designed system - using dbus for communication without too much of a wierd hacks.

A lot of the other stacks are using Qt Quick / QML.

In a past life I built some of these displays with C++/Qt. At the time, QML was still in alpha.

You should take a look at QML. It's wonderful.

I should take another look at it. Thanks for the reminder.

On the one hand, MZD-AIO is very cool, and getting Android Auto on my 2015 Mazda 3 is so welcome.
On the other hand, with or without Android Auto, my infotainment is wildly unstable, very slow to start (and inconsistently so), and generally sluggish. Starting this spring, Android Auto often crashes out a few seconds in, or if I try to interact with anything (though that could be in part or in whole on the phone side - stock Google Pixel 3...)

Last I contributed to the the AIO Auto codebase, it did have quite a few race conditions and nasty hacks to make it working. So having it be unstable is not unsurprising. The SoC in the infotainment also isn't a beacon of speed.
Are you using the AIO Auto implementation or the official Mazda one?

To my knowledge the 2015 was never given an official version, though my information could be outdated.

All Mazdas running the Connect system (for which AIO is built for) are compatible with the official Auto/CarPlay kit. So it should work :)

Presumably you have to go to the dealership and pay them to install it? I couldn't find anything online (other an AIO) for getting it through official channels.

The kit requires replacing the USB hub and adding a new USB cable to the infotainment computer. The equipment itself costs about 150$ and plenty of people did the installation themselves (apparently the hardest part is to get the plastic off from dashboard between the screen and the USB ports to run the new cable).
https://www.amazon.com/Mazda-CarPlay-Retrofit-0000-8F-Z34-TK... / installation instructions: https://cdn.shopify.com/s/files/1/2367/6353/files/CPAA_INSTA...
I heard that USA dealers charge about 400$ for both eqipment and installation work. Here in EU I paid about 250EUR.

Why? Tesla vehicles run a Linux based OS (iirc it's a derivative of Ubuntu) to do exactly that. We're not talking about engine management or anything real-time/safety critical (I hope!), but things like controlling media playback or sending a request to whatever controls lighting to turn on the cabin lights when you press a button on the touchscreen are totally fine.

Not that I disagree per se, but why does "containing malloc" mean "complicated"? I'm a lot more confident in 50 lines of well-written C that uses the heap than 500 lines of custom assembly that cleverly reuses the same static memory region for multiple purposes.

"Complicated" meaning there is no certainty to how long it will take, or even whether it will succeed at all.
Nobody uses assembly code if they can avoid it. But the machine instructions that implement the code are known and unchanging. And nobody re-uses that memory for different things at different times, if there is enough for each use to have its own. Which there always is, nowadays.

> but I am suspicious of any ECU containing malloc.
The sane thing to do when using a general purpose OS as an RTOS, you pre-allocate memory during an init phase then run the control loop as if statically allocated. An ECU (or any tight control loop really) should have ZERO need to dynamically allocate memory while running.

ECUs have very strict timing requirements that would be unsuitable for a general purpose time share system. Linux would be more for guidance and entertainment.
You wouldn’t want a malformed mp3 to cause catastrophic engine failure. You would isolate the two systems completely.
Likewise something like auto pilot would sit above the ECU and be isolated from anything accepting user data for the same reason - you need hard deadlines for the various navigation functions to operate within - not as tight as the ECU where you are counting click ticks but much more strict then typical general purpose OS.

I think I know actually ECU's running on linux. Not only the Spaceshuttle ran it's "ECU" with 3 dSpace RTLinux controllers. Also the standard Formula 1 ECU by McLaren Applied Technology (Woking UK) runs on that. They also supply the ECU's for NASCAR, IndyCar and Formula-E. Based on the very same technology, dSpace RT.
The main reason are the fast MMU IO drivers available for the IO boards, like the 1GB network card or firewire. You dont want to write those drivers. The OS itself is quite trivial compared to that. That's why they use the antique and insecure design of Linux, compared to a modern, small microkernel. https://www.mclaren.com/applied/case-study/formula-1-ecu/
I wrote a C++ cross-compiler for that. And we even linked Fortran BLAS into that. dSpace only supports C, but you really want overloaded ops for matrix calculations and such. There's no malloc, no exceptions, no terminal IO, no HD.

> Not only the Spaceshuttle ran it's "ECU" with 3 dSpace RTLinux controllers.
This doesn't make any sense. The SSME 1st gen controllers were from the 60s, the 2nd gen controllers 68k-based and designed in the early 80s. The main avionics were AP-101s, which predate Linux by decades as well.

Having worked on this project, and a handful of features in it, the "vehicle controls" mentioned here are things like putting the convertible top down, or raising/lowering the ride height of the vehicle. It's all higher-level communication down to dedicated systems that directly manage the feature.

I was interested in dabbling with this project until I came across this regarding Python support:
Note: Python 2.7.3 or greater excluding Python 3.x, which is not supported.
I'm not a developer by trade or talent, but I'm learning and I'm starting with Python. Everything I've read and been told about Python development says to start with 3.x because the 2.x branch is officially deprecated and should only be targeted if one is intending to learn specifically for maintaining a legacy project that still uses it. I'm learning to learn, not to work, so I think I'll wait until AGL is brought up to a current version of the language before I dive in and get myself confused.
Beyond that though, I do like the premise of having the infotainment stack open source from the ground up. It would be nice if manufacturers would pay attention to this project and write open source drivers for their hardware.

The Python 2 dependency would be on par for industrial/automotive stacks, unfortunately. This area of the industry is not really up to date regarding modern software development practices (sometimes for good reasons, but more often not).

I am surprised to see this posted here. Among the general population, people think adding features is easy, but I'd expect people on HN would understand tech debt.
Maybe they don't understand automotive is old. There is more tech debt than you can imagine.

I understand tech debt. But you can't just handwave all tech debt as 'it just happens, get used to it'. It's debt. You're supposed to pay it back at some point, and automotive/industrial/embedded is really, really bad at it, forever stuck in a 'it works, ship it' mentality.
If you're never paying back your tech debt, it's not tech debt. It's just poor software engineering practices.

You can't shut the line down. At best you have a week off for Christmas.

I work with embedded systems and I can tell you for a fact that there are exactly two reasons for not including Python or newer versions of it:
1) Hardware constraints. 2) Software must be real-time.
Non-embedded developers are usually unable to understand the constrains of embedded and chalk it up to "you are dinosaurs". Not true.

Sure, and I wouldn't run Python on embedded devices either [1] - and that's not what I was suggesting in the first place.
The lack of Py3 support is not about Python vs $language on $environment, it's about using EOL software in general.
[1] - my last public embedded software contribution was in fact an ELF loader for a smartwatch so that I wouldn't have to use micropython: https://git.card10.badge.events.ccc.de/card10/firmware/-/tre...

Looks awesome!

I get that, and I wouldn't want a higher level (interpreted) language like Python anywhere near safety-critical systems that require a real time OS and software. Given this project targets infotainment only, well let's just say I've seen terrible systems from the major automotive manufacturers (Ford, Chrysler, etc) in that space, not to mention the hilariously bad aftermarket infotainment products flooding the market right now, and I'd like to see an open source project come out swinging and not be encumbered by last decade's language choice.
But again, I'm not a developer and I'm just getting my feet wet so my opinion is likely wrong.

I really wish there were more beginner developer friendly diy options for car entertainment. I’ve been considering making a carputer out of a raspberry pi, however, half the solutions are proprietary (looking at Qt), however there were some demos of this running on an rpi. So that is a nice development.
I’d rather just put a normal Linux distro there, however, I’m not feeling like waiting 8 seconds every time I turn in my car.
Apple CarPlay is in the same boat requiring an MFi private certificate, and no way to provision a short lived cert like they do for iPhones.

which "proprietary" bits of Qt do you think you'd need? There isn't much that's not LGPL or GPL - and while the safety renderer could have automotive applications, you wouldn't care about it for a DIY entertainment system.

> I’d rather just put a normal Linux distro there, however, I’m not feeling like waiting 8 seconds every time I turn in my car.
Suspend-to-RAM works decently to avoid this, and to avoid it sucking the battery dry, Suspend-to-Flash can be used. Yes this will take some time for restoring RAM... but again, to avoid the user noticing that time, why not start the wakeup process when the user unlocks their car / opens a door?

Do you have any resources or guides on how to get suspend to ram working?

Arch Linux has a decent guide explaining the behind-the-scenes stuff: https://wiki.archlinux.org/index.php/Power_management/Suspen...
On desktops and laptops, it Usually Should Just Work (tm). The general culprits for issues are proprietary graphics and wifi drivers as well as shoddy BIOSes with broken ACPI tables, but I never had to dig this deep.
For embedded systems, the situation is more complex, as the chipset needs to support it - suspend itself is not the problem in most cases, but wakeup is - for example, the Raspberry Pi does not have a dedicated interrupt (https://github.com/raspberrypi/linux/issues/1281). If you're working with an embedded chipset, your best bet is the manufacturer, they have to deal with that in the BSP.

Whenever I read comments like this, I think "Why is he trying to use a 10+ years old Ubuntu, or similar"? Or do I just make way luckier buying choices? Suspend to anything has worked flawlessly out of the box for my devices since 10+ years. Do you have a specific example of what is not an "obviously bad hardware combination choice", but yet still does not correctly support Suspend-to-RAM?

> Suspend to anything has worked flawlessly out of the box for my devices since 10+ years.
lucky you - I have a laptop (GS65) where suspend-to-ram occasionally fails to unsuspend, even on windows

> however, half the solutions are proprietary (looking at Qt)
if Tesla could write their UI stack with the LGPL version of Qt, likely you can do so too (https://soylentnews.org/article.pl?sid=18/05/20/1737213)

I've been in the same boat. If you really want a hackable infotainment without a TON of work and electrical problems, your best bet is an Android head unit which supports aftermarket OS's. It's not perfect, but you save yourself the headache of getting the CAN Bus working, connecting the Audio, getting the startup right etc etc.

> I’d rather just put a normal Linux distro there, however, I’m not feeling like waiting 8 seconds every time I turn in my car.
I've been working on a Pi Zero W based carputer (really, it's headless and just interfaces with my phone over Bluetooth / the existing radio head unit and steering wheel menu and controls).
Boot time is probably around ~8 second mark currently (fully booted with buildroot / systemd, it's powered from 12V switched by the key) and it already feels fast enough. I normally start my car, put on my seat belt and look around. By the time I've finished doing that it has connected to my phone.
Of course, it would be possible to do even smarter things. Hook up a low power micro that can interface with the CAN bus and a MOSFET, and you can instead power it from the battery and turn it on when the car is unlocked and off after a timeout based on the engine going off.

You might want to give eCos a try. This should be runnable on a Raspberry. I managed to port Luvit (Luas NodeJS) in a few days. Then you should have a system booting within 500ms.
https://ecos.sourceware.org/ https://github.com/luvit/luvit
Sadly there is not a really big community around both of these.

Many manufactures use the Intel Windriver distro of Yocto. SQLite3 is king just like on a mobile phone. Dbus/Qt for GUI applications. Froglogic Squish is a PITA https://www.froglogic.com/squish/editions/qt-gui-test-automa... - need an open source alternative.
Test builds should have ksan/asan turned on by default.
Production builds should ruthlessly remove unused binaries and make heavy use of --gc-sections to cull dead code. Even SQLITE3 should have features whitelisted to make it more safe.
Several TLA+ and Alloy specs need to be written to ensure distro saftey.

"AGL includes the meta-updater Yocto layer that enables OTA software updates via Uptane, an automotive-specific extension to The Update Framework. Uptane and TUF are open standards that define a secure protocol for delivering and verifying updates even when the servers and network–internet and car-internal–aren’t fully trusted."
https://docs.automotivelinux.org/docs/en/master/architecture...

Half the battle is hardware, who's going to make the computers that will run in Alaska, and Mexico, for 20+ years?

NASA?

The same companies that have made the computers in cars or industrial control systems for the last 10-20 years?

More like 35+ years. Honda has had electronic fuel injection systems that used Intel chips at least as far back as 1985.

Isn't this mostly an issue for silicon fabrication? CPUs hardened against space radiation environments are based on existing "regular" designs and I'd expect almost no changes in digital logic.

For an infotainment system, or even things like AC and fan speeds, this sounds pretty cool.
But for the dataplane software responsible for dashboard clusters and the like, this scares the hell out of me, because linux isn't an RTOS and because of all the ways you'd ahve to protect that system from anything else running on that same kernel (eg: an update to applecar play or android auto adds/exacerbates a resource leak). If the vehicle system was designed with linux in mind from the beginning, having linux handle the data plane startup and restart as well could be ok (because you'd probably make user-space programs with direct hardware access to dedicated hardware and priority that preempts the kernel to make the code as predictable as possible), but combining the dataplane for driving the vehicle with infortainment into a single OS scares me. A container or two could work here, but now we're talking about running a container at a priority high enough to linux to fuck off it's in the middle of something, and the base OS would need to be locked down decently hard as well to ensure misconfigurations wouldn't bork that dataplane container.
Really, i'd much rather have it (keep it?) the way airplanes are designed where the dataplane and control plane for manuevering the vehicle are entirely separate from any other systems on the vehicle down to at least vlans with well-defined APIs for the maneuvering system reporting data to less-critical systems.

I just can't see the Linux kernel ever obtaining ISO 26262 certification at the ASIL-D level. It would take a special kind of masochist to even consider doing that.

I just can't see that the HMI requires any ASIL-D.
ASIL-D is most likely realized by a central ECU who manages state.

Agreed. SIL2LinuxMP generated more questions than answers, which is very good for a start. But is is discontinued and we'll have to see if elisa takes off or just pokes around in some corners. The problem is, a safety system must be fully understood and then all changes need to be reviewed not to destroy the given guarantees any more. Given the pace of Linux development, this would be a gargantuan endeavor or you'd need to stick with an old and stripped down version that does not have any benefits over a traditional rt os.

Are you saying this because you think the people contributing to AGL are unaware of how certification works?

I am saying this because I am very intimate with Linux and very intimate with ISO 26262 certification, but I try to avoid those situations a masochist will seek out for pleasure. Life is too short.

The general industry consensus right now is to validate it for a higher ASIL level (B) and then wrapping it in an ASIL-D compliant layer.

They are trying their best via the ELISA effort: https://elisa.tech

If curious see also this related thread from 2017: https://news.ycombinator.com/item?id=15289564

So sad that Blackberry closed up and effectively killed off QNX. It is the perfect sort of operating system for things like this.

they didn't kill off QNX... It is alive and well as a stand-alone company and still used in many, many automotive applications

If it is going to back as linux desktop -then no. We keep on hearing this is the year for linux desktop and it never comes. Sadly.

>We keep on hearing this is the year for linux desktop
No you don't. What you're hearing is this dead horse being constantly dragged out for another round of beatings. The problem is, nobody has ever defined what the phrase even means, so it can never be `true´.

> We keep on hearing this is the year for linux desktop and it never comes. Sadly.
"The future is already here — it's just not very evenly distributed." The year of the Linux desktop was a decade ago, and it remains perfectly usable.

How can it ever come, when the few desktops that could provide a full stack developer experience are ostracized in name of having 2020 computers replicating PDP-11s like I used to do in 1993?
Naturally the only ones that succeeded at it only rely on the Linux kernel as an implementation detail, and could actually use any POSIX like kernel.

Finally, we can have systemd running our car init!
Just don't name it 0day, or get DoSed...
/s

I remember Toyota put AGL on the Camry in 2018. Guess what? It was terrible. The system froze up and crashed all the time. So they finally caved and added Car Play support. Now their cars all support Android Auto as well.
Great! Now you have AGL in your car. First thing you doing is plug in your phone for Android Auto or Car Play anyway.
What car companies should focus on is adding things like wireless Car Play and Android support. Also Car Play supports multiple independent displays, which no auto manufacturer implements either.

To be clear here, CarPlay is not an OS. CarPlay (as is Android Auto) is an app that steams an interface to the car’s system.
It’s a replacement interface that runs on your iPhone, but it’s not an infotainment system. It doesn’t do any of the heavy lifting those systems have to do to make CarPlay possible.

My bad.

Nice to see Linux in automotive. Recently, Daimler CEO mentioned that they understand now, that they need an operating system.
Guess what, he said: "think of Windows for cars"
That made me laugh hard. If they do what he says, it will be just another nail in the coffin of good ol' Daimler.

OS for what ? for the entertainment system Windows might be better, you get official support and it should be fairly stable since you strip everything unnecessary from it.

Microsoft tried that, and it failed so hard they pretty much pretend it never existed. That world is ruled by Linux (and sometimes QNX) today.

Server room is ruled by Linux, on the embedded space there are other better suited RTOS alternatives, alongside the commercial ones, all the beloved FOSS alternatives are MIT/BSD based.
And even on server room, thanks to cloud, managed runtimes and servless, Linux's relevance is going to don't matter in the upcoming decade.

The specific context mentioned was integrated entertainment systems in cars, and no, there are no "better suited RTOS alternatives" in common use there. It's Linux or QNX.
Your generic talking points about Linux in other fields are completely irrelevant for that.

Fair enough, then I would point out to the bare metal Qt deployments and Android as well (which the Linux kernel is an implementation detail for the Java and ISO C userspace).

The original Ford Sync, which was developed entirely by Microsoft, was great. Ford decided that they would develop their own UI for the next version on top of Windows Automotive, and it was an absolute mess. They blamed Microsoft, switched the OS to QNX and developed an all new UI, which is still a mess to this day.
I don't think Windows Automotive got a fair shake.

MS made a bad situation worse there - they took over version 2 from the original devs, outsourced maintenance, and pushed back on change requests assuming Ford would let them build V3 from scratch instead of (painfully) improving V2. That, as you know, is not what happened!

Interesting! I didn't know Ford brought Microsoft back on board to try and rescue the disastrous v2. No surprise that that it didn't work out, though. If I remember correctly, the UI was Flash and the hardware was vastly underpowered.

> you get official support
How is that better than paying for Linux support, where there's a market for this service, rather than a monopoly?

I am working for the Daimler daughter which is building the next next-gen Infotainment system. Curious to hear what your concrete concerns are--given the alternatives (buying in to Android or some other ecosystem), what would you do?

QNX/Linux/eCos/Sciopta
What is it that you really need, more than CAN, SOME/IP and a small Web-Server/-Client ?
Sciopta can even be bought ASIL-D/SIL3 certified.
Fills your 4MB flash not even half

Windows for cars has existed for a while! I worked on a project using "Windows Embedded Automotive", I didn't particularly enjoy it.

Oh, my car had¹ an infotainment system based on Windows for Automotive. It even included the .NET runtime! It was fun to reverse it and.. to reimplement the ECU using a Linux board.

Reply on news.ycombinator.com