Comment by daniel-levin | Hacker News Reader

Comment by daniel-levin | Hacker News Reader

>> The Camry ETCS code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable).

>> Toyota loosely followed the widely adopted MISRA-C coding rules but Barr’s group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them.

How the ACTUAL FUCK did this happen!? The article makes Toyota's engineering team seem egregiously irresponsible. Is it typical for vehicle control systems to be this complicated? I would love to hear the other side of the story (from Toyota's engineers). Maybe the MISRA-C industry standard practices are ridiculous, out of touch and impractical.

I'm going to go out on a huge limb here and am prepared to get shot down. I've seen this first hand as inheritor of spaghetti firmware.

The limb is: don't let EE's lead a firmware group.

It's a natural division of labor. The hardware guys are familiar with all the component datasheets and the bus timings and all the other low level details. If the prototype is misbehaving, they'll grab a scope and figure it out. They're probably the only guys who can. Software is an afterthought, a "free" component.

Those same EE's who have performed that role get promoted to buck stoppers for hardware production. They're familiar with hardware design, production, prototyping, Design for Manufacturability, and component vendor searches. They might cross over with production engineering and that six sigma goodness. They're wrapped up in cost-per-unit to produce which is direct ROI. Software remains a "free" component; you just type it up and there it is.

The culture of software design for testability, encapsulation, code quality, code review, reuse, patterns, CMM levels, etc etc is largely orthogonal to hardware culture.

As a hardware designer I agree that hardware people are typically not very good software engineers.

I find we can reliably write small programs that will do everything they need to. But as the program grows, we are not adept at managing the explosion in complexity.

However I don't believe that's because hardware culture doesn't value software or anything like that. We just have no proper education in software engineering.

Speaking for myself, I grasp most fundamental software concepts. Memory structure, search algorithms, that sort of thing. I appreciate the value of abstract code qualities like testability, simplicity, reusability, etc.

I just have no actual education in how to design large programs from scratch that will achieve those goals.

I don't think it's about education. It's just that software isn't what hardware people mostly work with and you only get experienced in what you mostly work with. And when you're experienced, you gain some insight into how things will work.

I know some basics of EE but I have no gut feeling of how to build or analyze any hardware for real. It's just something I've read and made my mind to understand but it's not something I know.

Conversely, I've been writing software on the lowest to highest levels for decades and I have a hunch of how to start building something that will eventually grow really big and complex. I don't know exactly how I do it but depending on what I want to achieve I have, already at the very beginning, a quite strong sense of what might work and also what will definitely NOT work.

My take on hardware is that it's mostly a black box that usually does most of what's advertised (but workarounds are regularly needed, and I guess it must be really difficult to build features into silicon and have it work 100% as designed), and these quirks had better be encapsulated in the lowest levels of the driver so that we'll get some real software building blocks sooner.

This would be no basis to design hardware. I would make such a terrible mess out of it that even if I managed to design a chip and make it appear to work somewhat, it would fail spectacularly in all kinds of naive cornercases that a real EE would never have to solve because s/he would never venture to build any of them, just because s/he would know better from the start already.

Oh yeah.

Corollary: Don't let EEs hire your firmware engineers. They have no idea what to look for.

> The culture of software design for testability, encapsulation, code quality, code review, reuse, patterns, CMM levels, etc etc is largely orthogonal to hardware culture.

I'm confused by this statement. Substitute say ASME for CMMI and the rest of this sounds like good engineering practices in general. What, for example, about "hardware culture" is orthogonal to testability?

Well, for one thing, every instance of mission critical hardware needs a full test, because hardware is messy and stuff happens. You wouldn't want someone hurt because a gate on a chip got a stuck-at fault or one of your vendors had a little extra variation etc etc.

Software only needs testing when the tools or inputs change. So design for testability of software is driven by the needs of a development group (eg simulators, debug statements), while hardware (at a minimum) is aimed at a post production group (eg JTAG, test fixtures).

I agree there are many parallels, but I think the culture is different.

Maybe the MISRA-C industry standard practices are ridiculous, out of touch and impractical.

I currently work in the static analysis industry. Broadly speaking, the MISRA-C ruleset is, in my experience and in the experience of many customers who apply it, not impractical, clearly of benefit, and the majority of rule violations (including a number of rules violated by Toyota) easily found with static analysis tools.

I even know that Toyota is a customer of (at least) one of the static analysis tool companies (obviously Toyota is a huge company, and I've no idea if the specific muppets making this clusterf had any static analysis tools; just that Toyota as a company definitely has someone buying them). It seems that they either just didn't use it, or just didn't care (or were told not to care).

As an extra point of data, I sit opposite someone who is on the MISRA-C committee. He is a solid C coder who knows a great deal about the language and how to get it wrong. His day job is writing/maintaining static analysis tools for C programmes. Obviously this is no guarantee that the committee as a whole is good at maintaining the MISRA-C ruleset, but they do have at least one active, experienced and competent C coder (with lots of experience of having to actually automate detection of rule violations) at the table.

It's a matter of scale and economics. Every dollar cost reduction per unit is significant over the multiyear production cycle. The changes for cost reduction make the design process more complex and a Camry or any car is full of legacy features (the model designation is nearly 35 years old).

On top of that car companies are massive and capital intensive with all the lack of agility that requires...retooling a production line takes many years from design to cars at the dealer. The critical decisions about programming stack had to be made in the early 1980's when the first microprocessor based control systems appeared in automobiles.

Back then, automotive engineers and executives would have hardly predicted the software complexity that was on the horizon. It was state level actors who came up with Ada and Drakon...But those are money is no object and failure is not an option inspired. Toyota probably made money via their level of software quality...shutting down production for a year to get it right would have cost billions. It followed the bean counting and rode the tiger.

More interesting: how does it compare to the rest of the industry at that time?

I'm not really qualified here (and if someone is please post) but I've been poking around with the ECU on my 2001 audi and it doesn't seem to have ECC memory either. A section of the AM29F800BB eeprom is used as random-access memory. It seems to stick a couple CRC bits in every line so maybe this is how they're doing it?

I am starting to think that safety critical systems need to have schematics, pcb layouts, design documents and source code be registered with the gov before it touches the public.

I remember when nearly everything came with schematics so it could be fixed.

When hardware designers write code, the results are often ugly. Device drivers, firmware, embedded code, ...

Engineers tend to read specs and apply standards.

>I would love to hear the other side of the story

Me too, it's probably indeed bad, but there's ALWAYS another side to such stories. Unfortunately, we rarely get to hear those.

I'm in no way excusing Toyota. For those who want to understand how this happens, I can shed some light.

Early firmware was a replacement for analog electronics. Consider modeling the classic lunar lander. A simple digital computational loop with no branches, just math, has advantages over analog circuitry, such temperature stability, noise immunity, reproducibility, etc.

From there, embedded firmware grew in the same way complex mechanical systems grew, like mills and robots. Meanwhile, non-embedded software on workstations was growing, theories were developing, algorithms and organization and business models were applied. And firmware grew and grew, but mostly like an engine grows in that abstractions are minimized, optimization happens in-place and theory is largely tribal knowledge.

It's only in the last 10 years or so that firmware has metamorphosis end and is confronting other software. It looks especially conflictual (word?) to high-level devs like Ruby or what-have-you.

So yeah, this is no excuse for negligence and ignorance. I just hope my perspective helps ppl evolve.

The apollo guidance computer actually used a sophisticated software interpreter for most of its code. Not just a "computational loop with no branches"

http://en.wikipedia.org/wiki/Apollo_Guidance_Computer#Softwa...

In addition, there is no way in hell that the control algorithms it was using could have been developed without the use of computers. State-space control theory was specifically developed to take advantage of discrete-time control systems.

https://www.hq.nasa.gov/alsj/ApolloDescentGuidnce.pdf

MISRA-C is practical enough that NASA's JPL used it as a basis for their coding standard (http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf), and everyone would agree those Martian robot works pretty well...

> Barr’s group found 80,000 rule violations

I'm a bit puzzled by this figure given that according to the article the typical LOC count of this kind of software is within that order of magnitude ("tens of thousands of lines of code").

"Normal" C code that is written in a coding style that disagrees with the MISRA guidelines can have more than 1 violation per line of code, even if the quality of the codebase is otherwise quite good.

For example, the static analyser I'm working on finds 32700 violations of MisraC2012 in the SQLite codebase (130 kLOC).

This is because Misra is quite strict about the use of C: a simple statement like `if (ptr) f();` already violates two rules:

* use explicit `ptr != NULL` comparison

* always use braces with `if`

Once you get around to integer arithmetic, where MISRA is quite strict about not relying on implicit conversions or numeric promotion (which is non-portable due to depending on the size of `int`), it's not rare to have 5 or more violations in a single line of code.

Reply on news.ycombinator.com