The Quality of Embedded Software, or the Mess Has Happened | Hacker News Reader

The Quality of Embedded Software, or the Mess Has Happened

viva64.com

65 pointsDmitryNovikov55 comments

I agree it is a bit sensationalist, and certainly disparaging of EE's, but the truth that there is more and more software between you and an unpleasant experience is absolutely true.

One of the things that really annoys me is that there isn't more openness and sharing in the embedded space. Its not uncommon for me to go to a manufacturer lecture where the only compiler choices are proprietary offerings and their library functions and includes are riddled with proprietary licenses. In one case I pointed out bugs in their (re)-implmentation of some basic libc functions that were fixed in newlib years ago and pretty much everywhere else.

While the people around me fumble with their Windows [1] machines I pull up a gcc cross compilation chain and begin porting the proprietary "workspace" or "project" files into a bog standard make file. I've gotten to the point where I can have their 'demo lesson' running in a fully open toolchain faster than they can get the rest of the attendees windows and custom package install issues sorted out. Of course I've already got the basic tools installed, and generally the other students are installing an older version of the same toolchain I'm using but into some incompatible set of directories that just the vendor uses.

I went to Atmel, TI, and ST Micro chip seminars (looking for a processor for my educational development board) and had I installed the proprietary tools I would have ended up with three copies of GCC installed on my laptop!

I've been supporting efforts like the libopencm3 desire to build an open and common ARM support library, but more of that needs to happen to build in some good practices for these embedded developers to learn from.

>One of the things that really annoys me is that there isn't more openness and sharing in the embedded space.

Embedded products are time and capital intensive, which means that the cost of a compiler or other tool-set is very small relative to the total cost of product development. Because of this, the developers (and their companies) do not mind paying a few thousand dollars for a new piece of software. Anything the developer creates is often only useful for competitors to one's product, so there is very little incentive to share.

>I've gotten to the point where I can have their 'demo lesson' running in a fully open toolchain faster than they can get the rest of the attendees windows and custom package install issues sorted out.

If you spend a year working on developing an entirely new hardware & firmware product, you see that importing new files, finding libraries, and adapting software from one IDE to another is not a very important or time consuming part of the project. When you spend a few weeks discovering that there are undocumented hardware glitches in the integrated circuit, that the errata is inaccurate, or that you own hardware behaves unexpectedly, the IDE is of little consequence.

>I've been supporting efforts like the libopencm3 desire to build an open and common ARM support library, but more of that needs to happen to build in some good practices for these embedded developers to learn from.

As an embedded developer, the open ARM library is of little interest to me. ARM is proprietary anyway, and building open-source projects on top of closed source systems does not seem to be a productive use of time.

> people around me fumble with their Windows [1] machines I pull up a gcc cross compilation chain and begin porting the proprietary "workspace

Also the trend to "integrate" all that stuff in shiny windows applications makes it increasingly hard to have collaboration with shared repositories and automated builds, because "workspaces"/"Project files" tend to happily mix user-interface configuration, file modifications or code-generation settings. And changes in autogenerated code pop-up in unexpected ways that makes it increasignly hard to re-integrate concurrent development (e.g. by different developers).

It's pretty horrible. And sometimes your vendor's compiler isn't GCC but uses a lot of non-gcc extensions which you have to manually rewrite in something that's sane. Or uses ELF, but in a way that contemporary binutils only can link after you've done some ugly tricks with them...

I remember during EE course we had an assigned project to write some software for 8051 microcontroller, in assembly. I organized my code into small "functions" and used a consistent stack-based calling convention with pushing all the resisters on the stack and restoring them back on exit, just as if it was generated by a C compiler. The teacher looked at this and said "huh, look, this is not a CS lab. We do not do things like this in embedded software. Why don't you just keep everything global and use absolute addressing instead those crazy push/pop calls? And you should be reusing registers..." Anyways, it was lot of fun.

On a resource and computing-power constrained system used for a very specific function it would not be too far off if one said something like that(with caveats). Anything past that category and he would be dead wrong.

In other words, if you only have 2K or RAM to play with, a limited stack depth, a CPU clock of, say, 12MHz and a 12 clock instruction cycle you have no choice but to get very clever.

Pushing and popping a bunch of registers on every function call could rob you of much needed performance and resources. In fact, on such a system that alone could cause unintended side effects and introduce potential safety issues (talking real-time systems with a requirement for deterministic response). You could also have a situation where you overflow the stack and things go from bad to worst very quickly.

BTW, this, among other things, is one of the reasons I abandoned such small, 8-bit, resource-constrained architectures for most embedded projects a long time ago. The 8051 derivatives (Silicon Labs) have always been some of my favorites. I might still use them for small projects such as small control panels or simple sensor pre-processing. These days you can buy 32 bit processors running in the tens to hundreds of MHz for $10 or less. These provide you with the ability to write "proper" embedded software with all the luxuries of greatly increased resources and operating speed.

I like the way you explained it. The way it was written above seems like he was just told he's wrong but not any of the reasons why. Might just be my lack of knowledge in the area but it made the teacher's response more understandable.

There's a huge difference between answering "beware of blowing up the stack" and "in embedded we don't do like that".

Of course, Toyota failed to make sure they didn't blow up the stack as well as using a ridiculous number of global variables and poorly organised code.

I think we are in agreement here. I had assumed one of the reasons it's "not done like that" is "blowing up the stack" but only half of that was given at the time. Maybe I misunderstood altogether.

I just appreciated that there are real reasons for things, especially when they make sense.

You're perfectly right, but performance of my system was fine without any further optimisation and there was no point in doing some dirty tricks to get more performance, even though the system had 32k or 64k RAM (I don't remember now). Also, often it is enough to optimise just a few critical places, not the whole thing.

Before starting my software engineering (SEG) degree, I did some technical EE in Cégep [1] . We first learned assembly and coded in that for a year.

Then we had a course of C and I was so confused as to why everything was push/pop at every call, when in most situations it was useless and seemed like a waste of cycles.

Luckily, between this EE thing and my SEG education, I went in the army for 5 years so I forgot about this whole question and came looking at SEG with fresh eyes.

  [1] Cégep (they call it in Québec) is a mandatory step in 
    between high-school and university.  They offer pre-
    university training (2 years) and technical/skilled-
    worker training (3 years).  In the case of 'technical
    EE', you spend 3 years doing that and you can go to a 
    university for a EE/CEG/SEG[2] degree after (2-3 years), 
    or just go work.
  [2] CEG; computer engineering, SEG; software engineering.

I know a good many bad firmware engineers. A few of my theories about why:

- Many of them have a concentration in EE, not software. The software they produce is what you'd expect from an amateur.

- A lot of engineers do not spend time to think about the whole stack, from decisions at the hardware level all the way to how the user interacts with the product, and how the factory produces it. The more comfortable you are working in these domains, the better decisions you will be making. Unfortunately, for many firmware engineers their responsibility stops at (say) the USB protocol they've hatched.

- Firmware engineering is often seen as a lower-class of programming than desktop. (To this, I say: Ha!)

I've sure seen a lot of crappy firmware, much of it from contract houses that specialize in things like BIOS and board packages. They don't seem to hire carefully.

If you run across that rare gem who can write desktop apps and UI, write host drivers, write firmware, and use a scope, hang onto that person.

For my bachelors, I did a dual degree in EE and CS, and had a decent programming background before starting. Watching most of my EE colleagues programming was incredibly painful; it was very hack & slash until it works, and then that's it. In EE, we weren't really evaluated on the quality of our code at all, just whether or not the final product worked (as determined by a 'scope or other physical output).

My limited EE experience after school (I mostly do higher-level programming these days) reflects this; this is the education they left school with, and they didn't have anyone afterwards to teach them any better. Makes me cringe so hard, knowing what some of these guys are up to now.

I know a few good firmware engineers. A reason why they, too, produce "crap" is:

- Many of them come across hardware errors or suboptimal designs during a later stage in the FW development. In most cases they hack a workaround that doesn't involve a redesign to be able to move on with the next item on the list. Folks higher up the chain of command see this as an excuse to avoid a redesign: "we're not going to do a redesign for that, Mark fixed it in software". Examples: pretty much every ARM System Board out there.

- Someone picked a device that was available during the design phase, but went EOL a few months later before production. It happens.

- Someone found a cheaper device than originally planned but it doesn't really do what it says on the box. Typically peripheral stuff like ethernet or usb.

- And here's the big one: !!Crappy Datasheet!! Some companies, some really really big IC companies produce crap datasheets. Yo get accustomed to the fact that this new Cortex-M µC will probably have an errata or updated datasheet every few months.

- the redesign fixed their EMC problem to get the certificate, someone fixed the power supply. But now you get into trouble if you drive this and that device because the voltage takes a dip.

But I'd like to add: I agree with your post.

Mobile programming sucked many desktop/etc capable people out of the market too

   --iOS guy who came from embedded linux driver land.

I'm having the reverse experience. I moved from embedded C++ to writing desktop apps.

At my former job, Bjarne's C++ coding standard was required reading.

Here, anything goes. The last programmer actually convinced the team the reason his code gave different mathematical results each time it was run was because "floating point math is stochastic".

This is not about the state of embedded software. It is about one program, the accelerator control of a Toyota.

The author of the article is linkbaiting and fearmongering to get you to purchase his/her static code analyzer for C++.

As embedded software becomes ever more complex, it will benefit from methodologies developed for other platforms, such as desktop applications, etc. That is all that needs to be said. Please refrain from the "EEs don't know how to program" comments. Thank you.

As embedded software becomes ever more complex

This is part of the problem in itself; the Toyota software appears to be far more complex than it needs to be.

> Please refrain from the "EEs don't know how to program" comments.

No, no. As an EE I can tell you that lots of EE's really suck at software development. They never really took the time to study CS to a reasonable depth. As a result you see code that is just horrid and really naive. I've seen stuff that could blow up boards Hollywood style in a microsecond. Such things as potential timing conflicts turning on two MOSFETs such that you have cross conduction and draw hundreds of amps due to the direct short from +V to GND.

Now, on the other hand, "desktop" programmers are equally --if not more-- dangerous than ignorant EE's. They have no clue and no experience with hostile embedded environments. I mean, the simple startup configuration code of some modern microcontrollers is the kind of thing you'd never see in desktop or web development. Then you have a myriad of other issues: watchdog timers, drooping/unreliable power supplies, static discharge-induced reset, load dump in automotive, similar applications, or noisy signals, sensor and key-panel data, potential single bit flip situations in aerospace/radiation environments, thermal issues, etc. The list is huge.

These are issues that are far better understood by an EE. And so, the best embedded software developer is probably going to be an EE with real solid training of CS principles as well as an solid understanding of defensive or safety-critical electronic and software design principles.

An example that comes to mind was when we designed a high-power motor controller many years ago. This thing could handle thousands of Watts. Testing included full output short circuits, zapping it with static electricity, exposing it to RF via close proximity antennas and coils and other torture tests. We discovered that under very rare conditions the output to go to full power when power was first applied AND the power connection was, I'll say, uniquely noisy. Imagine a corroded connection to a power strip on a marine platform that is moving. When power is applied, if you look at it on an oscilloscope, it looked like the most horrible noise you'd ever seen --except this noise was actually powering the board. We had to build a custom power tester using old motor brushes and an old commutator driven by a small motor in order to simulate ugly power supply scenarios. We later replaced this with a programmable digital system that could do the same. The embedded software and the hardware was then modified to deal with these ugly conditions and much worst cases as well. The thing was nearly bulletproof.

So, yeah, drop any EE in front of a compiler and you'll probably end-up with really dangerous code. An EE with the right training and lots of scars would be the best option. Someone with a bachelors or masters in CS and no EE training and embedded world experience could be scary dangerous.

I agree wholeheartedly with this comment, as computer and embedded programmers approach completely different tasks from completely different paradigms. The embedded system may have to run for years on end, with no one pressing control-alt-delete, performing a reset, or an update; it may also have to be very robust against unexpected situations, fail-operational or fail-safe or fail-deadly depending on the product, and protect itself from exposure to dangerous signals and environments. On the other hand, computer software must be easily maintained, updated to add new functions, work on new systems, be easy for the end-user to operate, and limited in scope. Expecting people with such different tasks to behave similarly is entirely unrealistic, and foolhardy.

We are not the authors of this article. We have just translated the article and posted on our website. But times have you remembered about the analyzer, I want to show other people possibilities of the instrument: http://www.viva64.com/en/examples/

Ah, yes. Comfortable desktop developers theorizing about embedded software. MISRA is more about compatibility with non-standard (or pre-standard) legacy compilers from the 80s and miniscule improvements to the static analyzability of code than about software quality; following MISRA to the letter would actually decrease the maintainability of software significantly, and this is the only valid measure of code quality any practitioner should care about.

> via an onboard network like CAN or FlexRay, a more complex superstructure over CAN

FlexRay is a completely different bus system.

If you like this sort of thing, read comp.risks ( http://catless.ncl.ac.uk/risks ). On the other hand, it might give you an omnipresent sense of fear about all the computer-controlled safety critical devices around you.

I used to work in a world with lots of embedded software. My experience was that - regardless of domain or company -the "hardware" was regarded as just that - simply hardware. And it was tested as that - simply hardware - not as hardware that, oh by the way, had a helluva lot of critical embedded software. So issues that could have been caught at some level of testing weren't.

I read that report a few days ago. It's indeed shocking. One way to explain it is that, the split of EE and CS in universities, most firmware engineers are EE graduates, who knows the hardware to the register level, but not as good on software, not at all. The life-threatening related products, such as cars, medical devices, rockets,etc, probably some strict certificate on software development is good, otherwise it's totally subjective. The so-called computer engineering major is a nice try, students take courses in both EE and CS fields.

This is why the world needs something like SEMAT http://semat.org/

More and more of our environment is controlled by software and when it breaks, it doesn't just shut down somebody's store but it does real damage to real objects in the real world. Esepcially electrical engineers should be following rigorous processes in building and testing software.

Not that this exonerates Toyota in any ways, but if your car is accelerating out of control, why not just take it out of gear (ie. put it into neutral)?

Or have they removed that ability from newer automatic transmissions? (my last few cars all have manual transmissions)

On most new cars: the transmission is entirely computer controlled.

This is the case for "semi-automated manuals" like DCTs as well as automatics.

For e.g: to shift to neutral or disable a Prius at speed you have to "hold" the shifter in the neutral position [note: it doesn't seem to stay there like it would on a traditional automatic gear selector; it's spring loaded] or "hold" the ignition switch for 3-5 seconds. This is (presumably) to prevent the motor & trans from grenading itself.

I haven't driven a vehicle newer than model-year 2003 recently: so in a panic situation I would expect my cars to make the "money shift" and blow up their powertrains; not "ask me for confirmation" that I want to destroy my motor.

Having to wait to make a money-shift seems like a feature that looks excellent on paper, but terrible in practice. I find it difficult to buy a car which only has firmware-implemented kill switches.

Is this common, that an automatic transmission won’t immediately shift to neutral? I thought that automatic transmissions usually let you shift to neutral without pushing the shifter button in to let you shift to neutral without blowing through it at trying to put the engine in reverse. Shifting into neutral while driving shouldn’t break anything, unless maybe if the throttle is fully open and the rev limiter isn’t working, but that’s the most important situation for being able to shift into neutral, isn’t it? I don’t see why an engineer would want to try to prevent this.

The Prius doesn't have a tradition automatic transmission with a torque converter and multiple gear ratios. The gas engine is connected to the wheels at a fixed ratio along with two motor/generators in a planetary gear system.

Reverse is achieved by running the electric motor in reverse. There is no reverse gear.

Shifting the neutral on the prius just means disconnecting the MGs and turning off the ICE.

Because in manuals you have a physical link between you and the gearbox (usually), Almost all modern automatics simply have an informational link. If the computer freaks out and doesn't accept the message, no change.

So strange... Old automatics always had a sort of physical connection in which you could pop them out of gear in an emergency.

This is one of the reasons that I love manual transmissions. Sure, you'll probably still blow up your engine, but that's way better than dying.

One of the interesting things from one of the other articles was related to depressing the brake. Apparently one of the solutions to resetting the out of control acceleration was to completely remove your foot from the brake before trying to brake again.

This may sound curmudgeonly, but that is one of the reasons that fly-by-wire makes me so uncomfortable. In my car, the accelerator is attached to the throttle by a cable, and that's the way I like it :). I was driving my girlfriend's 2010 Pontiac Vibe the other day (essentially a Toyota Matrix) and stopped at a slippery intersection. When I went to get going again, the traction control system detected that it was slippery and, as far as I can tell, did throttle limiting to prevent the wheels from slipping. This is one of the most uncomfortable feelings I've ever had driving a car... I pressed the accelerator and there was no response.

> When I went to get going again, the traction control system detected that it was slippery and, as far as I can tell, did throttle limiting to prevent the wheels from slipping. This is one of the most uncomfortable feelings I've ever had driving a car... I pressed the accelerator and there was no response.

Most decent cars let you disable the traction control (which is necessary sometimes)...

But yes, sometimes the newer technology doesn't work as well as the older technology, depending on the conditions and driver skill. I've had ABS kick in a few times and it's really disconcerting, I almost prefer not having it. Traction control is great for tiny patches of ice or rain, but terrible when the roads are covered in fresh snow.

My old Saab 900 Turbo was the best winter car I'd ever had, could basically plow through anything the front bumper could clear (and never failed to start - even when it was minus 40 degrees). My new car is adequate, but not nearly as good when it gets really ugly...

And fly-by wire IS scary, they should never eliminate physical connections for certain functions (accelerator, brakes, steering).

The ABS in my car lost a sensor a few years ago and shut itself off. I see it as a feature :)

There's a way to disable the traction control, but figuring it out wasn't what I had in mind while stuck at a 4-way stop.

Also not trying to blame the victim, but it seems the even more obvious/natural approach would be to use the brakes. Those are hydraulic, and plenty powerful enough to counteract any force generated by a Camry's engine.

>Those are hydraulic

Are they hydraulic from the brake pedal to the brake pad, or is the brake pedal just a software input to a brake control system?

From one of the other articles on the Camry unexpected acceleration issue, emphasis theirs:

"Vehicle tests confirmed that one particular dead task would result in loss of throttle control, and that the driver might have to fully remove their foot from the brake during an unintended acceleration event before being able to end the unwanted acceleration".

The natural approach seems to cause as many problems as it might otherwise fix!

The brake pedal actuates the master cylinder, which is connected hydraulically to the calipers. Even if stomping on the brakes triggered more throttle problems, they are still plenty strong enough to stop the wheels from turning.

Ah, but what about anti-lock brake systems?

Anti-lock brake systems work by reducing hydraulic pressure at a wheel that is determined to be slipping. It doesn't affect how braking is activated.

A little black box regulates the applied pressure due to criteria X.

I submit to you that if X is buggy, the little black box may decide never to allow pressure to hit the brakes. This in turn prevents braking.

In a car with ABS and brake assist - standard features on new cars - software has the ability to completely override your input.

My instinct would be to turn the engine off.

Don't be evil programmer. Use PVS-Studio Static Code Analyzer for C/C++: http://www.viva64.com/en/pvs-studio/ :)

https://news.ycombinator.com/item?id=6636811

I also saw a Toyota Corolla pull suicidal acceleration, I wonder if it is also a software bug.

Software like that is where we need "software engineering", with comprehensive requirements, verification, code reviews, sign-offs, extensive testing, etc.

It may also be good if such software was open source, so that anyone outside the company could also review it, if they so chose. See also: http://www.tuxradar.com/content/karen-sandler-full-interview

See the thing is, you can't really do that all that easily in an embedded environment. Pretty quickly you get into an "Uncertainty Principle" situation, where it's difficult/impossible to test something without tainting the results.

For all of the code reviews and testing you want to do, how are you planning on having everyone test interrupt flows, re-entrancy, transients in interrupt signals, defective/leaky grounds, watchdog timers, memory bus clock errors, mutex/semaphores, pullups that are too weak or too strong, etc?

Code reviews are all well & good, to be sure, but it's not just software anymore, there are physical things to think about. "Software engineering" is usually too specifically "high level" to apply as well to embedded firmware. How do you plan on writing unit tests to run? How do you test a signal edge that seems to fall outside of a valid window once every few days?

Open source the code if you want (and if it's legal: a lot of times there are NDAs signed for processors), but it does no one any good if they can't have the same hardware, made with the same components, often times compiled with proprietary compilers.

It's about so much more than just code, so be careful of mis-applying great software engineering principles to a completely different situation.

I'm not very familiar with how car component software is developed, so I can't speak to that.

In avionics, the software requirements are only one part of the system. There's requirements for how the hardware is to behave, and tolerances for acceptable behavior. All of these pieces are tested. The software is tested by itself in a pure software environment, and it's tested running on the specific piece of hardware that it needs to run on, and that software+hardware combination is tested in various stages of increasingly complete avionics systems, culminating in testing the entire airplane.

But I was just talking about the software, where we do employ requirements, verification, testing, and "software engineering". That's only one piece of the system of course, but still a valuable piece. If the software doesn't work right by itself in an idealized environment, then it has little hope of working right when the hardware is going bonkers. And even in the idealized pure software environment, we simulate various system failures to do initial testing on how the software responds.

Regarding the EE bashing, there's an element of truth but overall it is an unfair characterization.

I think the true source of the problem lies not in the EE degree or lack of a CS degree, but in a specific, weird distrust of abstraction by in the institutional culture of embedded systems development. This opposition (and it is opposition, not merely ignorance) isn't coming from the degree per se but from the field as it is sometimes practiced.

My guess is that the reason for this culture is that long ago, there was a history of "fancy newfangled compilers" screwing up the code, engineers getting repeatedly burned, and learning not to trust such "fancy" tools. This attitude may have made some sense a long time ago, but now it's an institutional relic.

So the irony is: the engineers are avoiding abstraction in order to increase reliability, but they wind up decreasing it because the code becomes impossible to manage.

Anyway, this is not a defect in the way EE's are educated, it is a very specific subculture of embedded systems development, which traditionally happens to tend to hire EE's.

Is there a reason why something so fundamental as throttle control/braking ought to be governed by a computer, instead of by foolproof old-fashioned hardware?

The Prius like all US cars has a physical hydraulic brake system. The brake pedal is connected to the master brake cylinder. Stomping on the brake pedal in a prius, regardless of the computer state, will brake the car.

The throttle is computer controlled because it allows much greater efficiency.

Anti-lock brakes still have a significant electronic component. It's not a direct link like you'd expect.

"Old fashioned hardware" is anything but foolproof. Linkages bind and flex and change their behavior over time, hydraulics leak and piping must run in areas that can accumulate rust (I had the not so pleasant experience of losing my main brakes due to a rusted out brake line. Driving with only an E-brake is not fun!), steel is heavy, etc.

Computers do add benefit: ABS, traction control and stability control would be a nightmare to implement mechanically. There is no question that the software solution is a better idea, but the design and implementation must be done correctly.

I think it is more fair to say that, being old and well-used, the engineering behind old-fashioned hardware is more mature and its failure modes are better understood. Embedded electronic systems of various types will reach that point as they are more widely deployed and used for longer periods.

BTW, I have been through that exact failure scenario before myself (in an '81 Monte Carlo).

Reply on news.ycombinator.com