Hacker News Reader

Comment by iagooar | Hacker News Reader

Any chance one could virtualise some or most parts of the hardware?

Yes, but the problem being solved is often a double problem. A) develop embedded software to control a system of "stuff" B) characterize the system so you know what controls to apply.

The virtualization helps you ensure that A is doing what you intend to do, but not as much with B) knowing what your intent should actually be... and even if you can build a reasonable model of 80% of the behaviour, the really critical stuff tends to be out of the envelope type constraints like, oh the sensor gives this bogus reading under conditions X, so you need to ignore it or avoid condition X.

Edit: case in point (an adjacent story in HN)

https://news.ycombinator.com/item?id=13700798 "UPS Showcases New Delivery Drone, Fucks Up"

If you're developing the firmware, use a Hardware Abstraction Layer (HAL), and having the application logic only communicate with/through that. Have a version of this layer which can run on your host computer. The majority of the code then becomes testable with very quick cycles.

The HAL layer for device must be tested separately. Ideally do it in an automated systems test jig, which has the target hardware+software, and allow programmable input stimuli and output verification. You really should have such a setup for your final (per-board) QA anyway, so better to start developing it sooner rather than later.

This is money and time. Great if you have it.

Yes like everything else. HAL and host-based simulation usually lets you start developing the software earlier in the product cycle, which can make available a lot of time. If one doesn't have the capacity to invest in automated QA then things might get tricky no matter how things are done... Trying to simplify the problem / reducing scope might be best course of action.

I do agree that it is too costly though, and I think this is partly a tooling issue. Hoping that open source software and hardware can help here.

For instance, on-device testing is a first-class citizen in the microcontroller programming framework I've developed. Shameless plug: http://www.jonnor.com/2017/02/data-driven-testing-with-fbp-s...

This only works if you're not doing low level stuff.

I recently ran into an issue where an interaction with the DMA controller and the SDRAM controller in a MCU lead to invalid reads from the SDRAM. I still don't know what is actually going on with the MCU (SAME70, in this case).

Using a HAL lets you debug your application code more easily, but I don't think I've ever had a hard problem in an embedded system where the underlying cause wasn't related to some odd hardware behaviour under a corner case stimulus. There's no way to simulate that without a full model of the entire device (which isn't available).

Yeah, the HAL itself does not help much for hardware issues. But if tooling was built for automated on-device tests (to test your HAL and app logic), then you have a much better chance of having a system which can help you reproduce spurious hardware failures. For instance using generative techniques like fuzzing on the software side, or just a predictable test-patterns which can be observed with scope/logic-analyzer. On the testbench side use temperature stressing, under/overclocking, under/overvolting, adding capacitance or inductance to signal or power lines, modifying grounding schemes, can be done to try to provoke the situation more reliably and (maybe) get a better understand of the problem.

It doesn't always make sense.

Debugging a quality control machine I inherited:

The heat sensor would trigger randomly, about twice a week.

It was a software problem, as swapping out for a brand new one didn't fix anything, despite doing it three times.

Everything seemed above board, and the review didn't take that long, it was only 8000LOC, simple C.

I found the answer on a test device: The vendored library that we could link against, but not look at, was reading the temperature from a sensor over I2C... And then reading it into the main onboard chip via an analog pinout. The docs said digital.

Which meant, when an inspector walked passed the machine, and looked at it, their own electromagnetic presence would mess with the readout. It wasn't sealed properly against this, because we believed the documentation.

It's possible in some cases, problem is that work is often harder than the thing you're actually trying to do ><

Also a lot of the time you don't really have good specs or there are closed source bits e.g. firmware blobs for wifi.

The biggest problem to virtualization or other simulation methods is just return on investment. Most applications are so niche you can't justify anything but bare bones tools. Any money spent just getting your product in front of a potential customer will be better spent than hardening your development cycle.

You don't need to virtualise if you treat the embedded hardware as a dependency - in the same way that a database or a remote web service can be dependencies and we can unit test without access to them.

That said, there are some cases in embedded where there is just no possible substitute for the real thing. I've heard of physical actuators and webcams being used to drive devices from test harnesses in those cases. I think a popular example of this that's open source is published by one of the pay tv companies.

I've done a few projects where we built simple robots to physically manipulate the device under test. It saves huge amounts of manual test time.

Many of the Android app test facilities use webcams to report what's actually on screen rather than relying on software hacks (which are different for every device).

For code that is generally in open source land, you may be able to get some traction cross-compiling for x86 and using tools like qemu/virtualbox to test higher-level functionality. However, it's usually the peripherals and whatever high-speed serial links (e.g. wifi) that you need to get better visibility into.

Reply on news.ycombinator.com