So each "cycle" can be as long as, flash the thing, wait for it to warm up, get the device into the right state, test the thing you wanted to test (which might involve another device etc).
By far the most frustrating part was waiting to test it. I didn't want to run the washing machine (empty) just to test it, so I had to schedule debugging and testing times when we were doing laundry for real.
The moment it worked for the first time was definitely a happy one though!
The machine was several hundred thousand £s worth of lasers, optical table and stepper motors moving a stage that must have weighed half a ton. The purpose was to print digital holograms, line by line, on glass plates. Between each line there was a delay of some seconds, during which time various calculations took place; even the smallest 3"x3" holograms took half an hour to print. Larger ones could run overnight. Once printed, the (expensive) plates needed another half an hour of darkroom wet processing, after which they could be briefly viewed (the image disappeared again until the plate was completely dry - uneven shrinking destroys the interference pattern).
The byzantine hardware control software had been written long before my time - my job was merely to write the imaging software that fed this machine as it ran. There was an elaborate startup procedure to follow, and just before you'd hit "start" you'd run my little C program, which would synchronise with a timing signal from a serial port. Then, as the stage whirred back and forth, it would feed images to a little display - 30 Hz, pixel by pixel, line by line.
One day, just as we were getting confident of the rhythm of operating the machine, a larger hologram came out "back to front" in a strange way - one of the parallax axes was reversed. Perplexed, we carefully tried again with the same file - and it came out fine the second time. Unable to reproduce the issue, the matter was dropped.
Some weeks later, just as our confidence had begun to wax again - boom. Another inverted hologram. Another large plate wasted. Another careful investigation, another failure to reproduce. A Heisenbug!
This continued happening, off and on, most vexingly - and of course, never when I was there, and only with the larger, more expensive plates. Eventually one day I came in and it had happened again, and this time I noticed something peculiar - the hologram had finished printing, but my little C program was just sitting there, waiting on seemingly the last line, for a timing signal that would never come. And the light began to dawn.
My C program had the basic structure "buffer line, wait for signal, display images, go to start". It turned out that what was happening was people were hitting "start" too soon after launching it, before it had had a chance to properly buffer the first line. It thus missed the "start line" signal of the first line - which caused it to print line 1 as line 2, line 2 as line 3, etc. And because it was a rastering stage, this meant that every line was printed in the opposite direction to what was expected.
The really treacherous thing about it was, it only manifested when I wasn't there because I knew instinctively to allow the program enough time to buffer. It only happened with big, expensive plates because they had longer lines which took longer to buffer. And it only happened at all when the operator of the machine was sufficiently confident, or in a hurry, to perform the final two tasks of the checklist within seconds of each other.
The fix? A note in the checklist.
I miss the job, but I don't miss debugging that darned machine.
Also a lot of the time you don't really have good specs or there are closed source bits e.g. firmware blobs for wifi.
That said, there are some cases in embedded where there is just no possible substitute for the real thing. I've heard of physical actuators and webcams being used to drive devices from test harnesses in those cases. I think a popular example of this that's open source is published by one of the pay tv companies.
Many of the Android app test facilities use webcams to report what's actually on screen rather than relying on software hacks (which are different for every device).
The HAL layer for device must be tested separately. Ideally do it in an automated systems test jig, which has the target hardware+software, and allow programmable input stimuli and output verification. You really should have such a setup for your final (per-board) QA anyway, so better to start developing it sooner rather than later.
For instance, on-device testing is a first-class citizen in the microcontroller programming framework I've developed. Shameless plug: http://www.jonnor.com/2017/02/data-driven-testing-with-fbp-s...
I recently ran into an issue where an interaction with the DMA controller and the SDRAM controller in a MCU lead to invalid reads from the SDRAM. I still don't know what is actually going on with the MCU (SAME70, in this case).
Using a HAL lets you debug your application code more easily, but I don't think I've ever had a hard problem in an embedded system where the underlying cause wasn't related to some odd hardware behaviour under a corner case stimulus. There's no way to simulate that without a full model of the entire device (which isn't available).
The virtualization helps you ensure that A is doing what you intend to do, but not as much with B) knowing what your intent should actually be... and even if you can build a reasonable model of 80% of the behaviour, the really critical stuff tends to be out of the envelope type constraints like, oh the sensor gives this bogus reading under conditions X, so you need to ignore it or avoid condition X.
Edit: case in point (an adjacent story in HN)
https://news.ycombinator.com/item?id=13700798 "UPS Showcases New Delivery Drone, Fucks Up"
Debugging a quality control machine I inherited:
The heat sensor would trigger randomly, about twice a week.
It was a software problem, as swapping out for a brand new one didn't fix anything, despite doing it three times.
Everything seemed above board, and the review didn't take that long, it was only 8000LOC, simple C.
I found the answer on a test device: The vendored library that we could link against, but not look at, was reading the temperature from a sensor over I2C... And then reading it into the main onboard chip via an analog pinout. The docs said digital.
Which meant, when an inspector walked passed the machine, and looked at it, their own electromagnetic presence would mess with the readout. It wasn't sealed properly against this, because we believed the documentation.