Hacker News Reader

Comment by teuobk | Hacker News Reader

In the embedded world, sometimes you can't even be a puts/printf debuggerer. There have been some situations, typically early in development, where I've had to be a "turn the LED on or turn it off" debuggerer. Amazingly, that binary output is usually enough to fix whatever problem is happening and bootstrap into printf debugging or full ICE interactive debugging.

View on HN

One time I couldn't get to an LED so I used an IR thermometer pointed at chip to debug it.

Telling the difference between hot and cold and whether it was getting warmer or colder told me what I needed to know.

Impressive.

Saying as someone who once debugged something by ear making the motherboard emit a cathode ray tube like hum instead of blinking or beeping.

Nice! How'd you end up generating the hum? I assume you oscillated something, remember what it was?

I think it involved a NIC, but I don't remember what it was exactly that I was doing to it. It was over 10 years ago.

Back at an old job, I was working on a new embedded system. It took us about a month before we bothered setting up UART for printf debugging. Fortunately the CPU/devboard supported gdb out of the box. What we ended doing was having a global char pointer point to somewhere in the middle of the memoryspace, and we would log things by appending to that string. Then we could gdb in and print the contents of the string.

I have done things like modulating a GPIO pin or LED to return a byte or two and see it on a oscilloscope. One time I was debugging an embedded device from a customer for a temperature controller where periodically the readings would be out of range. Turned out the CPU IP had a bug with interrupt handling when a particular instruction was used and after confirming no other side effect in simulations, the solution was to avoid using that instruction in the firmware.

That sounds really interesting. How did you get from modulating a GPIO pin to figuring out that particular instruction was bad?

It is from a long time back so things are fuzzy now but here is my recollection. I wanted to determine the source of the corrupted temperature readings but every pin on the MCU were utilized so no easy way to read it out. I noticed one pin was used just for an activity LED so I modified that routine to pump out a serial bit pattern read from a variable. Then I instrumented various parts of the code related to the temperature measurement algorithm to see the earliest point where it was corrupted. I traced it to the interrupt routine that reads the temperature sensor. In investigating that code I modified it slightly which caused the problem to disappear. Experimenting showed that a particular instruction would be the source of the problem. At that point I consulted with the ASIC designer and the confirmed the bug and identified the scope of the issue. Also should add that various debugging hardware like ICE didn't work maybe due to timing differences.

Sounds like you were lucky to have that bad instruction disappear with your modification. Cool story.

Once upon a time in the olden days, if certain makes of PC couldn't successfully complete the Power On Self Test and couldn't initialise video output to report the problem, it would beep the fault code through the system speaker.

Most motherboards still do this.

Just not as often, I imagine.

Except, for some makes, it wouldn't get that far but at least that narrowed down the possibilities.

A Bus Pirate can sometimes save your bacon or at least make your life easier in these situations!

Just did that the other day trying to track down a hard fault at startup. Binary code on LEDs was all it took to zero in on the cause.

Oh I've done this too on my TI Launchpad projects. Supposedly there is a way to attach a debugger, but I haven't figured out how to do that yet. I turned the LED on and used longer and shorter "on" times to differentiate between 1's and 0's. It was fun, but the feedback loop was very slow.

Reply on news.ycombinator.com