Talking about print debugging has reminded me of a time I spent two weeks using a sideways form of print debugging to track down a timing bug. It was on an embedded system, and the bug when tripped would take out the serial communications line: at which point I couldn't get any diagnostics, not even print statements!

An in-circuit emulator was unavailable, so stepping through with a debugger was also not an option.

I ended up figuring out a way to be able to poke values into a few unused registers in an ancillary board within the system, where I could then read the values via the debug port on that board.

So I would figure out what parts of the serial comms code I wanted to test and insert calls that would increment register addresses on the ancillary board. I would compile the code onto a pair of floppy disks, load up the main CPU boards and spend between five and ninety minutes triggering redundancy changeovers until one of the serial ports shat itself.

After which I would probe the registers of the corresponding ancillary board to see which register locations were still incrementing and which were not, telling me which parts of the code were still being passed through. Study the code, make theories, add potential fixes, remove register increments and put in new ones, rinse and repeat for two weeks.