The most important code I ever wrote was probably the control system for a submersible robot we developed in grad school. It was important not so much from a money standpoint, but because a whole group of graduate students got their Masters' and Doctoral degrees using it, it was sort of my bequest to humanity. And boy was it a hack...
I had decided to use a new single board computer on the robot that used a compact PCI bus, which at the time was a brand-new standard. It was very expensive - $25,000 - which was a whole lot of money for a university lab, but the computer had specs that we just couldn't beat with other existing single-board computers at that time.
There were no available compact PCI motor controller boards, so we had to use a motor controller board that was built for a different bus standard, and then convert from the Compact PCI system to the other board using a bridge chip. The particular motor controller board we chose was based on an 8-bit motor controller IC, the LM629. This particular chip uses memory-mapped 8-bit registers, and in order to communicate with it you have to write and read the registers in a very specific order. If you do anything in the wrong order, or you try to write to a read-only register, or vice-versa, the chip generates an error.
I was a decent low—level C programmer at that time, and was able to crank out the code in two days. But it didn't work. Whenever we tried to communicate with the chip, it threw an error. I went over the code with a fine-toothed comb, and I was absolutely certain it was all correct. I had no idea what was wrong. I was looking pretty bad to my advisor; I was the C stud, and I couldn't even write this simple device driver. And worse, I had recommended that we use this particular computer system, which cost $25,000, far more expensive than any other SBC we had ever bought, and now I couldn't make the thing work.
Finally, after banging my head against it for a week and making no progress, we threw up our hands and asked the motor controller board vendor if we could bring our system to their facility and get their help debugging it.
We arrived at the vendor and set up. Their programmer checked my code, and he couldn't find anything wrong with it either. After two days the owner took pity on us and asked his best engineer, a digital logic expert, to help us. He carted in a $20,000 digital logic analyzer and hooked it up and had me run my code. What he discovered was that when I had issued an eight-bit read, the chip saw a 16-bit read, which it wasn't expecting, so it threw an error, because the high-order byte was getting read from a write-only register. But the code was clearly issuing an 8-bit read. So where was the 16-bit read coming from?
It turned out the bridge chip had a bug. When it saw an incoming 8-bit read request on one bus, it translated it into a 16-bit read on the other, then threw away the most significant byte. We called the manufacturer, and were told "that's known, documented behavior - it's clearly spelled out in the manual." And when we checked, sure enough, it was - it was mentioned on page 48 in the third footnote, in 8-point type.
The solution we eventually came up with was to cut all of the address bus lines on the motor controller board and shift them to the right by one, and then take the least significant bit line and connect it to most significant line on the address bus. That way, access requests to any odd 16-bit memory address would map into unmapped register space so the LM629 wouldn't see them. Then I rewrote the code to only use even memory addresses. Worked like a charm. But I still feel sorry for the grad students who had that robot after I graduated. There was no way they ever figured out what I had done. Or why.