My Hardest Bug Ever

The project is going swimingly well. Smoother than normal, actually. The calm before the storm. Suddenly, abruptly, your device fails. Maybe you were there, maybe someone else reported it; neither makes it less mysterious. You brush it off with more important things to do. It happens again. Now it bothers you; top priority. You try for hours to make it happen consistently. No amount of tin foil, bunny ears, or interpretive dance will make it happen again. So you give up, you start steel plating the code. Paranoid error checking everywhere, setting variables twice, resetting, extra padding on all the timing parameters. PARANOIA. You code begins to look like a lunatic wrote it. But the bug never happens again. Your paranoia and double guessing fixed the problem.

The Moment! From then on, whenever something doesn't quite _feel_ right with your projects, you start inserting strange, almost ritualistic incantations in your code. If anyone asks why you decided to sleep for 14.23ms instead of the prescribed 10ms, you just call them a devil and run away back to your ancient cross compiling GCC and make a sacrifice to the Upstream Gods.

I feel for the author. From that Moment on, his code will never be the same. His sanity, now left on the cutting room floor along with the trust he once had in Datasheets and Programming Manuals.

Embedded 4 Life.

I dunno, as an embedded programmer working on new HW and sometimes with a custom compiler, blaming the HW became the 5th thing, and the compiler the 6th...

Fractal problems are those that look really simple from 100,000 feet up (like a dot) and reach unimaginable complexity as you get closer (like a fractal). There are a lot of problems of this nature (like email, transportation, ecommerce, taxation, etc.) and solving them is worth a lot of money but requires a LOT of work.

http://www.quora.com/Programming-Interviews/Whats-the-hardes...

Another fun bug was when alpha version of CLR failed to restore one of the two registers used to control loop execution on the Itanium. (Yes, Ia64 had two registers - one for loop variable as seen by the program and one to actually control the loop execution).

https://bugzilla.mozilla.org/show_bug.cgi?id=772330#c21

"He called me and, in his broken English and my (extremely) broken Japanese, we argued. I finally said, "just let me send you a 30-line test program that makes it happen when you wiggle the controller." He relented. This would be a waste of time, he assured me, and he was extremely busy with a new project, but he would oblige because we were a very important developer for Sony. I cleaned up my little test program and sent it over."

Really? Being humble doesn't hurt.

At the same time I love when a smug face melts with a concrete proof and "I told you so". Save face and be a little more humble.

Technically, all bugs are caused by quantum mechanics.

I've diagnosed temperature problems (grab ice from the freezer, apply to chip, see it work...), clocking problems (you insert strategic delays, sometimes on the order of thousands of instructions) and just badly documented registers (make sure mystery bit number 13 gets toggled just right, or it's curtains).

It's fun stuff.

Nowadays I'm working on these big distributed clusters, far from the bare metal. But you know, just now I rather miss those days on the little embedded systems.

Basically the bug was in the Nintendo GameBoy, where it only happens if you press two buttons (left and right, or up and down) at the same time. Now you can't do that normally, since the controller won't allow it. But if you are hard-core QA (as the guy which this producer told me about it) - he ripped the controller, and manually wired some stuff - so he'll get LEFT and RIGHT pressed down at the same time, and only then the game would crash...

But then as we say sometimes here, for such things - it might get the... WNF (Will-not-fix).

    i=5;
    i=5;
    i=5;
    i=5;

As an embedded software engineer, you need to understand hardware nearly as much as the software. And depending on how far along the hardware is (pre 'gerber out', or deployed in field) it's usually up to software to "patch it over," "hide the issue," "fix it in software," if possible...

Coming from the other side of the fence, a place where we develop the hardware first and then bring it to life through software things are very different.

The first step in bringing up a not-trivial board is to go through a full characterization phase. This is where electrical, mechanical and short software tests are performed in order to determine if the hardware is operating according to requirements. Depending on the nature of the hardware this period can last months and require many re-spins (iterations where something is fixed or modified).

While this is taking place, and depending on the nature of the team, the application software is probably starting to be assembled on prototype hardware. In some cases this can't happen until you have actual hardware that works reasonably close to specs. Perhaps rev-1 hardware is used to jump-start software development while the hardware team goes through many revs in order to make adjustments and fix problems.

Seemingly weird hardware problems abound. I have been in situations where the signal is good at one end of a trace or cable and not so good at the other end. In the case of high speed design this can easily happen if there are problems with the design of the transmission lines carrying the signals. You can easily end-up with reflections that will wreck havoc on the signals as the go down the transmission line.

Another "weird" hardware issue in high speed design are signals that don't arrive at the destination within a specific timing window. Dynamic ram designs are one example of this. A clock is used to gate various signals at both ends of the transaction. Everything is sampled relative to this clock. If some signals, for example, control signals, arrive before, after or staggered with respect to their acceptance window you can have really weird effects.

With large FPGA designs you can have issues related to faulty design of the power distribution system. Power and signal integrity are major fields of study and truly necessary parts of modern electronics design. Traces on a board are like capacitors that need to be charged and discharged. If you have 200 traces switching from 0 to 1 simultaneously a lot of current will be require of the power system within nanoseconds (or picoseconds). If the power distribution system on the board isn't designed to deal with such transients you end-up with all manner of weird effects. For example, transmission lines might be perfect in impedance, crosstalk and time of flight yet signals arrive with lots of jitter and all over the place in terms of timing. The power distribution system on a board is like your heart, if it can't deal with demand you are not going to go from sitting to standing and then running without major problems.

This is only the tip of the iceberg. I could go on for pages and probably write a book about this. I've made enough mistakes.

And so, from the vantage point of a software engineer who also happens to be a hardware engineer blaming the hardware almost always comes first until the hardware is proven to be operating according to requirements.

In terms of the playstation issue on the original post. Well, from my perspective this is simply bad engineering on the part of those who designed the hardware. OK, this isn't the engine control computer on a Toyota. The sentiment is the same. Fault tolerant design is important, even for toys. Think consumer drones.

Buffer size on line printer out of sync with what the program (COBOL) was sending and would produce variable results every so often when WRITE BEFORE and WRITE AFTER got used.

Then the ones which work fine with debug libraries and not on production ones as turns out the debug libraries had a unintended fix to a bug that was yet to be known. Also had programs work fine when you don't use debug libraries.

Then we have driver bugs, when if you have even touched graphics will of encountered, more so when we have a shift in driver model as ME introduced and was stable enough for windows XP, same with Vista which is stable in 7 and 8.

Surfice too say the hardest bugs almost always turn out to be hardware or some external software being documented wrong or undocumented features. Also new revisions in hardware can casue the smallest of issues in the niches of uses, but it can be you and it is a small lonely world when you try to fix one of those.

Though for me the hardest bugs are always the ones which you know were abouts they are, you are just unable to prove it too those enable to investigate it too the level of being able to prove it is there issue.

Also some bugs for some are easier for others and we have all had a bit of code or aspect of life that we seem blinkered from seeing what is wrong; Somebody else could look at it and solve it in seconds. Then we have also been that somebody looking at others code or issue and seen the problem and solution. Sometimes we see the problem before they can see it as a problem. Had classic in the 80's doing a mailing list update to add on postal codes (same as USA ZIP codes only UK flavour). I advised that some address's will already have a postal code tagged onto the address lines and in some cases could end up with not only 2 post codes on the address but also not always the same as the one entered may of been slightly wrong. This was dismissed until 5pm friday on prepping to head 300miles home south for the weekend. Was late heading home that weekend. Though that would be an annoying bug more than a hard bug, however hard it was upon my weekend. But we learn how to highlight issues better, or overhighlight. I have learned much from weather warnings at least and how they lean more towards the worst case situation thesedays more than say in the 80's.

Lastly though we have those intermitant bugs, so rare and niche that the impact is so neglagable and negatable that they are easier to work around instead of trying too fix. Mostly such fix's would cost more to identify fully and address than any work around. I suspect we have encountered more of those than you realise. With that, is how hacking was born after all. Without bugs would we of had the hacker history gave us - that in itself is worth a thought.