Extreme debugging - a tale of microcode and an oven
alanwinfield.blogspot.com.br
alanwinfield.blogspot.com.br
Fast forward ten years and I'm doing contract work for a local company. Of course the contractor got the shittiest workstation, which dies every afternoon like clockwork. So I move the POS white box away from the window and out of the sunlight. "Magically" it only crashes once week. So I send a ticket into the queue, stating the obvious, then the IT folks quibbled and finally admit the motherboard needs to be replaced.
That's the time one must apply heuristics such as (A) expand one's domain knowledge through research or talking to others, or (B) expand one's observations on the subject, or (C) expand one's hypotheses into the realm of "Oh it's just not possible that...", or "Let's try something that looks really 'stupid'...".
Of course, heuristic (C), being presumption-checking in face-saving disguise, has been the most fruitful heuristic in all my years. Those successes often become the best "teaching moments".
But here's a success story that followed heuristic (B). It goes back to the early days of the MITS Altair 8800[1] and its "S-100 bus"[2], a physically large passive backplane bus for the 8080 CPU. My employer sold complete systems comprised of that hardware with general purpose business software we wrote (using 1979-and-later versions of Bill Gates' first commercial product [3]).
We had systems that would intermittently "cold-crash" with no observed relationship to anything "rational".
So, observe: Pull the boards out and inspect them for bent pins on chips, cold solder joints... nothing. OK look at the backplane connectors, topside: No dust or bent pins. Finally turn the emptied chassis upside-down and look at the 100 solder connections for each of the 18 or so card slots. Get out a magnifying glass because this will take some time...
...and there they were: Three "cakey" looking broken solder joints on the bus. Evidently, the backplane board flexed enough (or the S-100 connector pins themselves ccommunicated enough flexure to their solder joints) upon board insertion and removal to break a weak solder joint on the bottom of the big PC board that was the backplane bus. Resolder all 1,800 pins and promlem solved for the lifetime of the machine. Found the same failure mode in a couple other boxes in the time 7 years I worked there, otherwise maintaining software.
[1] https://en.wikipedia.org/wiki/Altair_8800 [2] https://en.wikipedia.org/wiki/IEEE-696 [3] https://en.wikipedia.org/wiki/Altair_BASIC
There obviously is time to pursue further domain knowledge of how these batteries fail in the field, but pressures of the market are now in full force.
Creeper world will actually use a stack based language like forth for it's next release.
(specifically, it will use a derivative called crpl for scripting)
This is not a "in the days of old, when men were men and women were men, too" kind of thing. Most of us will, probably, spend the next 20 years doing the web/mobile stuff.
I enjoy working on hardware and RF projects as a hobby. Not much fun doing it at a large corporation anymore....
I haven't used it yet but we have some really cool (ha!) equipment that will adjust the temperature from -20 to +60 (Celsius) in minutes.
I really hate going in those things when they're testing at high temp and high humidity, especially when I'm dressed for winter weather.
Combine RF + Temperature + Antennas - Then you are in for several months of interesting debugging - particularly if you want to optimize for performance at low power transmission over a wide variety (-55 to +85 Celsius) of temperatures. The math gets hinky really quickly.
After two sets of embedded engineers trying to solve the problem with software updates, our RF guru got involved.
He traced the problem to the RF transceiver losing PLL lock at high temperatures and the transceiver not setting the PLL status bit correctly. We had to fix our uC to reset the RF chip every X minutes based on worst-case temperature rise scenarios.
No one else would have figured this out for months. The bug had not been fixed through three generations of this transceiver and the guru had dealt with the issue years ago.