The software fixed the hardware | Hacker News Reader

The software fixed the hardware

jwz.org

162 pointsJ3L240442 comments

This type of thing happens all the time with embedded platforms. Software fixes for hardware defects are common. When we were building robots invariably the electronics team would always wire some motor backwards. Since the motors had similar power in both directions they would always ask me to just write the code to run motors they wired the wrong way backwards.

It was simple enough to do, but it always feels a bit wrong.

Other types of software workarounds are possible though too. I once modified the acceleration curve of the robot (clamped down the jerk) so that there would be reduced stress on the poorly made drive train and under many failure modes this changed the robot from essentially dead in the water to workable.

It seems surprising how many issues you can solve with code if you aren't used to it. But these hacks are actually everywhere in a lot of embedded systems and allow teams to continue to quick, cheap and dirty work when needed that can get patched up later.

You describe them as workarounds, but often this the intended design. If you only have manual control, then perhaps a driveshaft must be made larger and heavier for the case of the ham-fisted user. If the transmission is under software control, the software can be careful about accelerations to avoid overstressing the driveshaft.

The point is that when you consider the system as "mechanical, electrical and software" you get a lot more flexibility than if you think of it as "mechanical and electrical" but we do some stuff in software.

Certainly I agree the system is more complex and we could have designed it to work this way. But the stark reality was that these components were supposed to be wired a certain way in our design and the drive system was supposed to stand up to a lot more torque, as per the intended design.

So I think it is fair to call these particular fixes workarounds.

Your point that it is silly to look at these systems as anything other than integrated (often quite deeply) is well taken however. The fact that one can fix a mechanical defect using the software section of the system speaks volumes about just how integrated they are.

it always feels a bit wrong

It helps to think of it as "don't care" code rather than "error compensation" code. Which way should the engine be wired? It's cool; the software doesn't care. Just wire it up however. What order should I start these services? It's cool; they don't care. Just start them up however. As soon as your users stop caring about the stuff you can take care of, they'll have time to care about something else that's more important.

And in this case, don't forget to do a #define INVERSE_MOTOR_POLARITY 1 or similar, so that the hack is clear and doesn't bite you when porting to another platform.

It's not a bug, it's a customizable feature :)

You're in trouble though when the front motors are wired one way and the rear ones another, or one particular motor is miswired. Imagine #define FRONT_MID_SECTION_MINI_MOTOR_POLARITY (...)

Typically more like:

  #DEFINE MOTOR_0_WIRED_WRONG_BY_#$@#$_ELECTRICAL_TEAM
  #DEFINE MOTOR_1_WIRED_WRONG_BY_#$@#$_ELECTRICAL_TEAM
  ...

That said, the notion of portable embedded code when we're talking about the types of systems where these types of hacks thrive is pretty funny. These are not the type of routines people port or reuse and most of these hacks are very specific behavioral stuff to match specific pieces of hardware.

Hopefully you're not putting in hacks in the parts of the system you might actually reuse that's supposed to be more independent!

Code re-use is a principle that's often struggled outside of embedded systems. Inside embedded systems code it's downright scarce.

I think it's getting more common in the hobby community though. Just look at all the code and modules ready to use on AVRs, Stamps, etc.

It's only a shame that a lot of it is fairly unreliable C code. The situation for embedded C reminds me of PHP. You can find snippets for anything, but most of them won't be written by experienced library authors and can have an 'interesting' behaviour...

In my previous job we had several KLUDGE_TABLES as a way for our DBA to remind people that they were asking him to add a dirty hack that they promised to remove once the 'crisis' was gone. Five years later, they were still in there. At least the person querying it still had to type 'SELECT * FROM KLUDGE_…'

Well, the values actually sent out to the PWM were backwards. Thankfully I could just invert it at the last step in the code where it got sent and so while writing the actual code I could pretend everything was correct. It's hard to think of -127 going forward and 127 going backwards as anything but a dirty hack.

While the underlying electronics thankfully didn't care, the digitized values set up certain expectations.

Typically I get around this by having each motor's direction defined separately. Since you very often need to define the logic of a sensor ('1' can mean either ON or OFF depending on the sensor type) on a sensor-by-sensor basis, it makes sense to do this for motors as well.

It can get a bit mad sometimes though. The main issue is that if a hardware fault is found it is sometimes too late to rectify without a massive cost.

My management was recently trying to convince me to rewrite a FAT file system in high level code due to issues with power management, corruption and the disk we store our most precious data on.

I work in the safety critical/life-saving realm of embedded systems, and it has definitely been an eye-opener to see just how many bugs are 'forgiven' just because there is a software layer on top to make things work as expected instead of as designed ..

Once our hardware team mounted the camera in a cell phone device upside down. Later we had to write code to rotate the image 180 degree to compensate for it.

I just recently wrote some code to do pcb milling with pretty close tolerances. The actual milling table wasn't straight however, so when the substrate was clamped down it would have Z variations 3x larger than the desired cutting depth. A 5 minute touch probe run and a simple 2D compensation table removed the need for a time consuming tramming operations and expensive clamping hardware.

This wasn't so much fixing as augmenting. Software has the unique benefit that it can replicated essentially free while the physical improvements need to be administered separately in each and every instance.

Re. the backwards wired motors. Storing a config table in EEPROM seems to be a simple and portable solution to this. Do it in the logical way until you hit the output pins.

Do it in the logical way until you hit the output pins.

Ding, ding, ding!!! We have a winner. This is exactly correct: write your code the way everything should be in a perfect world, BUT have a hardware abstraction layer to separate that from reality. The upper layer code should just know that when the end sensor is reached, the hydraulic ram should be reversed. The HAL knows what state the end sensor must be in when it's triggered, and what polarity to send to the valve controlling the hydraulic ram.

Even in small embedded systems, a properly tiered architecture can save a lot of trouble.

Don't worry, it works the other way as well.

At my last job, I had to fix software defects in hardware.

I think I'd like to file a hug report...

Meh! If I had a dime for every time that I was told to fix a hardware problem in software, or provide a "limp home" mode if software detected a hardware fault, well... I'd have a lot of dimes. It's pretty standard behavior for electromechanical embedded systems.

The really important take home here is that the manufacturer sent out a software fix to a consumer product that is not online! That's a huge leap forward and is something the industry has been pondering for a long time.

I had a similar problem with my Roomba. Recently it began to bump into phantom walls where there were none, to the point where it would get stuck in the middle of the floor, turning around and around, stuck in a jail of invisible walls.

The Roomba detects this problem and emits a failure code of 9 beeps. I contacted iRobot about the problem, and their response was to ship me a fresh Roomba (without the battery + charging station) right away. Here are some pictures of the battery transplant:

http://dl.dropbox.com/u/1149620/roombasurgery/index.html

From the troubleshooting instructions that iRobot gives you upon encountering the 9-beep code, I suspect that there's probably just a faulty or possibly dirty (likely very inexpensive) sensor on the front bumper of the robot. I wonder if there's a similar fix to jwz's Roomba where I could patch for the bad sensor somehow (the roomba appears to have more than one way to detect it's stopped moving), and perhaps accept a slightly "dumber" but otherwise working Roomba.

My roomba has this problem from time to time. It's because the front wheel accumulates goo and stops rotating. The sensor that detects the wheel rotation then thinks the wheel is not rotating, and the robot moves erratically to free itself from the wall that it thinks is in its way.

Clean out the goo, and the robot is fine again.

This is nothing new. Read about Voyager 2. Specifically its platform problems between Saturn and Uranus in 1981, about 3 decades ago.

Can they send one to Mars to fix the rover?

They can and have! I always thought this was very cool. However, the current Mars rover problem is not a hardware problem, per se, it's a stuck-in-the-mud problem.

An elderly robot that walks with a limp that's stuck in the mud problem.;-)

The bad wheel that forced them to drive backward was a hardware problem. They should have designed it with 2 axes of symmetry!

I just love the hacking that space hardware designers are allowed to do ..

"oh, its already bagged up and installed on the rocket, we've got 2 years to fix it until its going to suffer 45 seconds of insane conditions, so just let it slide for now .."

Also it should be noted that devices like FPGAs allow for directly altering certain hardware structures via software.

In these cases it can actually be possible to fix a hardware bug (or add a hardware feature) via software, where the hardware itself actually gets fixed (ie its not just firmware working around an ongoing hardware constraint).

So, does it actually "fix" the hardware by say, oscillating the wheel in such a way as to clean or jiggle the sensor, or does it update the firmware to compensate for the wheel sensor's spotty readings?

The entry itself speculates that the new firmware simply compensates for the spotty sensor(s).

Another example occurs on this thread: http://news.ycombinator.com/item?id=1032063

Here's the link: http://news.ycombinator.com/item?id=1032152

When I saw it was jwz being linked to, I was worried I would have to suffer through a green on black web page. Oh, the relief.

Why can't this be done with a download and USB?

Does the Roomba have a USB port? jwz mentioned a 'serial' port...

It does not have a USB port.

The serial port is also a non-standard (for serial ports) connector (it is a standard DIN connector). My guess is that the average Roomba owner doesn't have a PC with a serial port, and probably wouldn't want to acquire an uncommon connector and wire up a serial converter cable, and then run some program on their PC to dump the firmware image in to the Roomba. The osmo dongle makes the process near foolproof for soccer moms and grandmothers and so on.

Incidentally, there is evidence that the OSMO dongle collects data, also ..

Part of my question is why it doesn't have a USB port.

Cost. USB is a very complex protocol that requires a controller. For most embedded systems a serial port, which is very very cheap and dead simple to implement, is sufficient.

There's really no need for a USB port on the Roomba, so it's silly to incur additional costs on every single unit to add it.

I work in embedded. Primarily, my work involves fixing hardware bugs in software. This is not surprising to anyone who works in embedded, but for those of my programmer friends who just see the hardware and have no clue just how much software is under the hood, it always seems to be a big shock to them that I have had to spend a few weeks fixing a hardware design that had one pin wired around the wrong way, instead of just getting the hardware fixed.

That said, one of the things I really like to see these days is new generations of extremely competent hardware designers who have absolutely no problems with the expenses of hardware iterations .. even though it means less work for me in the end, its good to see. I have lost count of the times I've had to write some god-awful driver code that just makes me want to puke my lungs out, just because the hardware guys were too lazy to breadboard something ..

Roombas are no match for Golden Retrievers.

As pets maybe... but I've never seen a Golden Retriever clean a floor.

Wrong breed. I've got a food-motivated wiener dog that'll destroy any roomba in a floor cleaning contest.

Reply on news.ycombinator.com