Comment by bjt2n3904 | Hacker News Reader

Comment by bjt2n3904 | Hacker News Reader

Parent Full thread

Easier said than done. This isn't the web world, things aren't so simple.

How do I mock a temperamental motor controller responding to AT commands? I mean, I'm sure it's hard enough to replicate the oddities of their serial port parser, but the temperamental part is gonna be tricky. This is an embedded system I'm talking to, which is parsing bytes coming over a serial port in an interrupt service routine at 16 MHz with 4 kB of RAM. Not an async callback with JSON in a nodejs app with GHz of processing power and 8 GB of memory. Emphasis on: temperamental.

Oh, and don't forget. This is a mechanical system, not a website. We'll have to put torque and strain gauges on the system to make sure it's doing what we want.

The skinny is, the system my python scripts talk to is far too complicated to just "stub out". You may be able to test 90% of the cases, but what you really worried about is that 10% of freak accidents that break things. Look up the Therac 25 and how that failed.

I work primarily in embedded, so I feel you :)

One can separate the I/O code from the application logic, and do data-driven testing of the application logic. Use recorded data from real-life systems to build a (partial) model of the system, at least enough to test individual interactions or short sequences of interactions. Capture this data continiously in production systems, so that when an incident occurs it can be cut-and-pasted to form a new testcase. The latter is critical to build up a library of regression tests for those nasty things one had to find the hard way - so that at least it only happens once.

With this basis you can optionally go to hardware-in-the-loop testing, where some of the same testcases run with real hardware instead of fake responses. This should then be instrumented with neccesary sensors/code to be able to observe its behavior. This ensures that your mock and system stays in sync, and increases coverage for non-deterministic cases. At least one such system should be hooked up to your CI server to test every revision of the software, to catch issues early.

When building the next product, consider shipping the instrumentation neccesary for QA into all products. That way the testing system is closer to real system, and one can integrate self-checking, failsafes and error reporting in production systems. This can be used by final QA (before shipping), by engineers debugging on-site and as part of periodic (or continious) health checks on the equipment.

I've done such things for electromechanical systems, though nothing with a potentially-dangerous motor yet.

Fortunately, on the web, things never go wrong. Hardware never fails. Traffic never spikes. You never face DDOS attacks or people actively trying to break your security. The network is never unpredictable. It isn’t a complex distributed system consisting of hundreds of interconnected parts. Every potential failure can be anticipated and easily tested against. And every component scales effortlessly as the system scales by orders of magnitude. A lowly web developers could never understand the Herculean feat of writing a Python script to communicate with a motor controller over a serial port.

Are you afraid of network security attacks to your embedded hardware? If not, then you probably don't care about missing security updates. What are you concerned about with python2.7 no longer being supported? Like, are you worried about getting python2.7 installed on a new piece of hardware five years down the road? I understand you dislike the tone from the python3 devs, but I don't know what you would like instead.

Reply on news.ycombinator.com