Comment by bjt2n3904 | Hacker News Reader

Comment by bjt2n3904 | Hacker News Reader

Parent Full thread

Hello from the embedded world!

The web world indeed can rapidly change their entire stack from LAMP to MEAN to Dockerized Q-Basic CGI scripts and load balance their Twitter clone. There's challenges for sure, but pushing updates is simple and quick, and you can always throw more cloud if you run into walls. If your complete platform rewrite fails, just roll the load balancer back over and try again tomorrow.

It's a lot more difficult when your Python application is communicating with a temperamental motor controller over an ancient RS-232 link with AT commands.

There are only ten of these devices in the world, and three in deployment. They perform critical functions, and they need to run 24/7 without failure or human interaction. (Did I mention these devices are remote, and it takes an afternoon to go out and investigate a failure?)

Every single failure results in a few lines of change in the code base. Those lines cost quite a bit -- both time, money, and loss. It's quite stable now, but that stability came at a price.

But haha, what am I talking about? Geez, I've had ten years to move off this "dead" platform. Let's run it through 2to3, or perhaps try a port to nodejs and deploy it tomorrow.

I’m not sure what you’re complaining about here. Those applications will still work.

The attitude of most python3 devs, which can be surmised as: "I moved my Twitter clone from python2 to python3, and it was easy. What's the big problem? You've had ten years to get this done! Hurry up, you're hurting my productivity."

Well if you’re not happy about the EoL part then that’s a fair response.

If you don’t care (because your apps will still work and they don’t need or can’t be updated) then it shouldn’t really come up...?

Just curious, what parts of your code are not portable to Python 3? The biggest barrier usually seems to involve strings/Unicode. Or are you relying on packages that don’t have Python 3 support?

In any case, it sounds like you have a stable Python 2.7 system running, so what’s the issue? Nobody’s going to come and take your code away from you.

So keep using Python 2.7 on some previous version of RHEL? There's still banks using COBOL, man. We don't insist that RHEL ships IBM COBOL with it.

But... would you not use tools that have a guarantee of long term support for such a use case?

Automated tests is extremely beneficial for such cases. You should know that the code works before you deploy.

Easier said than done. This isn't the web world, things aren't so simple.

How do I mock a temperamental motor controller responding to AT commands? I mean, I'm sure it's hard enough to replicate the oddities of their serial port parser, but the temperamental part is gonna be tricky. This is an embedded system I'm talking to, which is parsing bytes coming over a serial port in an interrupt service routine at 16 MHz with 4 kB of RAM. Not an async callback with JSON in a nodejs app with GHz of processing power and 8 GB of memory. Emphasis on: temperamental.

Oh, and don't forget. This is a mechanical system, not a website. We'll have to put torque and strain gauges on the system to make sure it's doing what we want.

The skinny is, the system my python scripts talk to is far too complicated to just "stub out". You may be able to test 90% of the cases, but what you really worried about is that 10% of freak accidents that break things. Look up the Therac 25 and how that failed.

I work primarily in embedded, so I feel you :)

One can separate the I/O code from the application logic, and do data-driven testing of the application logic. Use recorded data from real-life systems to build a (partial) model of the system, at least enough to test individual interactions or short sequences of interactions. Capture this data continiously in production systems, so that when an incident occurs it can be cut-and-pasted to form a new testcase. The latter is critical to build up a library of regression tests for those nasty things one had to find the hard way - so that at least it only happens once.

With this basis you can optionally go to hardware-in-the-loop testing, where some of the same testcases run with real hardware instead of fake responses. This should then be instrumented with neccesary sensors/code to be able to observe its behavior. This ensures that your mock and system stays in sync, and increases coverage for non-deterministic cases. At least one such system should be hooked up to your CI server to test every revision of the software, to catch issues early.

When building the next product, consider shipping the instrumentation neccesary for QA into all products. That way the testing system is closer to real system, and one can integrate self-checking, failsafes and error reporting in production systems. This can be used by final QA (before shipping), by engineers debugging on-site and as part of periodic (or continious) health checks on the equipment.

I've done such things for electromechanical systems, though nothing with a potentially-dangerous motor yet.

Are you afraid of network security attacks to your embedded hardware? If not, then you probably don't care about missing security updates. What are you concerned about with python2.7 no longer being supported? Like, are you worried about getting python2.7 installed on a new piece of hardware five years down the road? I understand you dislike the tone from the python3 devs, but I don't know what you would like instead.

Fortunately, on the web, things never go wrong. Hardware never fails. Traffic never spikes. You never face DDOS attacks or people actively trying to break your security. The network is never unpredictable. It isn’t a complex distributed system consisting of hundreds of interconnected parts. Every potential failure can be anticipated and easily tested against. And every component scales effortlessly as the system scales by orders of magnitude. A lowly web developers could never understand the Herculean feat of writing a Python script to communicate with a motor controller over a serial port.

Reply on news.ycombinator.com