Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

No, it's not acceptable to power cycle to fix problems.

Yesterday, I had to power-cycle a CNC milling machine. The machine (a Tormach 1100) started the spindle with the spindle RPM about a tenth of the value specified. This resulted in pushing an end mill into the workpiece (it wasn't spinning fast enough to cut properly), snapping off the end mill (like a drill bit) which went flying off, and damaging the workpiece. This is not a huge deal (it's about $25 in damage, and everybody wears safety goggles around milling machines) but it shouldn't happen. We discovered that all spindle speeds were far slower than they should have been; entering a new spindle speed did cause the spindle drive to change speed, but the speeds were about an order of magnitude low. The spindle speed control is digital and computer-controlled.

Power-cycling the machine (which runs Windows Embedded Standard, a Windows XP derivative) cleared the problem, and I was able to run several jobs successfully. But someone else had reported a spindle overspeed on Friday. Something has gone badly wrong in spindle speed control on that machine.

The "Recommended Best Practices" for this machine tool actually says "Reboot controller once a day to force both Mach3 and Windows to restart."[1] That's not good.

[1] http://www.tormach.com/uploads/883/SB0036_Mach3_Best_Practic... (a PDF file with the wrong suffix)

I would state this a different way. It's not an acceptable solution for the developer/designer/manufacturer to propose. However, for troubleshooting, it's the single best tool we have (and with all of the black boxes, the only tool in many cases).

Ouch!

I recall being told about a high end cnc machine that power cycleled and moved the tool to home straight through the work piece.

You are right. Not acceptable to have end-users power cycle.

That doesn't mean that individual components can't be power cycled on fault without affecting the rest of the system. That, of course, means writing the system in a certain way that allows that (allows fault isolation). You can do it with Erlang, OS processes, containers, separate hosts, etc.

In other words there is nothing that prevent the company to auto-power-cycle some component that crashed. Report the error and then issue a fix later, while you still get to use the machine.

Hi! Blog post author here. My initial draft had this at the bottom:

"Disclaimer: If you have a recurring problem, power cycling will not fix your root cause. "

At some point I must have decided the whole thing was too long and that sentence didn't make the cut.

I personally would not go as far as to say:"it's not acceptable to power cycle to fix problems".

I see it more like:"not all problems can be fixed with power cycling".

The PDF with the wrong suffix crashed Google Chrome Mac. But then I power cycled (reopened) the browser and now it downloads fine :)

If you're using Windows to run a milling machine, you're doing it wrong. It's not Windows problem that you're using it inappropriately. That's like taking a 1990 Ford Taurus with 250k on it to a track day and then complaining that the engine blew up. Technically the engine did fail, but the driver should have known better.

Windows Embedded is quite common in factories where I live. What alternatives would you suggest that are vendor supported?

If your vendor thinks that Windows is an acceptable control for a CNC milling machine, consider another vendor. Yes vendors might choose to use Windows, and think that it's OK. But just because a vendor chooses to use Windows doesn't somehow make you obligated to buy it and suffer, does it?

Sure they might be the only vendor, or someone inside your organization might choose them anyhow despite your protestations. But that doesn't change the fact that using Windows for these kinds of things is foolish.

Just because big companies do it doesn't make it smart, does it? I mean, if it did, then startups wouldn't be able to exist would they? Startups are able to be a thing because big companies sometimes do stupid stuff.

They all use Windows. Mine use Windows NT4 and Windows XP. That's the industry. If you want industrial CNC machines you don't get a choice.

There's a tremendous difference between using Windows to run the GUI and using Windows to do the real-time control of the servos and various other hardware.

Most of the time these machines run a small microprocessor which receives commands over some port and interprets them as it is instructed to. This is what's called a "hard realtime" system as it always responds within a certain amount of time, guaranteed, provably as per the design.

What Tormach is doing is eliminating the dedicated gcode interpreter hardware/controller and performing those operations strictly in software, on a program running on a PC. There's some utility to that, but pretending that it's as good as having a dedicated, realtime gcode interpreter is not honest.

"force both Mach3 and Windows to restart."

I would guess Windows is used for the higher-level functionality (GUI, possibly format conversions), while Mach3 does the lower-level stuff that requires precise timing.

Adding a second OS makes sense as it makes it way easier to keep the real-time stuff real-time.

Using XP embedded for the top layer shouldn't be that big of a risk. It may have lots of known exploits, but you can remove lots of the attack surface, and an alternative GUI may not have seen much security auditing.

This is just ideology.

I'd argue it's actually more about practicality and reliability. Windows isn't particularly well-suited for hard-real-time situations (like, presumably, a CNC milling machine). This is an environment that's usually better suited to RTLinux, VxWorks, etc. or some one-off dedicated program written in something like Ada.

Reply on news.ycombinator.com