DIY electronic leadscrew for metalworking lathe [video] | Hacker News Reader

DIY electronic leadscrew for metalworking lathe [video]

youtube.com

87 pointsOJFord51 comments

Yep, he's right about the timing issues. I've taken a deep dive into LinuxCNC recently, as I'm using it as the motion controller for my custom servo-driven 3dp/Mill, and one of the most important decisions one makes up front is what Linux kernel to run. The real-time kernels are all but necessary for doing heavy number crunching and encoder reading, and that's even on a powerful i5-powered desktop machine with custom IO hardware.

As you start to climb into the multiple-thousand dollar arena on a project that was planned with just a couple, you really start to realize why the manufacturing industry routinely pays significant fractions of a $MM for their machines.

Isn't it more that real time kernels are necessary period? The number crunching itself can't be that problematic...I just upgraded an old Anilam mill controller that ran 3 axes. It was running on a 386.

The new encoders on the servos are way better, but the replacement modern hardware is also way better.

I think in both cases (especially the upgrade), the event handlers and drivers are all done in separate, dedicated hardware

Well, you can run the simulation and such without it, but for doing any real work, yes they are necessary. That was kind of my point though: you could in theory put one of those kernels on an RPi or something (barring compatibility issues with the chipset), but you'd probably still be lacking the proper IO hardware, etc.

Absolutely agree. Modern operating systems are utter beasts when it comes to complexity and are far less predictable than simpler solutions. There are reasons microcontrollers are still everywhere.

Linuxcnc on a raspberry pi 4 is fast enough to run the motion-control loop of linuxcnc with 1kHz, worst case latency is below 100us with current rt-preempt kernels.

Generation of stepper signals or interfacing with servo amplifiers and encoders is best left to an external microcontroller of FPGA card, connected either via SPI or ethernet.

And yet the Beaglebone Black has been able to handle this down at the 100-200nS level in the PRUs. For years.

While I admire the ingenuity of people doing real-time with Linux, I just shake my head at the man hours wasted that they could be doing the "thing they want to do" instead of fighting Linux.

And the Beaglebone runs what OS? Linux?

The PRUs are just the replacement for the FPGA resp. microcontroller. You have your relatively relaxed control loop running at e.g. 1kHz on the main ARM and the tightly constrained stuff on the PRUs, not much difference to running it on something external connected via SPI. Some kind of external hardware is needed anyways, doesn't make much difference including a STM32.

The problem with beaglebone is lacking GPU afaik.

Beagles have a GPU, not powerful but there. Just no useable drivers w/o TI nda's AFAIK. Also Pocket Beagles cost much less than BBB's.

"Lacking" in the sense not capable enough to do complex toolpath backplots.

Maybe the thing they want to do IS fighting Linux?

How do the audio and video subsystems of standard Linux systems deal with real time requirements?

Using the RT Preempt kernel patch, if you care about latency.

Buffers.

Badly.

Only macOS has an architecture to deal with low-latency audio and video properly. Presumably iOS has some of those features as well, but it was on OS X that Apple did some deep architectural changes for proper low-latency audio and video.

It was admittedly several years ago, but I worked on a voice chat app for iOS and Android, and the behaviour on Android was horrifying. iOS had very consistent and great audio latency (approx 10ms if I recall). Android devices were all over the place; I think the worst one I tested was over 150ms.

It's interesting where he's pointing out the real-time debugging support in the chosen board.

One "weird" thing that popped out about that for me, is where he said it's not possible to do that with the "other platforms" he's shown.

Does anyone know if the debugging style he's mentioning is different from attaching a segger to the debug port on the Due? Asking because I've done exactly that before, when working through bugs in the g2core (multi-axis CNC controller) firmware.

To me, being fairly newbie level with it, it sounds like the same kind of thing. Perhaps the author just isn't aware some Arduino's do have a debugging port + commonly available tooling to use it?

I agree other ecosystems have similar debugging capabilities; I think the main driver for hardware selection was deterministic timing and ability to keep up.

See my comment further down beginning "Starting about 7:09 he talked about why the Arduino wasn't a good fit..."

Not sure I agree with the bit about Linux + Raspberry Pi not being able to handle precision timing down to the microsecond level. I'd be interested if anyone has thoughts.

I've done a fair amount of timing-critical design, including control of stepping motors and encoders for specialized uses. My experience has been that "real time" programming on a mainstream computer with an operating system is a headache, if it's possible at all. It's just a lot easier to do that kind of stuff on a mid range microcontroller, where the processor has exactly one task and predictable sequencing of operations.

Granted, it was a long time ago, but I remember trying to work with a CNC milling machine that generated all of the timing on a Windows computer. Everything worked fine unless you touched the mouse, which would cause motion to stutter. So, there can be a lot of gotchas that have to be accounted for when using a "big" computer for "little" things.

Yes. This is just a "keep motor A in sync with encoder B" problem. That should be on something at the Arduino level, although you might need a faster CPU version. Something with no caches, no superscalar, just dumb static memory and fixed-time instruction execution. Linux would just get in the way. The loop that keeps both sides in sync should probably be running around 1000Hz or so. In one of the OP's rather long and numerous videos, he mentions 10Hz not being enough. Right.

If it needs a user interface, that might be on a separate computer, with an interconnect to set the ratio in the motor controller over I2C or something equally dumb.

Also, you can get stepper motors with encoders, so you can tell if you missed a step and correct. You don't have to use a servomotor, which seems to have complicated this project. Shopbots and Tormach mills use stepper motors with encoders.

Many people have converted existing milling machines to CNC, and there are kits for that. This is a much simpler project.

Starting about 7:09 he talked about why the Arduino wasn't a good fit and explained his choice of the TI board (including speed, single-cycle floating point hardware, responsive interrupts and dedicated peripherals to latch consistent, jitter-free reads of the quadrature encoder counts).

Note it's a 4096 step encoder, mounted on a different shaft than the one he's driving, that can run upwards of 2500rpm. His motor control loop (not the complete end-to-end, but the bit that maintains output position to that desired) takes under a microsecond. His code works with either a Nema 24 stepper or Nema 23 servo, and he demonstrated both.

In the second video he maths out some projections for error/drift (factoring in floating point representation error) and its ultimate impact on thread accuracy (which provides some motivation for what you might initially consider "overkill" hardware).

10hz is just the refresh rate for the tachometer calculation.

Indeed, a bit of confusion about "Arduino" is that it has come to mean two things. One is the original Arduino board and its "branded" successors. The other is the development stack. In the latter case, it's GCC with some libraries and a beginner friendly editor. It's bare-iron programming.

Thinking about dealing with an encoder, let's say it has 8k counts per revolution, and the motor can do 10 revolutions per second, which is pretty fast for a stepping motor. Then the encoder steps are coming in at an 80 kHz rate, but an 180 MHz MCU can execute a couple thousand instructions in that time period, so it's not even breathing hard.

I've been quite happy with the Teensy boards. The Teensy 3.6 has a 32 bit MCU with FPU, running at 180 MHz and executes code sequentially AFAIK. You program it in the Arduino IDE, but Arduino is now just a thin layer on top of GCC, and you have full access to the MCU registers and functionality. There are a number of other attractive 32 bit processors such as the STM32 family.

That's the sort of thing I had in mind. Low-level Arduino-type microcontroller programming on a bigger engine.

The Arduino development environment is just an IDE and a library for board-level systems. It's C++ underneath, and all of the C++ language, although not the libraries, is available to you. You can do the same thing without the Arduino system, but setting up the build environment tends to be complicated.

If you need precision timing, you either need to program bare-metal, or use some hard real time OS like VXworks or QNX. Here, where it's all stepping and reading encoders, bare metal is the way to go. If you were coordinating a multi-axis machine, a real-time OS might be better.

I've always wanted to mess around with the beagle black since the two PRUs are supposedly built to handle cases exactly like this.

>it was a long time ago, but I remember trying to work with a CNC milling machine that generated all of the timing on a Windows computer. Everything worked fine unless you touched the mouse, which would cause motion to stutter.

Windows used to (maybe still does, idk) use an interrupt for mouse movement. That was likely the cause of your issue.

This was a problem when recordable CDs were first available. The retail CD writers had really small buffers, and a contemporary PC could barely supply data fast enough. If you touched the mouse while recording the CD instantly turned into a coaster.

Tldw? What do you mean by timing?

Measuring time (down to nanoseconds) is easy. Hard realtime scheduling with microsecond precision... just forget about it, unless you're building a custom distro & custom kernel and have in-depth knowledge about all the drivers that are going to be running on the system (you almost definitely do not).

EDIT: I found a relatively recent socket benchmark with worst case scheduling latencies ranging from tens to hundreds of usec: https://www.codeblueprint.co.uk/2019/12/23/linux-preemption-...

You can't really bang bits like you can with carefully cycle counted code on something like an Arduino, but you can put all your bits into a good sized buffer and use DMA to push them out a stable, hardware clocked rate. You still have to be able ping pong your buffers fast enough and that isn't really guaranteed, but for large enough buffers can be made to work, mostly, probably, well… except when…

This technique doesn't work when you need to respond to inputs rapidly, but is fine for driving steppers. I see he appears to have gone with servos and encoders, so that would be a bit of a problem since there are inputs to be read.

> put all your bits into a good sized buffer and use DMA to push them out

Sure, time-honored technique for audio. Very on the mark.

But it doesn't really fix the "latency is forever" problem. Buffer too much and your control will lag. Too little and buffers can bottom-out. Best is to over-spec the metal and then be the only code on it so you don't need to buffer at all.

(Worst is, you didn't get to spec the metal, and there's a whole OS full of someone else's Bright Ideas between you and that 200us-or-the-reactor-goes-blooie control loop. Good luck, you're gonna need it).

> you can put all your bits into a good sized buffer and use DMA to push them out a stable, hardware clocked rate.

Obviously; without buffering, glitch-free audio playback would be impossible. I made the assumption that when we're talking about timing on Linux, it's not about throwing buffers at a hardware clocked device.

Does libgpio or some other interface on Linux give you access to clocked DMA out of the box? Do the drivers on RPi support it?

All the gpio drivers I've worked with are just directly writing state to registers.

Yes, there are DMA driven gpio libraries.

In Linux you can isolate a cpu from the others, so you can get closer to a realtime solution (though you still get contention on cache lines, memory bus etc.). I haven't tried it though.

https://www.linuxjournal.com/article/6900

Ouch those graphs indicate worst case interrupt latency in the 100ms range... that's one old article!

His loop needs to deal with an encoder that generates 1-4K ticks per rotation, hooked to a spindle running, IIRC, 2K RPM. I think his concern was that was 8M events per second, all while also generating stepper outputs.

He mentioned not only microsecond but nanosecond.

It reminds me of Linux routers. They worked great until the PPS rate suddenly got high, like in a network loop or DDoS situation. The kernel would dynamically switch between interrupts and interrupt mitigation/polling, I couldn't find any way to force it to always poll.

If the system was in interrupt mode, and the PPS rate suddenly went way up, the system was so busy handling the interrupts that it couldn't find the time to switch to interrupt mitigation and would livelock.

The solution was fairly easy: Just have more CPU cores than physical network interfaces. Disclaimer: This was ~10 years ago.

He never says that. You probably could do the same with multiple 8 bit arduinos as well, I have done something similar myself with klipper. But your life while doing that will be miserable.

The first thing you have to add for anything serious is a real time clock that Raspi DOES NOT HAVE by default.

What he is saying is that Linux is not intended for that,not designed for that, because it it not a RTOS, and it is huge.

You can fill the void, but you will be reinventing the wheel. Low level hardware design is painful.

Linux is extremely complex, too complex for a person to understand. It will take man-years of work to handle all the details, that is, it cost hundreds of thousands of dollars just in salaries alone.

It is way easier to do what this man has done. And then if you want you add an additional raspi with linux as the controlling UI.

Any cpu with a cache cannot do it because you are not sure if the data is in cache and will arrive on time or not. If there is something else on the bus (USB, video, network...) you need to design the whole system to ensure that the cpu gets the bus when it needs it regardless of other uses - most computers are not designed this way. A multitasking cpu generally cannot do this because you cannot be sure some other process isn't using the cpu when you need something.

That isn't to say it can't be done. Audio is very time sensitive and Linux does it well with the right kernel options. (video is easier, though it uses far more cpu and data on the bus, it is less sensitive to delay).

Depending on how bad delay it might or might not work. As rule though most real time control is run by low speed 8 bit cpus that have exact timing. You have a high power cpu doing the calculations and then the microcontroller just runs the result of the calculations checking in for new orders as needed.

It depends on if you need closed-loop control (feedback-driven systems like DC motors with encoders) or open-loop control (where you just output a signal and you can trust that the system moves to a specific state without checking up on it).

A few years ago I made a 3d printer that used an open-loop stepper system in userland on the original raspberry pi. I got stepper precision down to 2 microseconds [1] — which IIRC is better than the arduino-based systems out there were using. Latency was a different matter — the steppers would still overstep a few times after triggering the endstops.

It takes some hacking (I used DMA and the audio peripherals to offload timing-critical parts from the cpu), but open-loop control is totally doable at 2 microseconds if you’re dedicated to getting it working. Maybe you can get down to 1 us on the newer raspberry pi’s. If you repurpose the video core, maybe you can find a way to do real-time closed-loop control, but that sounds even more involved than what in the end is a relatively simple DMA hack.

[1] https://github.com/Wallacoloo/printipi/blob/master/README.md... Read the scheduler code or check the video linked from the readme if you want more detail.

Not Raspberry Pi exactly, but the SoC used in Beaglebone has a couple "PRU" co-processors which run at 200MHz, share RAM with the main CPU, and have access to peripherals including some GPIO. It's not too hard to use those for precision timing while the main CPU does the typical Linux stuff.

I've heard that said, but I've never actually seen a real project that uses those coprocessors. I've always thought that odd as it seems like the perfect setup for real time control. (I have seen tutorials on how to use them but no non-educatuonal projects)

Yes, I think they're more of a curiosity for many hacker types, but they definitely do get used.

My Siglent SDG-2042X signal generator uses the same SoC, would guess it's using the PRUs to drive its DACs.

A member of our local makerspace used a PRU to drive a big display made with shift-register RGB LEDs (something like a WS2812 - can't recall details).

I have made a proof-of-concept interface to a fairly high end ADC that did a little bit of filtering in the PRU before sending the results to the main CPU.

It is a great setup, but most of the projects you'll hear of are in the hobbyist world where cost reigns supreme.

Raspberry Pi is much more popular and costs far less. The PRUs can be replaced by a couple of Arduino Nanos for a buck or two each. Won't run at 200MHZ, but there are very few control tasks that need to run that fast.

Ken Shirriff used them in his Xerox Alto ethernet gateway [0] for bit banging a 3Mb/s ethernet signal.

[0] http://www.righto.com/2018/01/xerox-altos-3-mbs-ethernet-bui...

have a look at machinekit (linuxcnc-fork): https://github.com/machinekit/machinekit-hal/tree/master/src...

With proper care it's doable, but that stack is bringing a whole lot of baggage you'll need to either strip out or isolate your control process from.

Having a bunch of cores is advantageous though. There are a variety of options for dedicating one core to the special purpose while the rest serve the general-purpose wild west.

But I'd be surprised if you didn't have headaches just using some python code with cpu affinity on an otherwise unchanged raspbian install.

Linux for sure doesn't for the reasons stated here by others.

I would love to see a real RTOS on a RPi though. Something based on sel4 would be very nice to see for instance. You won't get nanosecond precision due to the cache/system effects like people are saying, but microsecond precision should be more than doable.

It's quite doable but you have to do some low level plumbing to make it work, simply disable interrupts, pre-read your data at least once to warm up the cache and you're good to go. That's how I controlled my plasmacutter.

Linux is a GPOS (general purpose operating system) whereas the domain of servo controller programming (as shown in the videos) is falls under (RTOS). Even if there is no RTOS installed and the author is programming on bare metal with interrupt-service-routines (ISR), it is essentially a "Real-time Operating System" or RTOS. The author could also have used FreeRTOS. The primary difference between GPOS such as Linux and RTOS (bare metal roundrobin or using something like FreeRTOS, Micrium, etc) is that GPOS focus on "throughput" and RTOS specializes in "priority". RTOS works by priority-scheduling or roundrobin.

An example would be better. Say there is an external input to the system (a sensor that detects roller coaster position), you want a deterministic scheduling of the task without any concern about the rest of the state of the operating system. Doesn't matter what the OS is doing - when the roller coaster's position is sensed and there is an interrupt generated to call a subroutine (ISR), a RTOS will not block and it will guarantee execution. Meanwhile, Linux will schedule the task hoping it will get executed but there is no guarantee... I am sure you can hack into the kernel space and write a driver but it becomes a much more of a laborious task - it is just better to use the right OS from the get go.

Hope that makes sense.

A professional motion control system such as [1] comes with 2 components. RTOS element which is a proprietary controller and a GPOS component which is used to communicate, monitor, interface with the user (GUI) and do high-level functions. Usually this is a piece of software installed on a Windows OS, with dedicated PCI cards to communicate to the RTOS modules.

[1] https://www.aerotech.com/product-catalog.aspx

https://stackoverflow.com/questions/536506/how-do-real-time-...

Also, checkout LynxOS...this is the top-end of all RTOSs which has some POSIX (UNIX) like features. It can also run on powerful processors such as Intel and AMD. This is used in airplanes (avionics), trains and public works, super critical operations: https://www.lynx.com/products/lynxos-178-do-178c-certified-p...

I think a better way to describe an RTOS than as specializing in "priority" is that an RTOS has bounded latency. If it's hard real-time that means deterministically bounded, while soft real-time means probabilistically bounded. What that bound is (or the distribution for soft real-time systems) is up to the system designer. It could be a microsecond, could be a day. But it has to exist, whereas in a GPOS there's no bound at all.

I have seen some other people using an FPGA board to maintain instructions at a specific clock rate and interfacing with it via LinuxCNC.

When he talks about the interface [1] I think he touches a really important point: you have to know the user and usage to decide what interface to use.

As he says it is very tempting to use modern technology like touchscreens. But because he knows he will touch it with oily fingers that's not an option.

Great project!

[1] Part 1: Proof of Concept @15:10

Reply on news.ycombinator.com