Hacker News Reader

From near the bottom:

> One of the core new elements of the drone’s AI is the use of a deep neural network that doesn’t send control commands to a traditional human controller, but directly to the motors.

I saw that too - I'm assuming it means they're indeed using the DNN for stabilization. This has been done several times over the years, but generally with results which only rival PID and don't surpass it, so that's quite interesting. What's odd is that the physical architecture of the drone doesn't really make sense for this, so there must be some tweaks beyond the "spec" model. Hopefully some papers come soon instead of press releases.

They reference ESA's research in "Guidance and Control Nets", and when looking at ESA's page for their "Advanced Concepts Team" [0] they in turn reference ETH Zürich's research in RL for drone control. Specifically [1] this paper from 2023: "Champion-level drone racing using deep reinforcement learning" [2]. They use a 2x128 MLP for the control policy.

[0] https://www.esa.int/gsp/ACT/

[1] https://www.esa.int/gsp/ACT/projects/rl_vs_imitation_learnin...

[2] https://www.zora.uzh.ch/id/eprint/257405/

This is crazy, its dexterity and range of motion could potentially exceed all human modeled systems.

I assume that they shave off milliseconds by doing so, and a gyroscope (or similar) sends back the position/angle of the drone. And like this does it bypass the 'limited' onboard computer and instead uses a much better/faster computer?

Reports downthread suggest that the NN is running directly on the drone, in the form of a Jetson. Which would give much better latency and quality of video.

There's a few more details in the press release from the league itself. Sounds like they were really trying to put these things through their paces.

> The course design pushed the boundaries of perception-based autonomy—featuring wide gate spacing, irregular lighting, and minimal visual markers. The use of rolling shutter cameras further heightened the difficulty, testing each team’s ability to deliver fast, stable performance under demanding conditions

https://a2rl.io/press-release/9/artificial-intelligence-triu...

It used a small RL trained network running on the flight controllers MCU directly that controlled the motors given state (position, orientation ...) inputs. The Jetson handled vision processing.

I imagine the slower speed is a closer fit to combat drones (which have a payload and sometimes a fiber optic cable)? Also watching MultiGP they sorta move/accelerate too fast for me to fully appreciate the maneuvering.

Feels kinda similar to the innovation around manned aircraft about 100 years ago when we went from toy/observation platform to killing machine in only a couple of decades. With the ardupilot news today, it was hard to not watch this and imagine the applications to a combat environment.

> which have a payload and sometimes a fiber optic cable

The optic cable is for the human pilot. An AI piloted drone doesn't need it.

Although even autonomous combat/ISTAR drones may require fibre spools for BDA, ISTAR, etc.

If we are ok with AI drones autonomously choosing bombing targets, then you're right.

I think Ukraine would be ok with AI drones launched next to a Russian airfield autonomously choosing targets...

I expect Russia will be ok with it in any situation.

Obviously the Russians wouldn't be one hundred percent okay with drones autonomously choosing targets in or near Russian airfields...

They'd be fine with the drones taking out all of their 3 and 5 engine bombers though.

I suspect the first big use case and subsequent rewrite of the battlefield will come from AI drones targeting enemy drones.

It's been done before, specifically in the naval context.

> I imagine the slower speed is a closer fit to combat drones

A lot of comments are trying to draw connections to combat drones, but drone racing like this has been a hobby thing for a long time. The capabilities of the drones are set to have an even playing field, not to match combat drones or anything.

These aren't meant to have any parallels to combat drones, drones that fly long distances, or drones that carry payloads.

It's really just a special-purpose hobby thing for flying through a series of gates very quickly. Flight time measured in a couple minutes, no provisions for carrying weight.

We all understand that. People are simply observing that there an obvious path from this technology demonstrator to something similar in the battlefield.

> We all understand that.

I'm afraid not. RC/FPV is already a niche hobby, and media coverage is universally negative. No wonder laypeople mostly think of kamikaze drones when they see something like this.

You mean the stuff that anyone knew was possible 10 years ago, but was waiting for the tech to become much cheaper?

Yeah, now its just cheaper enough so its happening

> The spec DCL drone isn't very capable compared to the more open-specification drones in racing leagues like MultiGP

Yeah, I’m sure this is a great milestone but it isn’t notable until AI is beating MCK[1] who would be the “Lee Sodol of FPV”

[1] https://youtu.be/fmc1URVdyUs?si=jPPx5sHjU_ZDOghF

> which is present on the drone and use direct IMU input and ESC commands from the Jetson

Does the jetson board even have the appropriate UARTs on it to talk directly to the ESCs? Your typical hobby grade 5" class size ESC (either 4-in-1 or discrete) cannot talk to two different controllers at the same time. If it's already wired to the four UART outputs from a 30x30 size flight controller (such as something STM32H7 based running betaflight), the ESC cannot be in communication with any other device.

Right, that was my point about the physical layout not really being conducive to NN-driven stabilization, on my cursory read.

Digging in, you could probably bitbang DShot on Orin GPIOs. It would be really sketchy since it's presumably running Linux on out-of-order cores with giant cache, so there's nothing close to realtime going on, but it's so damn fast that the theoretical problem probably doesn't matter and I bet it would work fine.

Orin also has "safety island" for ASIL applications which has lockstep Cortex-R52s. This would be ideal to run an integrated flight controller RTOS but it's supposedly fused off on consumer boards, lame. There's also a Cortex-R5 called "SPE" that could probably run a flight control RTOS as well.

Regardless of all of this theory craft, I was either totally wrong or accidentally right in two ways, depending on how you want to look at it: according to a sibling post, the STM32 board actually runs the NN-backed stabilization loop rather than a PID one! I've seen this done in research before but never in a meaningfully successful way, so I'm even more impressed now - as far as I can tell, it's _two_ novel solves, an actually working PID-or-better NN-based stabilization system on STM32 _and_ inside-out vision based flight planning for racing on the Nvidia board.

Its not just a stabilization/rate control loop but the NN does both guidance and control. It is an MLP that takes state + upcoming gate positions/waypoints and controls the motors directly. I.e. there is no explicit path planning going on but rather the NN outputs motor commands that hopefully help the drone traverse the given waypoints in minimum time. I think this is quite cool because running a similar Optimal Control MPC setup is really not at all feasible on a normal STM32H7 Flight controller, especially at 1kHz.

Oh, I see how this works now! That's _incredibly_ clever, not at _all_ what I would have guessed, and quite innovative. Thank you so much for taking the time to post these explanations.

Also, this track looks nothing like a competitive drone race track, the obstacles are easier and it seems designed to cater to the autonomous drones.

It’s DCL, they’re all kind of poor like this IMO. Certainly nothing like MultiGP