Diffusion Models for Robotics Control | Hacker News Reader

Diffusion Models for Robotics Control

diffusion-policy.cs.columbia.edu

75 pointsPalmik13 comments

This is exactly what we need to enable true embodied AGI - think “mirror neurons” but for robotics.

General-purpose humanoid robots capable of interfacing with our physical world will be one of humanity’s most pivotal innovations.

* [If you’re working at the intersection of machine perception & embodied AGI let’s get in touch!] *

You mean that you think it is essential for developing AGI?

The term of art is "embodied intelligence". https://www.nature.com/articles/s42254-022-00481-z

Mildly woo. Lots of crossover with the soft sciences and liberal arts here.

As far as I understand, this approach is what Toyota Research Institute is basing their foundation model efforts on. https://youtu.be/w-CGSQAO5-Q

Yes, TRI is a co-author of this paper (along with MIT). I am curious how much Tesla’s approach differs with their robot, though.

This looks great! Language models get a lot of attention, but advances in robotics controls will have a huge impact on the world.

(It makes sense that language and image models get the most attention, as they can be worked on without leaving a desk or buying expensive application-specific hardware. Robotics is simply much more complicated than text or images.)

Language models are also hugely important for robotics applications. One example of significant LLM-powered increase in capabilities is this: https://youtu.be/CnkM0AecxYA?si=zXSj4MK-9snDQgL3

There are great simulation environments for robotics too. Check out the "Simulation Benchmarks" section of the article to get an impression (there are references to several systems at the bottom of the section).

Nice work. They still use a mid-level controller to translate the output of the policy into kinematics which is an all-too-frequently accepted part of robotics. Ideally control is learned end-to-end, which is how you get more natural robot motion.

Any examples of projects that do it like that?

Amazing. All these developments in robotics over the last few years after decades of stagnation really make me want to return to the field (my first job after school in 1997 was designing robots for nuclear reactors).

> was designing robots for nuclear reactors

I'm not sure AI is the best replacement for something that requires deterministic behavior! ;)

Why do you think this would require deterministic behaviour?

You just need good behaviour.

Eg you could use any non-deterministic technique, up and including reading bird entrails, you want to eg solve inverse kinematics problems and verify and review them before you enact them. (That review could be done automatically.)

More broadly, all of NP is defined like that: it's the kind of problems where you can in polynomial time verify / reject a candidate solution that you 'guessed' by whatever means.

Reply on news.ycombinator.com