Comment by throwup238 | Hacker News Reader

Comment by throwup238 | Hacker News Reader

This research is all beyond me so maybe someone can explain: How does this compare to the state of the art in using simulators to train physical robots? Does using transformers help in any way or can this just as easily be done with other architectures?

To the uninitiated this looks cool as all heck and yet another step towards the Star Trek future where we do everything in a simulator first and it always kinda just works in the real world (plot requirements notwithstanding).

Although I can also hear the distant sounds of a hundred military R&D labs booting up Metalhead [1] simulators.

Edit: Looks like the previous SOTA was still a manual process where the user had to come up with a reward function that actually rewards the actions they wanted to the algorithm to learn. This research uses language models to do that tedious step instead.

[1] https://en.wikipedia.org/wiki/Metalhead_(Black_Mirror)

Your edit is roughly correct. It's a bit odd that they're comparing a single human-made policy with the very best of the DrEureka outputs; in practice, I would expect to make multiple different reward functions and then do validation on the functions and hyperparameters based on the resulting trained models. However, their comparison isn't necessarily wrong, since it seems (I could be wrong) that they used the reward function from prior papers in the field of policy learning.

If you're interested in methods for actually learning policies for these sorts of dynamic motions, note that this paper is simply applying proximal-policy optimization. They're pulling in the training and implementation methods from Margolis's [1] and Shan's [2] work.

So, in sum, the contribution of this paper is exclusively the method for generating reward functions (which is still pretty cool!!!!!), not all the learning-based policy stuff.

[1]: https://web.archive.org/web/20220703005502id_/http://www.rob... [2]: https://arxiv.org/pdf/2309.06440

Transformers don't care if it is simulation or real data or even if you are using a different robot altogether. At some point it will be possible to show a robot a human demonstration and let it learn from that.

Reply on news.ycombinator.com