RT-X: the largest open-source robot dataset | Hacker News Reader

RT-X: the largest open-source robot dataset

robotics-transformer-x.github.io

147 pointsOsiris307 comments

The word on the street among the researchers I know from my grad school days is that this is basically a bunch of datasets that were created earlier on for specific papers, and the subset of researchers are repackaging them as a kind of weird academia twitter winnertakeall thing.

The author list also reminds me of the embarrassing practices of the physics community…

Nothing wrong with releasing a webpage with a list of datasets, but the press and need for a whole big announcement that claims it will change robotics makes them seem desperate.

Paper is pretty upfront about that, "we provide the Open X-Embodiment (OXE) Repository, which includes a dataset with 22 different robotic embodiments from 21 different institutions". Some of the data is new from Google's robots, but the rest is not.

The author list is long because Google is a big company with big teams.

Aside from this paper, I do think that generative modeling is going to change robotics. It seems likely that large transformer models could learn to do many simple tasks in response to natural-language instructions. This could be the burger-flipping robot.

The author list isn't long because everybody at Google got listed. There are a ton of full Professors in the author list. This is a cross-university effort.

They are attention-seeking, but they don't care about you, they want big grant money. Training is very expensive and it's a lot to ask for money for not just robotics hardware but now also big compute. Imagine trying to bid for a $100m H100 cluster like some sovereign nations.

I think one of the contributions is that the data are provided in the same format for all robot embodiments, i.e. action spaces, and observations. This way they can feed all the data to same model.

Love clicking a GitHub icon to go read the source code and learn more… and being taken to a google cloud storage bucket where there’s four files, a readme pdf that’s basically just links to several different google sheets documenting data that went into this, two ipython notebooks, and a pre trained model file in a big zip file which I can already tell will be the usual zipped pickle of a massive tensor flow model that deeply frustrating to work introspect and examine the code behind…

This feels like the GAN/capsule networks papers and “releases” I’d look at when I first tried to learn more about machine learning 4-5 years ago, back in then “Middle Ages” of ML, where the paucity of sources enriched the careers of Machine Learning experts who had no incentive to make their work really accessible beyond puff piece show off papers and a doi/citation to pad their h-rank or a link to the startup that funded it for their flimsy product concept that based on historical evidence was over 90% likely to fail to go beyond concept or prototype demo stage.

Edit! I might sound a bit bitter or annoyed but it’s because I’ve been getting into local ML for privacy and personal productivity and really trying to understand the capabilities and the hardware required for those capabilities… I’ve been particularly interested in potential robotics applications and I’m following this particular branch of ML intersecting Robotics quite closely as it affects the space robotics startup I founded, and it’s undoubtedly interesting work… I’m just deeply disappointed at how much of a throwback to a “worse” time this particular “publication” feels like. It’s not light on details of how they actually built it, the code feels like a throw it over the wall afterthought, and the distribution mechanism has had no effort put into making the material any more accessible, if the ipynb files were in a GitHub repo at least the notebook would be rendered so I could just read it in my browser without downloading and opening up the notebook in a code editor… it feels like there’s a code repo somewhere with a script that dumps these files into this bucket and that’s the repo where they run everything in Google colab and the strong association to Google’s DeepMind team makes me think that once I do open this up I’m going to find something that’s welded solid to googles custom TPUs and is basically useless to me… hence the mild vitriol because the entire publication gives me a bad taste in my mouth despite being extremely interested in the topic and willing to put work into understanding the current state of the art.

It would make an awesome toy for my son (and father) to have generative control over a robot.

Reply on news.ycombinator.com