MahiShafiullah

Thank you!

User fishbotics already answers a lot of these questions downstream, but just confirming it here as an author of the project/paper:

> - How does it know what objects are? Does it use some sort of realtime object classifier neural net? What limitations are there here?

We use Lang-SAM (https://github.com/luca-medeiros/lang-segment-anything) to do most of this, with CLIP embeddings (https://openai.com/research/clip) doing most of the heavy lifting of connecting image and text. One of the nice properties of using CLIP-like models is that you don't have to specify the classes you may want to query later, you can just come up with them during runtime.

> - Does the robot know when it can't perform a request? I.e. if you ask it to move a large box or very heavy kettlebell?

Nope! As it is right now, the models are very simple and they don't try to do anything fancy. However, that's why we open up our code! So the community can build smarter robots on top of this project that can use even more visual cues about the environment.

> - How well does it do if the object is hidden or obscured? Does it go looking for it? What if it must move another object to get access to the requested one?

It fails when the object is hidden or obscured in the initial scan, but once again we think it could be a great starting point for further research :) One of the nice things, however, is that we take full 3D information in consideration, and so even if some object is visible from only some of the angles, the robot has a chance to find it.

A large motivation behind this line of home-robot work for me is thinking about the elderly, people with disabilities, or busy parents who simply don't have enough time to do it all. I am personally hopeful that we can teach AI to take the jobs that no one wants rather than the jobs that everyone wants :)

In fact, Hello Robot already shared a teleoperated demo of folding shirts! https://www.youtube.com/watch?v=QtG8nJ78x2M&t=180s But yes, a second arm is needed.

Thank you! A large motivation behind this line of home-robot work for me is thinking about the elderly, people with disabilities, or busy parents who simply don't have enough time to do it all. I am personally hopeful that we can teach AI to take the jobs that no one wants rather than the jobs that everyone wants :)

It's always a trade-off! You could have more accurate sensors and motors that are more expensive, or you can have cheaper motors with no sensors and higher accumulated errors. Since this is more of a research project than a product, we went for a cheap robot with the slower-but-more-accurate approach.

The title has multiple meanings, some credits definitely should go to OK computer/radiohead, but also "OK Google" for controlling a home assistant, open-knowledge (OK) models, etc.

No, although it has some of the same people on the team (aka I'm the first author there, and my advisor is advising both projects :) )

The primary difference is that this is zero-shot (meaning the robot needs 0 (zero!) new data in a new home) but has only two skills (pick and drop); where Dobb-E can have many skills but will need you to give some demonstrations in a new home.

The mapping process can be done with any RGB-D cameras, we use an iPhone pro but any apple devide with AR-Kit should work. Once we have a sequence of RGB-D images with associated camera poses, we can just backproject the pixels (and any associated information, like CLIP embeddings) using the depth into voxels.

Here is the page for Hello Robots: https://hello-robot.com/

Thanks! We collect all the data and analyze it post-facto to see what may have caused the failure. For example, on the 5 trials you mentioned, the Realsense gave wrong depth on transparent or semi-transparent objects, and so the pointcloud generated from the robot's head camera was simply wrong.

Second that. Something I put out recently had an (admittedly video heavy) webpage that had 1TB of traffic over the past month. Cloudflare handled it for free for me, but at ArXiv’s scale it’s bound to be a problem.

Dobb·E is free (and open source) now :)

I am glad you mentioned automobiles! They are among the most heavily engineered everyday product that we use every day, and this engineering and cost optimization has been going on for close to a hundred years now. So, I would read this same statement as "$20k and 100 years of engineering gets us a lot" rather than "$20k gets us a lot". I am hoping better engineering and the wonders of modern supply chains will get us a home robot at 1/10th of the current price when the time is right.

I'm sorry, should have been clearer. The Spot robot with a manipulator arm (which has been used in some home projects before, see [0]) I believe totals up to about $200K. The base robot is $75K, which is what you linked to. I mentioned that figure because that seemed more relevant.

[0] https://adaptiveskillcoordination.github.io/

Smaller robots are unfortunately limited in how much force/torque they can apply and how high they can reach, two factors that we found can severely limit the applicability of a home robot. As a result, unfortunately, we don't have much plans for extending to a smaller robot at this moment.

However, all of our designs are open source! So if you or someone else is interested, it could be a fun weekend project to design a "Stick" equivalent data collection tool for the XGO CM4. I would love to see how that turns out.

> For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app.

I agree with you there. My impression was that the cinematic format is closed, unlike .r3d which is a bunch of open format files in a .zip. Do you know whether Apple has published the specs for the cinematic video format anywhere, or if there are there good libraries for handling it?

I am sorry, what problems are you facing? I tested it with Safari on both my phone and laptop just now and couldn't find an issue, so I would really appreciate it if you can reply here or (more appreciated) email me with images etc. on the email listed on my profile.

I much appreciate your heads up, and thank you for taking your time on this.

Thank you Suren! It will get there when it gets there and so will I :) I do miss you folks, though.

I discuss this a little bit on another thread: (https://news.ycombinator.com/item?id=38461559)

But the tl;dr is that building prototypes is pricey, and the cost should definitely come down once the community moves from prototyping to mass-producing robots.

Hello Robots (the company behind the Stretch robot we're using) is also trying to bootstrap and build a sustainable product rather than blitzscaling and burning out fast [0, 1, 2] for which I respect them a lot. It's all too common of a story in robot world where have a great company showing lots of promise and then a year or two later they shut down after burning through investor money. I don't want to see it repeat.

[0] https://techcrunch.com/2018/07/25/mayfield-robotics-ceases-p...

[1] https://www.theverge.com/2023/2/24/23613214/everyday-robots-...

[2] https://www.theverge.com/2019/6/19/18682780/jibo-death-serve...

Tool usage is one of the next ideas we would like to explore; because, if you think about it, the reacher-grabber that is currently used as an end-effector is just another tool.

If we can switch between tools quickly we can possibly unlock a wide range of applications: imagine a robot that can switch arms from a whisk to a spatula to a duster to to a mop all over the course of preparing a meal, and then switch to a screwdriver or a wrench for fixing something afterwards. Will take work to design modular parts like that, but the end result can be quite exciting indeed.

Great observation!

Right now we rely on the out-of-the-box Hello Robots controller [0] implementing position control. Since our policy only predicts a single position into the future at a time, the motion ends up being either jerky and fast, or smooth and slow. Improving on the controller is probably one of the highest ROI improvements we can make moving into the future.

[0] https://github.com/hello-robot/stretch_body

OpenEXR looks very interesting! Especially if OpenEXR can compress depth data nearly as efficiently as we can compress image data by making everything a video, it would be a game changer. For example, our RGB dataset is 814 MB right now while the RGB-D dataset is 77 GB, even though depth "just" adds one new channel to our data.

The robot can carry small weights on its own, yes, although it's way more stable when the arm is retracted closer to the body.

Pick-and-place style problems are one of the earliest tried on our robot platform, Hello Stretch, which is why this project doesn't spend too much time on it. You may enjoy something like the OVMM project [0] which focuses entirely on open-vocabulary pick-and-place problems: "Pick up $A from $B and put it in $C".

[0] https://ovmm.github.io/

We don't want any private information, so currently we are manually going over every new demo to make sure private data (face, hands, any other identifying information) doesn't get included in the dataset. Pretty hard to "blitzscale" this way, but I personally think doing it right is more important.

Thank you!

Yes, we have considered launching an open source effort to scale up the data, perhaps to be comparable to what Google has collected through paid tele-operators [0]. What do you think would be the right incentive structure for everyday volunteers to participate in this effort?

[0] https://everydayrobots.com/thinking/rt-1-robotics-transforme...

Thanks for the words of encouragement! If I may ask, what is the level of "maturity" you expect before adopting a home robot? Clearly, a robot that can wash and fold laundry is already there, but what are the other necessary/sufficient qualities?

Not in our plans anytime soon, sorry :) We're focusing on home robots right now instead of rather unsettling space mutants.

Thank you for the suggestions!

1. I'm ordering a microscope mount right now! We went with 3D printing because we could get a very good coupling between the stick and the robot mount. See this figure for a comparison: https://i.imgur.com/vWopcFB.jpeg

2. Our dataset (and dataset export code) gives the depth as numpy files and trajectory information in a JSON file with calibration and trajectory information! The r3d export format from the Record3D app we used is simply a .zip file with renamed extension so we were ok with using it. Do you have suggestion as to which cinematic format you are thinking?

Water!

Our robot-and-phone setup is not very safe around liquids haha. Hopefully the hardware experts among us can solve that problem sooner than later.

We are also working on learning multi-stage skills, and early results are promising (See section 4.1 on the paper). However, with multiple steps the chances of failure also goes up, so that's something we will have to make more robust :)