AGI is not needed to transform human instructions into tasks - GPT shows LLMs can do a reasonable job of taking task definitions and expressing possible ‘code’ solutions to them; prompt chaining is a useful approach for guiding LLMs to solve problems, so turning Dave shouting down ‘hey atlas, pass me that toolbag’ into ‘generate a series of robot execution steps to pass Dave the toolbag.’, together with some previously gathered context like ‘ Dave is on a scaffold 15 feet up; the toolbag is on the ground; there’s a platform about 5 feet off the ground next to the scaffold; there’s a plank here’, into an actual plan like ‘move the plank to make a ramp to the platform; get the toolbag; carry it up the ramp into the platform; throw the toolbag to Dave’ is actually not really beyond the realm of LLM capabilities

(Seriously, I tried this with ChatGPT and it doesn’t do badly)

Once you’ve got the model to produce a plan, you can do things like have Atlas explain it to Dave before attempting it (giving Dave plenty of opportunities to tell Atlas the plank won’t hold its weight, or that throwing a heavy bag at a person working at height is a clear OSHA violation; or, presumably to suggest Atlas could perform a sick flip off the platform at the end). Then once the plan’s approved again, the LLM can try to trim those steps into python code or whatever to call the actual atlas goal management APIs.

This seems feasible given LLM tech, but it’s likely there are better approaches - not everything here is a nail that the GPT hammer should be used to hit home. The point is that there seem to be pathways to combine LLMs into the mix of human - robot interaction that could solve the challenge that right now when Dave needs his toolbag handing to him, a team of MIT PhDs need to spend three months expressing that problem in subproblems of vision and manipulation that Atlas can understand.

But it’s also not AGI. It’s a human-robot interface mediated through LLM.