It does seem like this work (and a lot of robot learning works) are still stuck on position/velocity control and not impedance control. Which is essentially output where to go, either closed-loop with a controller or open-loop with a motion planner. This seem to dramatically lower the data requirement but it feel like a fundamental limit to what task we can accomplish.
The reason robot manipulation is hard is because we need to take in to the account not just what's happening in the world but also how our interaction alters it and how we need to react to that.
Robot Learning right now is either Reinforcement Learning or Imitation Learning (often latter) and I'm not sure how one would collect data that capture this.
Edit: I'm surprised Google still do robotics work, I thought Everyday Robotics was shut down [1]
[1] https://www.therobotreport.com/alphabet-closes-everyday-robo...
As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work. The fingers don't need to know how the feet work, for example. There should be semi-autonomous "intelligence" embedded throughout the system, with only necessary feedback being fed back up.
For example, when you reach out to pick up a green can, your brain makes the decision to do the task but it's your spinal cord and peripheral nerves that carry out the detailed work – orienting the hand, managing grasp strength, controlling the arm movements etc. This process is mostly unconscious – you don't need to actively think about how to tense each muscle in the same way that an embedded controller wouldn't need to understanding the working of the entire robotic system to carry out its specific task.
Much like the model suggested, the human body communicates feedback across layers — this process is crucial to maintaining balance, coordination and effectively reacting to the environment. For instance, if your fingers touch a hot stove, the sensory receptors in your skin will immediately send a signal to your spinal cord and a reflex action will make you pull your hand back even before you consciously perceive that the stove is hot.
For example, as I enter this text, most of my fingers are holding the phone and do not need to be told to keep holding it.
The low level controllers should also be able to respond to simple things.
It's like when I get hurt, my hand yanks back before my brain realizes what has happened. Or when I almost trip and my legs correct before I realized it happened.
Low level tasks should be at a low level and not bother the higher professors.
I wonder if it would help a LLM running a robot to have a separate controller calculating it's position and what is around it, and feed that into the LLM constantly.
It would be like when a video game has a radar display or map to let you know where you are in relation to other things.
You've very much correct that their spatial awareness is terrible. Something as simple as drive forward, then back, turn left, etc. works just fine and they can generally translate it to a specified message format reasonably reliably, but give them something more complex to execute, like drive a robot in a square pattern (an example answer would be go forward, turn right, go forward, turn right, etc.) they start to generate nonsense.
I also tested it with the 30B WizardLM at the time which performed almost as well in terms of message format but had even worse awareness.
Part of the problem is that the training data contains next to no examples that would teach it how 3D space works. I considered making a dataset of driving a robot around with human movement commands and then logging the aggregated sensor data and commands for fine tuning so the prompt format would be pre-learned, but I'm not entirely sure how much it would help.
>As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work.
This is true for us ofc, and is encompassed in what is called the "Moravec's paradox" [2]. You don't understand it because it's unconscious process, and It's harder to reverse-engineer an unconscious process (motor movement) than conscious ones (calculating math, playing game, writing text, reading).
But the thing is that in the real world we do need to take in to account everything, including noise and time-delay. Evolution gave rise to complex language in the last 100k years compared to millions of year for motor movement. I do agree that there must be some "hierarchical" structure for complex motion, but we currently don't know where and how that hierarchy is. Boston dynamics uses Model Predictive Control for complex movement, which means that at least some model of the world is required, for dexterous motion. Now if that model is part of LLM or not is a hard guess.
But if we don't know this it's hard to say what kind of data we need to collect to train a LLM-model applied to embodiment (robotics).
Researchers in the past have already made the mistaken assumption of "oh cognition and language is the hard part of intelligence. Perception and motion is easy" [3] and then their work amounted to nothing because turns out the latter was way, wayyyyyy harder.
There's an implicit bias in us that think Language, puzzles and logic are harder [2] and therefore models that accomplish this can just be rammed in to the "easier" issues.
Edit: I too would like "LLM models will solve these" attitude because otherwise the research I'm doing right now is a dead-end, but the more I try (with my limited compute) the less I'm sure
[1] https://imgur.com/eWsH5ui originally https://twitter.com/DhruvBatraDB/status/1641871357020614656
[2] https://en.wikipedia.org/wiki/Moravec%27s_paradox
[3] https://youtu.be/x10964w00zk?list=PLSQhB89mdG7PsZsDz2_5hZL8C...
Aside from some basic life support systems, don't almost all movements start with conscious effort? Whether you are deliberate about the exercise or not, you practice and practice until you develop 'muscle memory' where it becomes unconscious: walking, dribbling a basketball, holding a G chord on a guitar, etc.
A principal idea behind this work is that you can collect data in one domain to avoid collecting data in another. Training an LLM can train the 'reasoning' portion of the robot so that it can perform real-world skills with less training.
And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent map. Although the latter is solved with SLAM). it is way, way more versatile (if their claims are correct) so this is cool result.
as for sim2real, that's a whole different set of issue.... Simulator have come a long way, but from last time I talked with team-lead of Mujoco sim, it seems like completely realistic ones (at a reasonable compute time) is still far away. Deform-able objects are hard to simulate
My point is just that the "reasoning" you mention have a limit;[1] there's no amount of reasoning with language that can zero-shot a robot to do gymnastics or ride a bide. I'd be more than happy to be proven wrong on this point though, then my line of research is not dead-end =)
Edit: also, the model should be smart about how to use its "context window", when the robot is taught how to do a task and it needs to do that task, it must retain the knowledge.
In the near term I expect we will see much more general robotics that know how to do lots of tasks and can follow basic instructions, but lack the ability to develop complex new skills over time. Robots doing dishes and laundry will soon be feasible, just don’t expect unbounded self improvement.
EDIT: And here's the paper pdf - https://robotics-transformer2.github.io/assets/rt2.pdf
(preferably a physics simulator library, not an entire framework)
idk there is plenty of universities that have robotics labs.. Probably the best value is to apply for masters etc if you are interested in that..
Good off-the-shelf arms are low thousands last time I looked. You can DIY for $1-2k https://www.anninrobotics.com/robot-kits
It becomes expensive when you want an robot arm with high dexterity and torque-sensors, which is often a requirement for some tasks, but position-based control can be done on cheaper models.
Quadruped with an arm for <$1000 would be a steal. Got any recommendations?
It runs on Raspberry Pi which is a big +, as version 1 was running on custom hardware. People had to plug a Jetson Nano or other device to run their code (such as this [2]). It comes with their own App, connected via bluetooth, for simple commands too.
I (and most kickstart backers) funded/bought it for either hobby or research purpose (some backers were from labs) so if you're looking for *practical* application this one might be a bit too weak.
But version 1&2 did come with a processing power that could run YOLO detection so there's enough power to test things out and learn.
[1] https://www.robotshop.com/products/xgo-mini-2-quadruped-robo... [2] https://www.youtube.com/watch?v=xqfwCwu4-Rc
Does it has servos feedback too ?
There's a cool part where they ask the robot to pick up a lion from a group of toy figures it hasn't seen before. After it does it correctly the NYT reporter asks it to pick up the extinct animal and the robot picks up the dinosaur toy.
I would then have one cat and one messed up LLM robot.
Progress!
To fully match the human model, AGI based on neural networks will need to dynamically receive information from sensors of all kinds [to be able to sample at least five major categories of physical stimuli, like us]. If we want to reach the level of the neural networks that we carry in our heads, we have to learn more about the role of other brain structures, such as glial cells, which are known to influence the activation and moderation of synapses in the human brain [which correspond to the 'weights' in artificial neural networks], and then apply them to our technological endeavors.
Without it two things will happen
1. We will never have strong AI 2. Strong AI that works virtually only, aware of the world and material reality, but denied access to them by its creators.
Imagine that it is possible to take a snapshot of a human mental state [an analogue of .h5 weights] and then run it on hardware as we do with our neural networks. It would certainly be a nightmare for a mind endowed with "qualia" to find itself enclosed in a metal box [brain in a vat] with no access to the senses. There are clinical cases analogous to this condition, for example in states where the patient is aware of everything around him but is unable to respond.
These issues highlight the ever-present ethical boundaries of IA research. Without integration with real-time sensors AI will always be incomplete and/or inhumane.
$35K for a robot that can putter around the house doing basic stuff is just not that high of a bar. With a 10 year life span, that's $3.5k/year, or $10/day. Doing 1 hour of useful minimum wage work around the house is just not that high of a bar - doing laundry, cleaning, tidying up, weeding, wiping surfaces down, taking out the garbage etc. If it can do some combination of those, it would make sense for basically every household. And it doesn't need to be able to do the crazy parkour of Boston Dynamics to achieve this. Our world is generally designed to be operable by all sorts of people - disabled, old etc. Crazy athleticism isn't required to do useful work.
They thing to keep in mind about the "Boston Dynamics approach" is that in order to solve the hardest real world problems, you need to do one of two things: 1) Overshoot the capabilities of the system so that it is robust when actually deployed subject to the uncertainty of the real world (e.g. motivating athletic intelligence). 2.) Grossly constrain your environment so that uncertainty is not a factor. This is (and has been) happening in warehouses and factories for decades.
1. Choosing a workflow(i.e. a series of general steps that can achieve a goal) 2. Generating low level policy actions given a state(i.e. sensor data).
Like lets say the Workflow is to clean the room. Now there is a big chair in the way and a USB wire there too, the robot could just decide to move around the chair but it could pick up the wire and place it on the table and clean the area. Generating policy actions this way seems to be very much in line with simple VQA based things proposed in the paper, or so it would seem...
P.S.: I also was told that the key here is that in automated planning you can't have a human in the loop doing the actual learning. If you are going to prompt engineer or get a human in the loop to a degree that you effectively fool yourself that the robot is solving a problem then its not planning.
[1] https://en.wikipedia.org/wiki/Sussman_anomaly [2] https://chat.openai.com/share/16a8a0e9-7422-41da-a192-6393cc... [3] https://twitter.com/rao2z/status/1599462959788744704?s=20
[1] https://arxiv.org/abs/2305.15771 [2] https://youtu.be/x10964w00zk
They're an advertising company with a mission to "organize the worlds information", pivoting to robots? Do they need robots to organize the worlds information?
No idea how big of a breakthrough this is, but it is absolutely undeniable that Googles PR team is going wild on anything AI since OpenAI provided them with their first real existential crisis / code red.
Well, my bookshelves are a mess and I'd be ok with getting a robot to help organize my information, if I had some assurances that it won't kill me, or at the very least a "Don't be evil" clause.
I'm so sick of Google I just paid for Kagi.
Seriously though, I’m excited to see where this goes. AI is progressing so fast now that newcomers loudly proclaim “AI is dead” and “going nowhere” when they haven’t seen SotA beaten for a whole month, because they don’t remember the times when a small improvement in a decade was big news.
There under "demo" are a few videos at 2x and 4x speed. It's slow. None of the videos include audio of the verbal commands or the latency between commands and action.
Edit. So many downvotes -- here have a /s (sarcasm)
Decades have passed, and with pretty much no actual surprises on the way, we're steadily marching towards creating robots with the power to destroy humanity. We haven't made much progress on time travel though, so whatever happens, we probably won't have a chance at a do-over.
Since it didn't happen, maybe a Terminator Genisys type reset of the timeline has occurred. So we have a few more decades buffer again, to maybe 2050?