‘Deep learning’ technique enables robot mastery of skills via trial and error
newscenter.berkeley.edu
newscenter.berkeley.edu
What's exciting here is that the entire system is trained end-to-end (including the vision component). In other words, it's heading towards agents/robots that consist entirely of a single neural net and that's it; There is no software stack at all - it's just a GPU running a neural net "code base", from perception to actuators. In this respect this work is similar to the Atari game-playing agent that has to learn to see while also learning to play the game. Except this setting is quite a lot more difficult in some respects; In particular, the actions in the Deepmind Atari paper are few and discrete, while here the robot is an actual physical system with a very large-dimensional and continuous action space (joint torques). Also, if you're new to the field you might think "why is the robot so slow?", while someone in the field is thinking "holy crap how can it be so fast?"
It's an ambitious project that wanted to "Evolve" agents made of sticks + joints that ran in a 3d world with physics. The joints had signal inputs + activation functions that are similar to AN activation functions.
This technique seems to be the best bet of all the machine learning techniques to be solvable by Moore's law. If it currently takes about three hours to learn to do these simple tasks with no previous spatial data for the objects, then as the article states "In the next five to 10 years, we may see significant advances in robot learning capabilities through this line of work."
Yes! this story and yesterdays RNN one with this comment https://news.ycombinator.com/item?id=9584988 "It's like a child learning to talk"
make me believe we are very close to some kind of breakthrough using "boring" non-magical methods.
We evaluated our method by training policies for hanging a coat hanger on a clothes rack, inserting a block into a shape sorting cube, fitting the claw of a toy hammer under a nail with various grasps, and screwing on a bottle cap. The cost function for these tasks encourages low distance between three points on the end-effector and corresponding target points, low torques, and, for the bottle task, spinning the wrist. The equations for these cost functions follow prior work.[23]
The reference 23 points at S. Levine, N. Wagener, and P. Abbeel. Learning contactrich manipulation skills with guided policy search. In International Conference on Robotics and Automation(ICRA), 2015.
I haven't read that paper yet.
Interestingly, they initialise the visual learning model using the ImageNet images. Was it 3 years ago that was considered a pretty much intractable problem, and now the fact a CNN can work on it well enough to be useful isn't even worth a complete sentence.
EDIT: Downvoter, care to explain the reason for downvoting?
Source for the brain having computational units that aren't neural nets? I'd love to read more on this.
As an aside, there are a number of useful properties seen in biological neural networks that aren't yet incorporated into multi-layer perceptrons. E.g. short-term plasticity, axonal delays, spiking neurons, etc. I expect that some of these will find their way into the MLP formulation when we can figure out an effective mathematical way of doing so.
I didn't downvote, but it's tiresome to hear "Yeah, that's all cool and everything, but I'm concerned that it might not be the absolute best." Nobody claimed it was the most efficient approach ever possible, only that it was cool, surprising, and reasonably ground breaking.
You can't really tell if it's the most efficient approach until you try it, and compare it with a model that you think might be a more efficient approach.
And I doubt it was trained. It seems to be following a predefined procedure instead of learning by itself.
It seems to have come a pretty long way if you ask me!
But you can't remember everything, so ideally you'd also have an attention model that's capable of looking back at what you've written, and make edits. When I balance parentheses, I don't maintain a counter of how many parentheses are open, I look back at what I've written and count them.
What struck me the most is this number: 92,000. That's the total number of parameters in the neural net guiding the robot.
In other words, the robot learned to do this with a tiny toy neural net!
--
[1] Thank you not just for this helpful explanatory comment, but also for all the friendly explanatory lectures, presentations, blog posts and open-source code you have shared online over the past few years. You deserve recognition for it.
Would love to see youtube channel describing these deep learning /machine learning / AI subjects.
Are there any prospects for integrating neural nets with more traditional code? For all their power, neural nets share the limitations of humans: they learned behaviors are approximate and not perfect. So for example if your deep learning robot wanted to play chess, it would be nice to have him switch to using a chess engine instead of learning to play chess. Or you might want to hardcode a few very efficient moves into your otherwise autonomous industrial robot.
Is that possible?
This talk by Sergey Levine, Pieter Abbeel's PostDoc outlines Berkley's end-to-end deep-training visuomotor control in detail.
Here is the paper :
End-to-End Training of Deep Visuomotor Policies, Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel.
Learning Contact-Rich Manipulation Skills with Guided Policy Search Levine, Wagener, Abbeel
This UCB project looks awesome!
BTW, I took Hinton's Coursera neural network class a few years ago, and it was excellent. Take it if that course is still online.
The robot is then shown the task a few times. A human controls the robot for a few minutes, performing the task.
Then an innovative policy search finds a robust policy so the robot can perform the task from any initial position and is robust to changes such as the addition of a shirt to the hanger task after training.
Potentially the robot can learn from videos of humans performing the task - i.e. by copying people.
i.e. when given wood with a protruding nail + hammer, it relates the task to a previously trained Whac-A-Mole scenario and begins hammering the nail in.
Then again, why do we make so many containers with these ungainly screw caps? Ever use those caps (popular in Japan) with the locking track that only take a quarter-turn to close? Examples
http://www.amazon.com/Yu-Be-Moisturizing-Skin-Cream-Skin-1/d...
http://www.amazon.com/Biotene-PBF-Toothpaste-Ounce-Pack/dp/B...
This means that the neural nets used by babies are pre-wired to be good at specific tasks. Then, babies use those neural nets to do "deep learning" for the final part of the process.
Starting from nothing and learning how to do a job is a big step. But having something would be a better start position. What that something is, though, is hard to define.
I tend to disagree. In my perception (as father of two if it counts) is that babies are very poorly wired if at all.
They struggle with basic survival skills like breastfeeding. Some babies get it in the first couple of days, other take weeks of "training" with the help of adults. Awareness of needing sleep seems to be entirely absent (crying is not the best strategy for animals to sleep, huge bug).
Things like language, shapes and intent, are all developed later, and can go entirely undeveloped without stimulation and feedback, so I'd say they are already a product of learning and not pre-wiring.
The only thing I can think of that is most certainly pre-wired is crying. They nail that from day one.
Also don't forget that they are already sensory capable of a lot several weeks before being born, and voice recognition for one thing is something they learn around that time.
Breastfeeding is a tough process, how long does it take a fighter pilot to learn how to properly dock to a supply plane for mid-air refuelling?
That's a complicated procedure with lots of bits an pieces that need to work just-so for milk delivery to take place and quite often it is not just the infant that needs to learn.
As a father myself, I don't agree. I find it impossible to believe that babies are wired poorly, or randomly, or just are amorphous blobs of learning. They're active and inter-active from a very early age. Even before they're born.
Their brain is still growing connections, and re-wiring itself based on sensory input / feedback. i.e. blind people co-opt the vision centers to process sound.
But there is a vision center. There are portions of the brain which are pre-wired to be good at certain activities.
If nothing else, look at the inputs. The nerves from the retina and ears go somewhere. They don't just disappear into random parts of the brain. They're pre-wired to certain areas. Those areas are in turn pre-wired to be good at accepting certain inputs.
In contrast, many animals have much more hard-wired behavior. And insects are little automatons. Are we really going to say that animals are pre-wired with... nothing? And that they learn all of their behavior after they're born?
I find that even harder to believe than the idea that the brain is pre-wired to be good at some things.
Indeed. And this can persist into adulthood...
Think of it as the Home Brew Computer Club for Robotics/AI :)
This would also speed it up imo. Since some things can easily be solved using regular algorithms. Our brains also come with some pre wired functions.
Interesting research.
"We learn about the Markov decision process (and what happens when you use it in the real world and it becomes a partially observable Markov decision process) "
http://www.thetalkingmachines.com/blog/2015/5/21/how-we-thin...