Imitation Learning (2023)
geohot.github.io
geohot.github.io
In practice, deep learning based behavior cloning works when there is enough data for the model to interpolate to the next “frame”, the proof of which is GPT and other LLMs whose primary training objective is practically behavior cloning. However, adding a neural network means there are some hallucinations, which can range for silly (for LLMs) to murderous (for self driving). However, this presents a huge opportunity as well, in that anyone who can solve hallucinations can make headway into not one but two billion dollar problems at once.
However, the older I get the more I appreciate these folks because they are excited about the problem usually way more than people who know the systems rote
In some cases you get a charlatan trying to recycle something, not open to feedback that they aren’t Kepler or whatever, but in my experience most of these folks are trying to figure the world out and are excited when something seems to fit
Now, I try (not always successful) to “yes and” them into trying to figure out how their method compares to existing methods, with the idea that - hey maybe they figured out an edge case by accident that we can all learn from
That said, this is geohot and he definitely isn’t new to BC/IL
The question I would ask is, would geohot have been better to have gone to school to learn this and deferred all the work he has done?
Who is farther ahead, the people who already knew this from school, or the guy who is building and learning this as he goes?
A PhD in ML is worth less to me than day to day operational product engineering experience utilizing the fast changing ML tooling landscape
This is how you get a four-week transition from "I can fix twitter search in an internship!" to "twitter search is unfixable." Ideally, a PhD teaches you to bust your behind doing a deep dive on a seemingly unsolvable problem for three-five years.
If only we could recalibrate ourselves to stop the search for easy money and remind ourselves that some problems are just genuinely hard.
The best senior engineers on the planet have a BS and 30 years experience
They are expected to be in management roles.
Could you remind me what he has done since his jailbreaking day, i.e. in the last ten years, that can't be described as "setting VC money on fire?" Because my opinion would be, that yes, the world would have been better off if he did go to school.
We raised $18.1M and have made $28M in revenue to date.
Where are you getting your narrative? Are you confusing comma with someone else?
The entire point of having a structured approach to teaching (and this includes apprenticeships and the like), is to accelerate the process of learning.
Some people think that SGD was invented 10 years ago by ML “scientists”.
Everyone is a beginner at something, even OP.
Exploring curiousity, whether it's new to someone, or everyone is a good quality.
I wonder how many of the students who have attended these classes have applied all that math that's existed for so long into a self driving solution
However, instrumentation assumes that there’s a control regime that could actually control whatever the system is mechanically, and that’s generally not true.
So it’s almost a chicken and an egg problem where you can do instrumentation for non-autonomous-control systems in order to get state-action-reward data, but because you don’t actually have an actuated control system that you can specify and build mechanically, your targets for state-action-reward tuple aren’t the same
That is to say unless you’re actively collecting data from an autonomous system that’s being used non-autonomously then you’re not gonna be able to transition from a non-autonomous control regime to an autonomous control regime
Essentially let us say you have a world with data from a distribution X. Now you have a dataset with data from the distribution Y. Your problem boils down to the fact that when you run a model trained on the distribution Y, it regular ends up in states in the distribution {X-Y } for which it does not know what to imitate to, which leads it further into out of distribution land and so on till your car crashes. Now unless you limit the scope of X, you cannot come up with any real world guarantees, but in the real world there is no scope on X. In the real world tomorrow a road could cave, which would likely be so OOD, no model (or human driver) in the world can save you.
So really your only solution intuitively is if Y is so large and expansive, it models X for all the cases we care about (reaching human driver performance), or if your model is so big and generalizable that it generalizes to a good fraction of X being trained on Y. With deep learning we have seen generalization due to both more data and larger model with more data showing much greater improvements and that might be the only solution to this dilemma. Also a little bird told me, in Tesla’s latest end to end approach, they train the model, run it in a car with a driver they hired, the driver then intervenes at the last moment, they take the driver data and add it back to the dataset and do this as infinitum with their hope of solving self driving this way.
The problem with robotics, is even if you assume with data scale these problems are solved, in robots you are strictly limited in the model sizes you can use. Too big a model will not run in real time and make it effectively useless. But having a smaller model goes against all the trends we’ve seen in deep learning. This really just makes it exponentially harder to solve robotics compared to language or vision with our current paradigm of deep learning, I won’t put it out of the realm of possible though. A few years back I strongly thought language would not be solved this way and I could not be more wrong.
I'm just making stuff up, but I'm assuming this is because of teacher forcing itself? LLMs for eg model sentences by predicting the next word, but retains the past state via the KV cache. Unsure if these systems have a KV cache equivalency? I'm assuming LLMs are in a different regime due to the sheer amount of data as well.
Also I guess a simulation based system could work - ie by going around teacher forcing, and instead doing generation every say 30 steps, and making the dataset itself larger by training on the errors themselves. Sadly this means one has to make the simulation system. But maybe Comma uses some encoder CNN + some LSTM (or attention) based past state thingo? tbh I'm just making stuff up - ignore me - brain dump.
Out of the more recent advancements, I think ACT[2]s approach seems like the more interesting direction. Similar to MPC that predict(calculate) not only the next action but predict sequences of actions, discard all the actions except for one, at each loop.
Mapping the input to only the current action seems to be prone to error, while mapping the multiple steps is more robust.
My hunch is that there's probably some pretty fundamental limitations to our current methods (Transformers etc) though, which makes applying it to things such as language when error accumulation is essentially neglible (relatively speaking) fine but, to cars and robots maybe not.
But as you're a fellow Robot Learning (BC/LfD I'm assuming) practitioner, in the field of BC (Not IRL) what have been the more robust method for your case?
I mean two different Aloha setups (2 x 2x WidowX) in the same lab. This is the very minimal level of "generalization" you may expect, barely enough to be called generalization, but still, the learned policy seems to overfit to the individual robot's joint space quirks.
> in the field of BC (Not IRL) what have been the more robust method for your case?
Different things for different definitions of robust. What's your definition?
It's been interesting, for sure! More robust than ACT as long as you are not using action chunking (has the same problem as ACT). Downside is, it can be too slow/require too much data to train. I have a sneaking suspicion that at that scale of data even visual nearest neighbor stuff would be similarly robust.
Of course, no one has figured out an iota of useful generalization, which is sad across the field.
If you wrongly assume that the person is deriving an absolute steering angle from the image then you will fail.
If you correctly assume that the person is deriving a relative/correction steering angle from the image then you will succeed.
The problem is central to using ML tricks for anything: when you use maximum likelihood estimates for training anything, you get no guarantees for unlikely situations. That’s why you get self driving cars dragging someone stuck beneath them or LLMs just straight up spewing fiction.
If you want to drive in a straight line, then some Kalman filtering would help.
If you don't want to kill too many people, then AGI would help.
That's basically why all the RL approaches for game playing don't work nearly as well in the real world: because in the real world policy is a function of state, action, error.