>> In Minecraft, the team used a protocol that gave Dreamer a ‘plus one’ reward every time it completed one of 12 progressive steps involved in diamond collection — including creating planks and a furnace, mining iron and forging an iron pickaxe.
And that is why it is never going to work in the real world: games have clear objectives with obvious rewards. The real world, not so much.
For a more general system, you can annotate videos with text descriptions of all the tasks that have been accomplished and when, then train a reward model on those to later RL against.
Of course the delay is much bigger with working a job than most RL games but fundamentally it's the same problem.
Take children to school -> they safely arrive on time.
Autonomous driving -> arrive at destination without crashing.
Call centre -> customers are happy.
Or maybe there is some art to finding happiness in simple things like having folded clothes or surviving the commute?
I guess you can always find some well-specified, measurable goal/reward, but then that choice limits the performance of your model. It's fine when you're building a very specialized system; it gets more difficult the more general you're trying to be.
For a general system meant to operate in human environment, the goal ends up approaching "things that humans like". Case in point, that's what the overall LLM goal function is - continuations that make sense to humans, in fully-general meaning of that.
>> Take children to school -> they safely arrive on time.
>> Autonomous driving -> arrive at destination without crashing.
>> Call centre -> customers are happy.
Define a) "folded", b) "safely", c) "destination", d) "happy".
Also define the reward functions for each of the four objectives above.
Destination -> Like, close to the destination? I don't see how that's hard.
Happy -> you can use customer feedback for this
Folded -> this is indeed the trickiest one, but I think well within the capabilities of modern vision models.
Really? What about fires? Falling off cliffs? Causing others to crash?
Your "examples" are all hand-wavy and vague and no good to train an RL agent. You've also not provided a reward function.
I encourage you to read deepmind's work with robots.
>> Quantitatively, the QT-Opt approach succeeded in 96% of the grasp attempts across 700 trial grasps on previously unseen objects. Compared to our previous supervised-learning based grasping approach, which had a 78% success rate, our method reduced the error rate by more than a factor of five.
https://research.google/blog/scalable-deep-reinforcement-lea...
That was in 2018.
So what do you think, is vision-based robotic manipulation and grasping a solved problem, seven years later? Is QT-Opt now an established industry standard in training robots with RL?
Or was that just another project that was announced with great fanfare and hailed as a breakthrough that would surely lead to great increase of capabilities... only to pop, fizzle and disappear in obscurity without any real-world result, a few years later? Like most of DeepMind's RL projects do?
https://www.youtube.com/watch?v=x-exzZ-CIUw
It looks pretty awesome. Let's see what happens.
https://youtu.be/03p2CADwGF8?si=BXeWXqu1_3WMS4yy
A robot assembling a puzzle with machine vision!
And it's only from the 1970's.
But, like you say- let's wait and see. I always do the former but I'm still waiting for the latter.
Tell that to the people here who are trying to turn their startup ideas into money.
DeepSeek used RL to train R1, so that is clearly not true. But ignoring that, what is your alternative? Supervised learning? Good luck finding labels if you don’t even know what the objective is.
And why do I have to offer an alternative? If it's not working, it's not working, regardless of whether there's an alternative (that we know of) or not.