Ladder: Self-improving LLMs through recursive problem decomposition

I'll take a cold logical machine super-intelligence over the mad human lunatics wielding current iterations of "A.I." technologies in some really terrifyingly dangerous ways. As someone else commented on some other thread earlier "I look forward to being paperclips".

Also, do you think this is what O3 is doing?

sounds like MS :(

They had some killer research projects with various teams around the world, but eventually they all got snuffed.

https://www.pleacher.com/mp/mquotes/mobquote.html

We have done a lot of that improving historically by publishing research and textbooks. I can solve (some) problems today in minutes that would have stumped Isaac Newton for a life time (or at least a few weeks).

Of course, you are hinting at a more general distillation, I suspect.

> Curriculum learning: Training that begins with easy examples, gradually increasing difficulty.

> RL (Reinforcement Learning): Learning via trial-and-error with rewards, like training a robot or model to optimize actions.

> ~7y ago: ~7 years ago (circa 2018).

> OpenAI Gym days: Refers to using OpenAI Gym, a toolkit for RL, popular in robotics/AI research ~2016-2018.

> LLMs and robotics: Large Language Models (LLMs) now leverage RL techniques from robotics for better performance.

I think the last one is a semi-hallucinatory stretch. LLMs are large language models, ie. ChatGPT, Sonnet, Grok, R1. Robotics are ... robotics. Building robots.

The actual answer to what the comment is saying is that until maybe a year back, we trained language models - still with RL, but with RL on token error, which isn't "real" RL because it executes tasks "by coincidence". That is, it happens to be that when you train a model to predict text, it also gains the ability to do tasks in the bargain, because the text contains agents that do tasks. A year or so ago, we started training models by having them do a task, judging if the task was successful or failed, and then performing RL on task outcome rather than token prediction. This is a return to "classic RL", but we had to pass through the "token RL regime" first so that the model could make progress on realistic tasks at all. It also means that LLMs can now increasingly be employed in robotics, where task RL training rules, as there is no massive preexisting robotics movements dataset like there is for text.

(Also, NLP is Natural Language Processing, ie. what LLMs do.)

RL - Reinforcement Learning. You have a carrot and a stick. You run a model through iterations (in LLMs you generate n completions), you score each of them based on some reward functions, and if the result is correct you give it a carrot (positive reward), if the result is incorrect you give it a stick (negative or 0 reward). (simplified ofc)

OpenAI gym is (was?) an environment that allowed "AI agents" to be simulated in an environment. You could for example play games, or solve puzzles, or things like that. oAI gym was a "wrapper" over those environments, with a standardised API (observe, step (provide action), reward; rinse and repeat). You could for example have an agent that learned to land a lunar lander in a simple game. Or play chess. Or control a 3d stick figure in a maze.

It is long, but don't get scared off. He goes over a ton of different stuff related to model training, but makes it very easy to understand.