RLHF is just barely RL
twitter.com
twitter.com
I expect language models to also get crazy good at mathematical theorem proving. The search space is huge but theorem verification software will provide 100% accurate feedback that makes real reinforcement learning possible. It's the combination of vibes (how to approach the proof) and formal verification that works.
Formal verification of program correctness never got traction because it's so tedious and most of the time approximately correct is good enough. But with LLMs in the mix the equation changes. Having LLMs generate annotations that an engine can use to prove correctness might be the missing puzzle piece.
Well, to give an example: the complexity class NP is all about problems that have quick and simple verification, but finding solutions for many problems is still famously hard.
So there are at least some domains where this model would be a step forward.
And in any case, there are plenty of problems in NP that are not NP hard, too.
Yes, approximation is also an important aspect of many practical problems.
There's also lots of problems where you can easily specify one direction of processing, but it's hard to figure out how to undo that transformation. So you can get plenty of training data.
No, running it as a linear program is still slow.
I'm talking about small n=50 taking tens of minutes for a trivial linear program. Obviously the actual linear program is much bigger and scales quadratically in size, but still. N=50 is nothing.
But here again: solutions to your problem are easy to verify, so it might be interesting to let an AI have a go at solving it.
The interesting kind of programming is the kind where I'm figuring out what I'm building as part of the process.
Maybe AI will soon be superhuman in all the situations where we know exactly what we want (win the game), but not in the areas we don't. I find that kind of cool.
I've found when porting that the answers to these are sometimes not universal for a codebase, but rather you are best served considering case-by-case inside the code.
Although I suppose an AI agent could be created that holds a conversation with you and presents the options and acts accordingly.
Every decade it's 'we don't need programmers anymore'. Then it turns out specifying the problem needs programmers. Then it turns out the auto-coder can only reach a certain level of complexity. Then you've got real programmers modifying over-complicared code. Then everyone realizes they've wasted millions and it would have been quicker and cheaper to get the programmers to write the code in the first place.
The same will almost certainly happen with AI generated code for the next decade or two, just at a slightly higher level of program complexity.
I literally refuted this in my comment…
That being said, some kind of “no-code” is not necessarily a bad idea, as long as you treat it as just an abstraction for people who actually are programmers, like C versus assembly, or high level languages vs C.
In fact I worked for a train manufacturer that had a cool “no code” tool to program automated train control software with automated theorem proving built in, and it was much more efficient than there former Ada implementation especially when you factor the hiring difficulties in.
The same is true for math. There is a clear reward function when the goal is well-specified, e.g., "we need a sequence of mathematical statements that prove this other important mathematical statement is true."
>when the problem being solving is well-specified
I’d suggest that the Python code for your example prompt with reasonable defaults is not actually that far from the prompt itself in terms of the time necessary to write it.
However, add tricky details like how you want to handle connection pooling, differing retry strategies, short circuiting based on one of the results, business logic in the data combination step, and suddenly you’ve got a whole design doc in your prompt and you need a senior engineer with good written comms skills to get it to work.
I feel we're looking at the need for a measure of the computational complexity of problem specifications -- something like Kolmogorov complexity, i.e., minimum number of bits required, but for specifying instead of solving problems.
Ah, got it. I was just trying to keep my comment short!
Remember all those attempts to transform UML into code back in the day? This sounds sorta like that. I’m not a total genai naysayer but definitely in the “cautiously curious” camp.
I think generative AI is already a "really good autocomplete" and will get better in that respect, I can even see it generating good starting points, but I don't think in its current form it will replace the act of programming.
There’s the task of writing syntax, which is the mechanical overhead of the task of telling the computer what to do. People should focus on the latter (too much code is a symptom of insufficient automation or abstraction). Thankfully lots of people have CS degrees, not “syntax studies” degrees, right?
I feel like this is a very different type of programming, even if in some cases it would wind up being the same thing.
the reason why we spend time programming is because the problems in question are not easily defined, let alone the solutions.
Here I have assumed something about the set that f lives in. I am taking for granted that a probability measure can be defined. In theory, perhaps there are difficulties involving the various weird infinities that show up in computing, related to undecideability and incompleteness and such. But at a practical level, if we assume some concrete representation of the program then we can just define that it is smaller than some given bound, and ditto for a number of computational steps with a particular model of machine (even if fairly abstract, like some lambda calculus thing), so realistically we might be able to not worry about it.
Also, since our input and output sets are bounded (say, so many 64-bit doubles in, so many out), that also gives you a finite set of functions in principle; just think of the size of the (impossibly large) lookup table you'd need to represent it.
If you’re the most junior level, sure.
Anything above that, things get fuzzy, requirements change, biz goals shift.
I don’t really see this current wave of AI giving us anything much better than incremental improvement over copilot.
A small example of what I mean:
These systems are statistically based, so there’s no probability. Because of that, I wouldn’t even gain anything from having it write my tests since tests are easily built wrong in subtle ways.
I’d need to verify the test by reviewing it and, imo, writing the test would be less time than coaxing a correct one, reviewing, re-coaxing, repeat.
1) Addition of the natural numbers 2) equality of two real numbers
When you restrict your tools to perceptron based feed forward networks with high parallelism and no real access to 'common knowledge', the solution set is very restricted.
Basically what Gödel proved that destroyed Russel's plans for the Mathmatica Principia applies here.
Programmers can decide what is sufficient if not perfect in models.
What should the colleagues of the business person review before deciding that the system is fit for purpose? Or what should they review when the system fails? Should they go back over the transcript of the conversation with the LLM?
1) The business person made a mistake in their conversation/specification.
In this case the LLM will have generated code and tests that match the mistake. So all the tests will pass. The best way to catch this before it gets to production is to have someone else review the specification. But the problem is that the specification is a long trial-and-error conversation in which later parts may contradict earlier parts. Good luck reviewing that.
2) The LLM made a mistake.
The LLM may have made the mistake because of a hallucination which it cannot correct because in trying to correct it the same hallucination invalidates the correction. At this point someone has to debug the system. But we got rid of all the programmers.
That an LLM or a human is where the code comes from, doesn't make much difference.
Though it does kinda sound like you're assuming all LLMs must develop with Waterfall? That they can't e.g. use Agile? (Or am I reading too much into that?)
How do they do this? They can't trust the tests because the tests were also developed by the LLM which is working from incorrect information it received in a chat with the business person.
Mediocrely.
Sometimes the current process works, other times the planes fall out of the sky, or updates causes millions of computers to blue screen on startup at the same time.
LLMs in particular, and AI in general, doesn't need to beat humans at the same tasks.
Once the system can both test and update the spec etc to fix errors in the spec and build the program and ensure the result is satisfactory, we have AGI. If you argue an AGI could do it, then yeah it could as it can replace humans at everything, the argument was for an AI that isn't yet AGI.
Excel sheets are not fuzzy and underspecified.
> It's OK if a human employee is in the loop and has to intervenes sometimes
I've never worked on software where this was OK. In many cases it would have been disastrous. Most of the time a human employee could not fix the problem without understanding the software.
But we're not anywhere close to maximally automated. Today (many? most?) office workers do manual data entry and processing work that requires very little thinking. Even automating just 30% of their daily work is a huge win.
If you have those many well written tests, you can pass them to a constraint solver today and get your program. No LLM needed.
Or even run your tests instead of the program.
One very strong LLM could generate billions of tests alongside the working code and then train another smaller model, or feed it into the next iteration of training same the strong model. Strong LLMs do exist for that purpose, Nemotron 320B and Llama 3 450B.
It would be interesting if a dataset like that would be created like that, and then released as open source. Many LLMs proprietary or not, could incorporate the dataset in their training, and have on the internet hundreds of LLMs suddenly become much better at coding, all of them at once.
In other words, to what extent does Agile's ubiquity prove our competence in turning product goals into de facto reward functions?
Also look out for optimization the clever way.
Sounds like this work would involve asking questions to collaborators, guess some missing answers, write specs and repeat. Not that far ahead of the current sota of AI...
While writing simple functions may be mechanistic, being a developer is not.
'guess some missing answers' is why Waterfall, or any big upfront design has failed.
People aren't simply loading pig iron into rail cars like Taylor assumed.
The assumption of perfect central design with perfect knowledge and perfect execution simply doesn't work for systems which are for more like an organism than a machine.
Next gen LLMs, with their encyclopedic knowledge about the world, won't have that problem. They'll get the design correct on their first attempt because they're already familiar with the common pitfalls.
Of course we shouldn't expect LLMs to be a magic bullet that can program anything. But if your frame of reference is "visual programming" where the goal is to turn poorly thought out requirements into a reasonably sensible state machine then we should expect LLMs to get very good at that compared to regular people.
I keep looking for discoveries that show any movement there. But LLMs are still basically pattern matching and finding.
They can do impressive things, but they actually have no concept of what the 'right thing' even is, it is statistic not philosophy.
Why do you think they would need a more defined spec than a human?
There's nothing about AI that makes such operations intrinsically impossible, but they require much more than just the ability to generate working code.
Anything you don't define, is literally undefined behavior the same as in a compiler. The human will do something, and maybe you like it and maybe you don't.
A perfect spec is just another way to dedcribe a formal language, ie any programming language.
If you don't care what you get, then say little and say it ambiguously and pull the slot machine lever.
If you care what you get then you don't necessarily have to say a lot but you have to remove ambiguity, and then what you have is a spec, and if it's rigorous enough, it's a program, regardless what language and syntax is used to express it.
There is a difference between a human and an ai, and it is more than a difference of degrree, but filling in gaps with something that fits is not very significant. That can be done perfectly mechanistically.
In general, implementations can be vastly more complicated than even a complicated spec (e.g. by having to deal with real-world network failures, etc.), whereas a spec needs only to describe the expected behavior.
In this context, this is actually super useful, since defining the problem (writing a spec) is usually easier than solving the problem (writing an implementation); it's not just translating (compiling), and the engineer is now thinking at a higher level of abstraction (what do I want it to do vs. how do I do it).
However I agree that's the hard part. I can write a spec for finding the optimal solution to some combinatorial problem - where the naive code is trivial - a simple recursive function for example - but such a function would use near infinite time and memory.
In terms of the ML programme really being a compiler - isn't that in the end true - the ML model is a computer programme taking a spec as input and generating code as output. Sounds like a compiler to me.
I think the point of the AK post is to say the challenge is in the judging of solutions - not the bit in the middle.
So to take the writing software problem - if we had already sorted the computer programme validation problem there wouldn't be any bugs right now - irrespective of how the code was generated.
You haven't disproved this idea, merely re-stated the default obvious intuition that everyone is expected to have before being presented with this idea.
Their point is correct that defining a spec rigorously enough IS the actual engineering work.
A c or go program is nothing else but a spec which the compiler impliments.
There are infinite ways to impliment a given c expression in assembly, and doing that is engineering and requires a human to do it, but only once. The compiler doesn't invent how to do it every time the way a human would, the compiler author picked a way and now the compiler does that every time.
And it gets more complex where there isn't just one way to do things but several and the compiler actually chooses from many methods best fit in different contexts, but all of that logic is also written by some engineer one time.
But now that IS what happens, the compiler does it.
A software engineer no longer writes in assembly, they write in c or go or whatever.
I say I want a function that accepts a couple arguments and returns a result of a math formula, and it just happens. I have no idea how the machine actually impliments it, I just wrote a line of algebra in a particular formal style. It could have come right out of a pure math textbook and the valid c function definition syntax could just as well be pseudocode to describe a pure math idea.
If you tell an ai, or a human programmer for that matter, what you want in a rigorous enough format that all questions are answered, such that it doesn't matter what language the programmer uses or how the programmer impliments it, then you my friend have written the program, and are the programmer. The ai, or the human who translated that into some other language were indeed just the compiler.
It doesn't matter that there are multiple ways to impliment the idea.
It's true that one programmer writes a very inefficient loop that walks an entire array once for every element in the array, while another comes up with some more sophisticated index or vector or math trick approach, but that's not the definition of anything.
There are both simple and sophisticated compilers. You can already right now feed the the same c code into different compilers and get results that all work, but one is 100x faster than another, one uses 100x less ram than another, etc.
If you give a high level imprecise directive to an ai, you are not programming. If you give a high level precise directive to an ai, you are programming.
The language doesn't matter. What matters is what you express.
Also the reasoning is something business is dissonant about. The majority of planning and execution teams stick to processes. I see way more potential automating these than all parts in app production.
Business is going to have a hard time, when they believe, they alone can orchestrate some AI consoles.
It also seems less unique to code. You could also have a chat bot write an encyclopedia and see if the encyclopedias sold well. Chat bots could edit Wikipedia and see if their edits stuck as a reward function (seems ethically pretty questionable or at least in need of ethical analysis, but it is possible).
The maybe-easy to evaluate reward function is an interesting aspect of code (which isn’t to say it is the only interesting aspect, for sure!)
Certainly "compiled" is one reward (although a blank file fits that...) Another is test cases, input and output. This doesn't work on a software-wide scale but function-wide it can work.
In the future I think we'll see more of this test-driven development. Where developers formally define the requirements and expectations of a system and then an LLM (combined with other tools) generates the implementation. So instead of making the implementation, you just declaratively say what the implementation should do (and shouldn't).
There are objective criteria like 'compiles correctly' and 'passes self-designed tests' and 'is interpreted as correct by another LLM instance' which go a lot further than criteria that could be defined for most kinds of verbal questions.
Do you think it's possible that some version of LLM technology could discover new physical theories (that are experimentally verifiable), like for example a new theory of quantum gravity, by exploring the mathematical space?
Edit: this is just incredibly exciting to think about. I'm not an "accelerationist" but the "singularity" has never felt closer...
Where I think AI models might prove useful is in those cases where the problem is well defined, where formal methods can be used to validate the correctness of (partial) solutions, and where the search space is so large that work towards a proof is based on "vibes" or intuition. Vibes can be trained through reinforcement learning.
Some computer assisted proofs are already hundreds of pages or gigabytes long. I think it's a pretty safe bet that really long and convoluted proofs that can only be verified by computers will become more common.
Now, it is possible that it could come up with some theory that is radically different from current theories, where quantum gravity arises very naturally, and that fits all of the other predictions of of the current theories that we can measure - so we would have good reasons to believe the new theory and consider quantum gravity probably solved. But it's literally impossible to predict whether such a theory even exists, that is not mathematically equivalent to QM/QFT but still matches all confirmed predictions.
Additionally, nothing in AI tech so far predicts that current approaches should be any good at this type of task. The only tasks where AI has truly excelled at are extremely well defined problems where there is a huge but finite search space; and where partial solutions are easy to grade. Image recognition, game playing, text translation are the great successes of AI. And performance drops sharply with the uncertainty in the space, and with the difficulty of judging a partial solution.
Finding physical theories is nothing like any of these problems. The search space is literally infinite, partial solutions are almost impossible to judge, and even judging whether a complete solution is good or not is extremely difficult. Sure, you can check if it's mathematically coherent, but that tells you nothing about whether it describes the physical world correctly. And there are plenty of good physical theories that aren't fully formally proven, or weren't at the time they were invented - so mathematical rigour isn't even a very strong signal (e.g. Newton's infinitesimal calculus wasn't considerered sound until the 1900s or something, by which time his theories had long since been rewritten in other terms; the Dirac delta wasn't given a precise mathematical definition until much later than it's uses; and I think QFT still uses some iffy math even today).
This is a very good point I think a lot of people miss. (Including some who should know better.) Pontificating about speculative physics is all right for Aristotle but you need actual experiments to ground your results.
Better, when we spot mistakes even if we couldn't create the work with the error. Think art: most of us can't draw hands, but we can spot when Stable Diffusion gets them wrong.
Worse also, because there are many things which are "common sense" and wrong, e.g. https://en.wikipedia.org/wiki/Category:Paradoxes_in_economic..., and we would collectively down-vote a perfectly accurate model of reality for violating our beliefs.
And sure, coverage and lints are objective metrics, but they don't directly imply the correctness of a test. Some tests can reach a high coverage and pass all the lint checks but still be incorrect or test the wrong thing!
Whether the test passes or not is what's mostly correlated to whether it's correct or not. But similarly for an image recognizer the prompt of whether an image is a flower or not is also objective and correlated, and yet researchers continue to find adversarial inputs for image recognizer due to the bias in their training data. What makes you think this won't happen here too?
So are rules for the game of go or chess ? Specifying code that satisfies (or doesn't satisfy) is a problem statement - evaluation is automatic.
> but they don't directly imply the correctness of a test.
I'd be willing to bet that if you start with an existing coding model and continue training it with coverage/lint metrics and evaluation as feedback you'd get better at generating tests. Would be slow and figuring out how to build a problem dataset from existing codebases would be the hard part.
The rules are well defined and you can easily write a program that will tell whether a move is valid or not, or whether a game has been won or not. This allows you generate virtually infinite amount of data to train the model on without human intervention.
> Specifying code that satisfies (or doesn't satisfy) is a problem statement
This would be true if you fix one specific program (just like in Go or Chess you fix the specific rules of the game and then train a model on those) and want to know whether that specific program satisfies some given specification (which will be the input of your model). But if instead you want the model to work with any program then that will have to become part of the input too and you'll have to train it an a number of programs which will have to be provided somehow.
> and figuring out how to build a problem dataset from existing codebases would be the hard part
This is the "Human Feedback" part that the tweet author talks about and the one that will always be flawed.
In the end, your are replacing the application code by a spec, which needs to have a comparable level of detail in order for the AI to not invent its own criteria.
It's more interesting to imagine what just might work. One thing that has plagued programmers for the past decades is the difficulty of writing correct multi-threaded software. You need fine-grained locking otherwise your threads will waste time waiting for mutexes. But color-coding your program to constrain which parts of your code can touch which data and when is tedious and error-prone. If LLMs can annotate code sufficiently for a SAT solver to prove thread safety that's a huge win. And that's just one example.
It's not even about LLMs IMHO. It's about letting a computer crunch many numbers and find a pattern in the results, in a quasi religious manner.
This is interesting, but doesn't it still need supervision? Why wouldn't it generate tests for properties you don't want? It seems to me that it might be able to "fill in gaps" by generalizing from "typical software", like, if you wrote a container class, it might guess that "empty" and "size" and "insert" are supposed to be related in a certain way, based on the fact that other peoples' container classes satisfy those properties. And if you look at the tests it makes up and go, "yeah, I want that property" or not, then you can steer what it's doing, or it can at least force you to think about more cases. But there would still be supervision.
Ah -- here's an unsupervised thing: Performance. Maybe it can guide a sequence of program transformations in a profile-guided feedback loop. Then you could really train the thing to make fast code. You'd pass "-O99" to gcc, and it'd spin up a GPU cluster on AWS.
write performance oriented and memory safe C++ code. Current coding assistants are glorified autocomplete for unit tests or short api endpoints or what have you but if you have to write any safety oriented code or you have to think about what the hardware does it's unusable.
I tried using several of the assistants and they write broken or non-performant code so regularly it's irresponsible to use them.
I imagine it's great for CRUD apps and generating unit tests, but for anything reliable where I work, it's not even close to being useful at all, let alone a game changer. It's a shame, because it's not like I really enjoy fiddling with memory buffers and painstakingly avoiding UB, but I still have to do it (I love Rust, but it's not an option for me because I have to support AIX. V8 in Rust also sounds like a nightmare, to be honest. It's a very C++ API).
The best are shockingly good… so long as their context doesn't expire and they forget e.g. the Vector class they just created has methods `.mul(…)` rather than `.multiply(…)` or similar. Even the longer context windows are still too short to really take over our jobs (for now), the haystack tests seem to be over-estimating their quality in this regard.
The worst LLM's that I've seen (one of the downloadable run-locally models but I forget which) — one of my standard tests is that I ask them to "write Tetris as a web app", and it started off doing something a little bit wrong (square grid), before giving up on that task entirely and switching from JavaScript to python and continuing by writing a script to train a new machine learning model (and people still ask how these things will "get out of the box" :P)
People who see more of the latter? I can empathise with them dismissing the whole thing as "just autocomplete on steroids".
Will this be able to be done without spending absurd amounts of energy?
Even if they were, a human-quality AI that runs at human-speed for x10 our body's calorie requirements in electricity, would still (at electricity prices of USD 0.1/kWh) undercut workers earning the UN abject poverty threshold.
Then the LLM generates and iterates on the code until it passes all of the tests. New requirements? Add more tests and repeat.
This would be legitimately paradigm shifting, vs. the super charged auto complete driven by LLMs we have today.
Indeed, systems like AlphaProof / AlphaGeometry are already able to win a silver medal at the IMO, and the former relies on Lean for theorem verification [1]. On the open source side, I really like the ideas in LeanDojo [2], which use a form of RAG to assist the LLM with premise selection.
[1] https://deepmind.google/discover/blog/ai-solves-imo-problems...
Remember: The reason we need RLHF at all is that we cannot write a loss function for what makes a good answer. There are just many ways a good answer could look like, which cannot be calculated on the basis of next-token-probability.
So you start by having your vanilla model generate n completions for your prompt. You the. manually score them. And then those prompt => (completion,score) pairs become your training set.
Once the model is trained, you may find that you can cheat:
Because if you include the desired score in your prompt, the model will now strive to produce an answer that is consistent with that score.
But you need a model to generate score from answer, and then fine-tune another model to generate answer conditioned on score. The first time the score is at the end and the second time at the beginning. It's how DecisionTransformer works too, it constructs a sequence of (reward, state, action) where reward conditions on the next action.
https://arxiv.org/pdf/2106.01345
By the same logic you could generate tags, including style, author, venue and date. Some will be extracted from the source document, the others produced with classifiers. Then you can flip the order and finetune a model that takes the tags before the answer. Then you got a LLM you can condition on author and style.
To mitigate this, you have to have a system outside of that, which penalizes "gaming" the reward function. This system has to have some idea of what real value is, to be able to spot cases where the reward function is high but the value is low. We have a hard enough time of this in the money economy, where we've been learning for centuries. I do not think we are anywhere close in neural networks.
There is a step like this in ML. I think it's pretty interesting that topics from things like economics pop up in ML - although perhaps it's not too surprising as we are doing ML for humans to use.
You can “favorite” comments on HN to bookmark them.
This is probably the most cursed problem ever.
Assume you could develop such a system, why wouldn't you just incorporate its logic into the original fitness function and be done with it?
I think the answer is that such a system can probably never be developed. At some level humans must be involved in order to adapt the function over time in order to meet expectations as training progresses.
The information used to train on is beyond critical, but heuristics regarding what information matters more than other information in a given context might be even more important.
In any reward function, either there are valuable things that are not rewarded, or unvaluable things that are. But having multiple systems to evaluate this, does help.
Go is a game that is fundamentally too complex for humans to solve. We've known this since way back before AlphaGo. Since humans were not the perfect Go players, we didn't use them to teach a model- we wanted the model to be able to beat humans.
I dont see language being comparable. the "perfect" LLM imitates humans perfectly, presumably to the point where you can't tell the difference between LLM generated text, and human generated text. Maybe it's just as flexible as the human mind is too, and can context switch quickly, and can quickly swap between formalities, tones, and slangs. But the concept of "beating" a human doesn't really make much sense.
AlphaGo and Stockfish can push forward our understandings of their respective games, but an LLM cant push forwards our boundary of language. this is because it's fundamentally a copy-cat model. This makes RLHF make much more sense in the LLM realm than the Go realm.
Maybe, by chance, the model produces an output that is a little better than its average; that is, better than professional researchers. This will be ranked favorably in RLHF.
Repeat this process and the model slowly but surely surpasses the best humans.
Is such a scenario possible in practice?
When I ask chatgpt for code I don't want them to imitate humans, I want them to be better than humans. My reward function should then be code that actually works, not code that is similar to humans.
To clarify, current LLMs (without RLHF, etc.) have a very straightforward objective function during training, which is to minimize the cross-entropy of token prediction on the training data. If we assume that our training data is sampled from a population generated via a finite computable model, then Solomonoff induction achieves optimal sequence prediction.
Assuming we had an oracle that could perform SI (since it’s uncomputable), how different would conversations between GPT4 and SI be, given the same training data?
We know there would be at least a few notable differences. For example, we could give SI the first 100 digits of pi, and it would give us as many more digits as we wanted. Current transformer models cannot (directly) do this. We could also give SI a hash and ask for a string that hashes to that value. Clearly a lot of hard, formally-specified problems could be solved this way.
But how different would SI and GPT4 appear in response to everyday chit-chat? What if we ask the SI-based sequence predictor how to cure cancer? Is the “most probable” answer to that question, given its internet-scraped training data, an answer that humans find satisfying? Probably not, which is why AGI requires something beyond just optimal sequence prediction. It requires a really good objective function.
My first inclination for this human-oriented objective function is something like “maximize the probability of providing an answer that the user of the model finds satisfying”. But there is more than one user, so over which set of humans do we consider and with which aggregation (avg satisfaction, p99 satisfaction, etc.)?
So then I’m inclined to frame the problem in terms of well-being: “maximize aggregate human happiness over all time” or “minimize the maximum of human suffering over all time”. But each of these objective functions has notable flaws.
Karpathy seems to be hinting toward this in his post, but the selection of an overall optimal objective function for human purposes seems to be an incredibly difficult philosophical problem. There is no objective function I can think of for which I cannot also immediately think of flaws with it.
Imagine if you could give a model "how you think" and your knowledge, experiences, and values as context, then it's "Explain Like I'm 5" on steroids. Both exciting and terrifying at the same time.
That was sort of implicit in my first suggestion for an objective function, but do you really want the model to be optimal on a per-user basis? There’s a lot of bad people out there. That’s why I switched to an objective function that considers all of humanity’s needs together as a whole.
Both things can co-exist without being in conflict of each other.
My (hot) take is I personally don't believe that any LLM that can fit on a single GPU is capable of significant harm. An LLM that fits on an 8xH100 system perhaps, but I am more concerned about other ways an individual could spend ~$300k with a conviction of harming others. Besides, looking up how to make napalm on Google and then actually doing it and using it to harm others doesn't make Google the one responsible imo.
You defined your predictor as being able to minimize mathematical definitions following some unspecified algebra, why didn't you define it being able to run chemical and pharmacological simulations through some unspecified model too?
I suspect that a lot of LLM prompts that elicit useful capabilities out of imperfect sequence predictors like GPT-4 are in fact most likely to show up in the context of "prompting an LLM" rather than being likely to show up "in the wild".
As such, to predict the token after a prompt like that, an SI-based sequence predictor would want to predict the output of whatever language model was most likely to be prompted, conditional on the prompt/response pair making it into the training set.
If the answer to "what model was most likely to be prompted" was "the SI-based sequence predictor", then it needs to predict which of its own likely outputs are likely to make it into the training set, which requires it to have a probability distribution over its own output. I think the "did the model successfully predict the next token" reward function is underspecified in that case.
There are many cases like this where the behavior of the system in the limit of perfect performance at the objective is undesirable. Fortunately for us, we live in a finite universe and apply finite amounts of optimization power, and lots of things that are useless or malign in the limit are useful in the finite-but-potentially-quite-large regime.
If Karpathy were to focus on automating LEAN proofs it could change mathematics forever.
> AlphaProof is a system that trains itself to prove mathematical statements in the formal language Lean. It couples a pre-trained language model with the AlphaZero reinforcement learning algorithm, which previously taught itself how to master the games of chess, shogi and Go
https://deepmind.google/discover/blog/ai-solves-imo-problems...
The entire point of alpha zero was to eliminate this human influence, and go with pure reinforcement learning (i.e. zero human influence).
Unlike the above, open problems can't be solve by computing (in combinatorial senses). Even humans can only try, and LLMs do spew out something that would most likely work, not something inherently correct.
And here I mean that it the act of making a single (plausible) move that must match the expressiveness of language, because otherwise you're not in the domain of Go but the far less interesting "I have a 19x19 pixel grid and two colours".
The part that is missing from Karpathy's rant is "at scale" (the researchers only ran 3 iterations of the algorithm on small language models) and in "open domains" (I could be wrong about this but IIRC they ran their games on a small number of common english words). But adversarial language games seem promising, at least.
I am not sure how that works at the prediction stage as language isn’t the problem here.
(by the way, I don't think "debating" is the right term for the SPAG game - it's quite subtle and isn't about arguing for a point, or rhetoric, or anything like that)
https://en.m.wikipedia.org/wiki/Reinforcement_learning_from_...
For actual reinforcement learning with a feedback loop that aims to increase overall performance the current techniques are SPPO and Meta's version of it [0] that slightly outperforms it. It involves using a larger LLM as a judge though, so the accuracy of the results is somewhat dubious.
I have to use coding as an example, because that's 95% of my use cases. I type in a general statement of the problem I'm having and within seconds, I get back a response that speaks my language and provides me with some information to ingest.
Now, I don't know for sure if everything sentence I read in the response is correct, but let's say that 75% of what I read aligns with what I currently know to be true. If I were to ask a real expert, I'd possibly understand or already know 75% of what they're telling me, as well, with the other 25% still to be understood and thus trusting the expert.
But either with AI or a real expert, for coding at least, that 25% will be easily testable. I go and implement and see if it passes my test. If it does, great. If not, at least I have tried something and gotten farther down the road in my problem solving.
Since AI generally does that for me, I am convinced of their helpfulness because it moves me along.
Advertisement and propaganda is not necessarily helpful for consumers, but just needs to be convincing in order to be helpful for producers.
If what you want it AGI then design an architecture with the necessary moving parts! Current approach reminds of the joke of the drunk looking for his dropped cars keys under the street lamp because "it's bright here", rather than near where he actually dropped them. It seems folk have spent years trying to come up with alternate learning mechanisms to gradient descent (or RL), and having failed are now trying to use SGD/pre-training for AGI "because it's what we've got", as opposed to doing the hard work of designing the type of always-on online learning algorithm that AGI actually requires.
The tide of money flow means we are probably locked into transformers for some time. There will be transformer ASICs built for example in droves. It will be hard to compete with the status quo. Transformer architecture == x86 of AI.
RL is one way to implement goal directed behavior (making decisions now that hopefully will lead towards a later reward), but I doubt this is the actual mechanism at play when we exhibit goal directed behavior ourselves. Something more RL-like may potentially be used in our cerebellum (not cortex) to learn fine motor skills.
Some of the things that are clearly needed for human-like AGI are things like the ability to learn incrementally and continuously (the main ways we learn are by trial and error, and by copying), as opposed to pre-training with SGD, things like working memory, ability to think to arbitrary depth before acting, innate qualities like curiosity and boredom to drive learning and exploration, etc.
The Transformer architecture underlying all of today's LLMs have none of the above, not surprising since it was never intended as a cognitive architecture - it was designed for seq2seq use such as language models (LLMs).
So, no, I don't think RL is the answer to AGI, and note that DeepMind who had previously believed that have since largely switched to LLMs in the pursuit of AGI, and are mostly using RL as part of more specialized machine learning applications such as AlphaGo and AlphaFold.
Thinking to arbitrary depth sounds like Monte Carlo tree search? Which is often implemented in conjunction with RL. And working memory I think is a matter of the architecture you use in conjunction with RL, agree that transformers aren't very helpful for this.
I think what you call 'trial and error', is what I intuitively think of RL as doing.
AlphaProof runs an RL algorithm during training, AND at inference time. When given an olympiad problem, it generates many variations on that problem, tries to solve them, and then uses RL to effectively finetune itself on the particular problem currently being solved. Note again that this process is done at inference time, not just training.
And AlphaProof uses an LLM to generate the Lean proofs, and uses RL to train this LLM. So it kinda strikes me as a type error to say that DeepMind have somehow abandoned RL in favour of LLMs? Note this Demis tweet https://x.com/demishassabis/status/1816596568398545149 where it seems like he is saying that they are going to combine some of this RL stuff with the main gemini models.
I hadn't read that paper, but yes using prediction failure as learning signal (and attention mechanism), same as we do, is what I had in mind, but it seems that to be useful it needs to be combined with online learning ability, so that having explored then next time one's predictions will be better.
It's easy to imagine LLM's being extended in all sorts of ad-hoc ways, including external prompting/scaffolding such as think step by step and tree search, which help mitigate some of the architectural shortcomings, but I think online learning is going to be tough to add in this way, and it also seems that using the model's own output as a substitute for working memory isn't sufficient to support long term focus and reasoning. You can try to script intelligence by putting the long-term focus and tree search into an agent, but I think that will only get you so far. At the end of the day a pre-trained transformer really is just a fancy sentence completion engine, and while it's informative how much "reactive intelligence" emerges from this type of frozen prediction, it seems the architecture has been stretched about as far as it will go.
I wasn't saying that DeepMind have abandoned RL in favor of LLMs, just that they are using RL in more narrow applications than AGI. David Silver at least still also seems to think that "Reward is enough" [for AGI], as of a few years ago, although I think most people disagree.
To be clear, i'm not claiming that you take an LLM and do some RL on it and suddenly it can do particular tasks. I'm saying that if you train it from scratch using RL it will be able to do certain well defined formal tasks.
Idk what you mean about the online learning ability tbh. The paper uses it in the exact way you specify, which is that it uses RL to play montezuma's revenge and gets better on the fly.
Similar to my point about the inference time RL ability of the alphaProof LLM. That's why I emphasized that RL is done at inference time, like each proof you do it uses to make itself better for next time.
I think you are taking LLM to mean GPT style models, and I am taking LLM to mean transformers which output text, and they can be trained to do any variety of things.
Narrow skills like playing Chess (DeepBlue), Go, or math proofs are impressive in some sense, but not the same as generality and/or intelligence which are the hallmarks of AGI. Note that AlphaProof, as the same suggests, has more in common with AlphaGo and AlphaFold than a plain transformer. It's a hybrid neuro-symbolic approach where the real power is coming from the search/verification component. Sure, RL can do some impressive things when the right problem presents itself, but it's not a silver bullet to all machine learning problems, and few outside of David Silver think it's going to be the/a way to achieve AGI.
At some point we need to start recognizing LLMs for what they are and stop making outlandish claims like this. A moment of reflection ought to reveal that “open domain problem solving” is not what an LLM does.
An LLM, could not, for example, definitively come up with the three laws of planetary motion like Kepler did (he looked at the data), in the absence of a prior formulation of these laws in the training set.
TFA describes a need for scoring, at scale, qualitative results to human queries. Certainly that’s important (it’s what Google is built upon), but we don’t need to make outlandish claims about LLM capabilities to achieve it.
Or maybe we do if our next round of funding depends upon it.
It seems to me RL was quite revolutionary (especially with protein folding/AlphaGo) - but using a minimal form of it to solve a training (not prediction) problem seems rather like bringing a bazooka to a banana fight.
Using explore/exploit methods to search potential problem spaces might really help propel this space forward. But the energy requirements do not favor the incumbents as things are now scaled to the current classic LLM format.
You could use Symbolic Regression instead, and the LLM will write the code. Under the hood it would use a genetic programming library like with SymbolicRegressor.
Found a reference:
> AI-Descartes, an AI scientist developed by researchers at IBM Research, Samsung AI, and the University of Maryland, Baltimore County, has reproduced key parts of Nobel Prize-winning work, including Langmuir’s gas behavior equations and Kepler’s third law of planetary motion. Supported by the Defense Advanced Research Projects Agency (DARPA), the AI system utilizes symbolic regression to find equations fitting data, and its most distinctive feature is its logical reasoning ability. This enables AI-Descartes to determine which equations best fit with background scientific theory. The system is particularly effective with noisy, real-world data and small data sets. The team is working on creating new datasets and training computers to read scientific papers and construct background theories to refine and expand the system’s capabilities.
https://scitechdaily.com/ai-descartes-a-scientific-renaissan...
Sampling board state from an abstract board space isn't a statistical inference problem. There's no missing information.
The whole edifice of science is a set of experimental and inferential practices to overcome the massive information gap between the state of a measuring device and the state of what, we believe, it measures.
In the case of natural language the gap between a sequence of symbols, "the war in ukraine" and those aspects of the world these symbols refer to is enormous.
The idea that there is even a RL-style "reward" function to describe this gap is pseudoscience. As is the false equivocation between sampling of abstracta such as games, and measuring the world.
If you have a process, eg., points = sample(circle) which fully describes its target as n->inf (ie., points = circle as n->inf) you arent engaged in statistical infernece. You might be using some of the same formula, but the whole system of science and statistics has been created for a radically different problem with radically different semantics to everything you're doing.
eg., the height of mercury in a thermometer never becomes the liquid being measured.. it might seems insane/weird/obvious to mention this... but we literally have berkelian-style neoidealists in AI research who don't realise this...
Who think that because you can find representations of abstracta in other spaces they can be projected in.. that this therefore tells you anything at all about inference problems. As if it was the neural network algorithm itself (a series of multiplications and additions) that "revealed the truth" in all data given to it. This, of course, is pseudoscience.
It only applies on mathematical problems, for obvious reasons. If you use a function approximation alg to approximate a function, do not be suprised you can succeed. The issue is that the relationship between, say, the state of a theremometer and the state of the temperature of it's target system is not an abstract function which lives in the space of temperature readings.
More precisely, in the space of temperature readings the actual causal relationship between the height of the mecurary and the temperature of the target shows up as an infinite number of temperature distributions (with any given trained NN learning only one of these). None of which are a law of nature -- laws of nature are not given by distributions in measuring devices.
It just took decades and impressive breakthroughs to solve, I wouldn't really call it "trivial". However, I do agree with you that they're a class of problem different from problems with no clear objective function, and probably much easier to reason about that.
As in, if i need to infer the radius of a circle from N points sampled from that cirlce.. yes, I'm sure there's a textbook of algorithms/etc. with a lot of work spent on them.
But in the sense of statistical inference, you're only learning a property of a distribution given that distribution.. there isn't any inferential gap. As N->inf, you recover the entire circle itself.
compare with say, learning the 3d structure of an object from 2d photographs. At any rotation of that object, you have a new pixel distribution. So in pixel-space a 3d object is an infinite number of distributions; and the inference goal in pixel-space is to choose between sets of these infinities.
That's actually impossible without bridging information (ie., some theory). And in practice, it isn't solved in pixel space... you suppose some 3d geometry and use data to refine it. So you solve it in 3d-object-property-space.
With AI techniques, you have ones which work on abstracta (eg., circles) being used on measurement data. So you're solving the 3d/2d problem in pixel space, expecting this to work because "objects are made out of pixels, arent they?" NO.
So there's a huge inferential gap that you cannot bridge here. And the young AI fantatics in research keep milling out papers showing that it does work, so long as its a cirlce, chess, or some abstract game.
AGI won’t come from this set of tools. Sam Altman just wants to buy himself a few years of time to find their next product.
ie., that when you're taking data from a therometer in order to estimate the temperature of coffee, the issue isnt simply partial information
Its that the information is about the mercury, not the coffee. In order to bridge the two you need a theory (eg., about the causal reliability of heating / room temp / etc.)
So this isnt just a partial/full information problem -- these are still mathematical toys. This is a reality problem. This is a you're dealing with a causal realtionship between physical systems problem. This is not a mathematical relationship. It isnt merely partial, it is not a matter of "informaton" at all. No amount could ever make the mecurary, coffee.
Computer scientists have been trained on mathematics and deployed as social scientists, and the naiveté is incredible
The multi-trillion dollar question is: What is the objective reward that would induce AI models like LLMs to behave like AGI -- while adhering to all the limits we human beings wish to impose in AGI behavior?
I don't think anyone has even a faint clue of the answer yet.
No, the reward for finding the right objective function is a good future for all of humanity, given that we already have an algorithm for AGI.
The objective function to acquire trillions of dollars is trivial: it’s the same minimization of cross-entropy that we already use for sequence prediction. What’s missing is a better algorithm, which is probably a good thing at the moment, because otherwise someone could trivially drain all value from the stock market.
The phrase "the multi-trillion dollar question" has nothing to do with acquiring trillions of dollars.
The phrase is idiomatic, indicating a crucial question, like "the $64,000 question," but implying much bigger stakes.[a]
---
[a] https://idioms.thefreedictionary.com/The+%2464%2c000+Questio...
By the way, I agree with you that "a good future for all of humanity" would be a fantastic goal :-)
The multi-trillion dollar question is: How do you specify that goal as an objective function?
Today's Transformer-based LLMs are just what the name says - (Large) Language Models - fancy auto-complete engines. They are not a full blown cognitive architecture.
I think many people do have a good idea how to build cognitive architectures, and what the missing parts are that are needed for AGI, and some people are working on that, but for now all the money and news cycles are going into LLMs. As Chollet says, they have sucked all the oxygen out of the room.
Yes, RLHF is barely RL, but you wouldn't use human feedback to drive a Go game unless there was no better alternative; and in RL, finding a good reward function is the name of the game; once you have that, you have no reason to prefer human feedback, especially if it is demonstrably worse. So, no, nobody would actually "prefer RLHF over RL" given the choice.
But for language models, human feedback IS the ground truth (at least until we find a better, more mathematical alternative). If it weren't and we had something better, then we'd use that. But we don't. So no, RLHF is not "worse than RL" in this case, because there 'is' no 'other' RL in this case; so, here, RLHF actually is RL.
"Is it the same for this s y of inputs?" May be fine for a subset of things, but then that's a binary thing. If it's slightly wrong do you score by number of outputs that match? A purely binary thing gives little useful help for nudging a model in the right direction. How do you compare two that both work, which is more "idiomatic"?
[0] https://deepmind.google/discover/blog/alphadev-discovers-fas...
for testing, if you use something like quickcheck, you might find bugs that you wouldn't otherwise find.
when it comes to idiomatic, I'm not sure - but if we're at the point that gpt is writing code that works, do we really care? as long as this code is split into many small pieces, we can just replace the piece instead of trying to understand/fix it if we can't read it. in fact, maybe there's a better language that is human readable but better for transformers to write and maintain.
> when it comes to idiomatic, I'm not sure - but if we're at the point that gpt is writing code that works, do we really care?
Yes - it's certainly preferable. You may prefer working over neat, but working and neat over working but insane spaghetti code.
Remember this is about training the models, not about using them later. How do we tell, while training, which option was better to push it towards good results?
function add(a,b) {
return 4
}
The approach I was thinking of is that assuming we start with the Java program:
public class Addition {
public static int add(int a, int b) {
return a + b;
}
}
public class Addition {
public static int add(int a, int b) {
return a + b;
}
public static class AdditionAssert {
private int a;
private int b;
public AdditionAssert a(int a) {
this.a = a;
return this;
}
public AdditionAssert b(int b) {
this.b = b;
return this;
}
public void assertExpected(int expected) {
int result = add(a, b);
assert result == expected : "Expected " + expected + " but got " + result;
System.out.println("Assertion passed for " + a + " + " + b + " = " + result);
}
}
public static void main(String[] args) {
new AdditionAssert().a(5).b(3).assertExpected(8);
new AdditionAssert().a(-1).b(4).assertExpected(3);
new AdditionAssert().a(0).b(0).assertExpected(0);
System.out.println("All test cases passed.");
}
}
def add(a, b):
return 4
def addition_assert(a, b, expected):
result = add(a, b)
assert result == expected, f"Expected {expected} but got {result}"
addition_assert(a=5, b=3, expected=8)
addition_assert(a=-1, b=4, expected=3)
addition_assert(a=0, b=0, expected=0)
Sure, a board game with an objective winning function at which computers are already better than humans won't get much from RLHF. That doesn't look like a big surprise.
On the other hand, a LLM trained on lots of not-so-much curated data will naturally pick up mistakes from that dataset. It is not really feasible or beneficial to modify the dataset exhaustively, so you reinforce the behaviour that is expected at the end. An example would be training an AI in a specific field of work: it could repeat advices from amateurs on forums, when less-known professional techniques would be more advisable.
Think about it like kids naturally learning swear words at school, and RLHF like parents that tell their kids that these words are inappropriate.
The tweet conclusion seems to acknowledge that, but in a wishful way that doesn't want to concede the point.
If you want to exceed human intelligence, then design architectures for intelligence, not for copying humans!
So there are 3 levels of optimization in discussion here:
1. for the next token (NTP)
2. for a single turn response (RLHF)
3. for actual task completion or long-term objectives (RL)
So, I’m defaulting on RLHF is great in at least those ways until an alternative is empirically proven to be better. I also hope for larger, better, open-source collections of RLHF training data.
https://www.surgehq.ai/case-studies/anthropic-claude-surgeai...
And if you don't have a way to say that something could be exhausted, then you will look for heuristics to choose more profitable places to search. Hence the HF added.
Human expectation/preference alignment is the explicit goal, not a way to achieve something else. RLHF (or an alternative such as ORPO) is used to take a raw pre-trained foundation model, which by itself is only going to try to match training set statistics, and finetune it to follow human expectations for uses such as chat (incl. Q&A).
Like, I don't know what you think is wrong on that? The human/expert feedback is there to provide scores that we don't know how to fully codify, yet. Is effectively acknowledging that we don't know how to codify the "I know it when I see it" rule. And based on those scores, the model updates and new things can be scored.
What is not accurate in that description?
Note that using codified-HF (as is in fact already done - the actual HF being first used to train a proxy reward model) doesn't change things here - tuning the model to maximize this metric of human usability IS the goal. The idea of using RL is to do the search at training time rather than inference time when it'd be massively more expensive. You can think of all the multi-token model outputs being evaluated by the reward model as branches of the search tree, and the goal of RL is to influence earlier token selection to lead towards these preferred outputs.
And I should say that I said "codified" but I don't mean just code. Labeled training samples is fine here. Doesn't change that finding a model that will give good answers is ultimately something that can be conceptualized as a search.
You are also blurring the reinforcement/scoring at inference time as compared to the work that is done at training time? The idea of using RL at training time is not just because it is expensive there. The goal is to find the policies that are best to use at inference time.
I preferred Zitron’s point* that we would need to invent several branches of science to solve this problem, but it’s good to see the point made tweet-sized.
> OpenAI needs at least $5 billion in new capital a year to survive. This would require it to raise more money than has ever been raised by any startup in history
They were probably toast before, but after Zuck decided to take it personally and made free alternatives for most use cases they definitely are, since if they had any notable revenue from selling API access it will just keep dropping.
I have to wonder how long Anthropic can remain independent too, although at least they don't have toxic workplace and founder exodus issues to deal with. Maybe they get acquired by Amazon?
The article has a single point it repeats over and over again: OpenAI (and "generative AI as a whole"/"transformer-based models") are too expensive to run, and it's "close to impossible" for them to either limit costs or increase revenue. This is because "only 5% of businesses report using the technology in production", and that the technology had no impact on "productivity growth". It's also because "there's no intelligence in it", and the "models can't reason". Oh, also, ChatGPT is "hard to explain to a layman".
All that is liberally sprinkled with "I don't know, but"s and absolutely devoid of any historical context other than in financial terms. No technical details. Just some guesses and an ironclad belief that it's impossible to improve GPTs without accessing more data than there is in existence. Agree or disagree; the article is not worth wading through so many words: others made arguments on both sides much better and, crucially, shorter.
I don’t think have a single overall thesis is the same thing as repeating oneself. For example “models can’t reason” has nothing at all to do with cost.
Anyway, it's just my opinion: to me, the length of the article was artificially increased to the point where it wasn't worth my time to read it. As such, unfortunately, I'm not inclined to spend any more time discussing it - I just posted my takeaways and a warning for people like me. If you liked the article, good for you.
> "models can't reason" has nothing at all to do with cost.
Yeah, that one falls under "no technical details".
Technical details about cost? Or technical details about how models can’t reason?
For example I don’t really need “technical details” to confidently assert that Quicken cannot do ray tracing. Everyone is welcome to try it, and exactly zero people will report being able to use Quicken’s ray tracing abilities in an automated business-critical way.
There isn’t an enormous burden of proof required to back up a description of software that is consistent with the limitations that normal users run into. I don’t know anyone, for example, that would trust it to do basic math in the same way that they would trust other software or even a normal person to do it. Do you?
> even a normal person to do it. Do you?
There are intelligent people capable of reasoning with dyscalculia. Sorry, but being unable to do arithmetic is not enough of an argument.
I have no idea what Quicken is, an honestly, ray-tracing (using the definition I'm familiar with) is a mechanical process that doesn't require any intelligence.
EDIT: Here's an article that has the technical details (and it's not a 15-minute read): https://www.theguardian.com/technology/article/2024/aug/06/a... The original article is 80% PR fluff aimed at emotional influence, the one from Guardian is just good journalism. I have an allergy to the former.
https://arstechnica.com/information-technology/2023/02/man-b...
"Adversarial training, RLHF, and input-space contrastive methods have limited performance. Why? Because input spaces are BIG. There are just too many ways to be wrong" [1]
A way to solve the problem is projecting onto latent space and then try and discriminate/predict the best action down there. There's much less feature correlation down in latent space than in your observation space. [2]
[1]:https://x.com/ylecun/status/1803696298068971992 [2]: https://openreview.net/pdf?id=BZ5a1r-kVsf
Sure, Go is not solved yet. But RL is just fine continuing to that asymptote for as long as we want.
The funny part is that this applies to people too. Masters don't like to play low ranked people because they're unpredictable and the ELO loss for them is not worth the risk. (Which does rise questions about how we really rank people)
As that timeline can approach infinity, just adding extra training may not actually be a sufficient compromise.
There might be logic that says an 'old' link (> 12 hours say) with no comments doesn't need to be cross linked to if submitted later (or other rule).
In any case, @mods and @dang do not work (save by chance) .. if you think it's worth bring to attention then there's generally no downside to simply emailing direct to hn # ycombinator dot com from your login email.
Robot because physical world provides a lot of RL for free (+ and -).
Humanoid because known quantity.
Cute because people put up with cute a lot more, and lower expectations.
Sometimes judging a "good enough" policy is order of magnitudes more easier than formulating an exact reward function, but this is pretty much domain and scope dependent. Trying to estimate a reward function in those situations, can often be counter productive because the reward might even screw up your search direction. This observation was also made by the authors (researchers) of the book "Myth of the objective"[2] with their picbreeder example. (the authors so happens to also work for OpenAI.)
When you have a well defined reward function with no local suboptima and no cost in rolling out faulty policies RL work remarkably well. (Alex Ipran described this well in his widely cited blog [3])
Problem is that this is pretty hard requirements to have for most problems that interact with the real world (and not internet, the artificial world). It's either the suboptima that is in the way (LLM and text), or rollout cost (running GO game a billion times to just beat humans, is currently not a feasible requirement for a lot of real world applications)
Tangentially, this is also why I suspect LLM for planning (and understanding the world) in the real world have been lacking. Robot Transformer and SayCan approaches are cool but if you look past the fancy demos it is indeed a lackluster performance.
It will be interesting to see how these observations and Karpathys observations will be tested with the current humanoid robot hype, which imo is partially fueled by a misunderstanding of LLMs capacity including what Karpathy mentioned. (shameless plug: [4])
[1] https://openai.com/index/learning-from-human-preferences/
[2] https://www.lesswrong.com/posts/pi4owuC7Rdab7uWWR/book-revie...