AI researchers debate how close we are to recursive self-improvement
dwarkesh.com
dwarkesh.com
What if the bottleneck is physical chips and data centers and power and only so much can be squeezed out of algorithms?
What if AI seems more impressive than it is, because the things it is good at happens to be hard for us, and it's hard for us because it's new to us (processing lots of abstract information), but the things that we are good at (manipulating physical world) are objectively orders of magnitude harder?
I doubt the human brain is at global maximum.
True optimal what? Optimal intelligence? What would that be?
Elon Musk, obviously
When a severe security bug crops up, one postmortem tells where we messed up. Someone somewhere didn't do his job. Maybe they prioritised speed, money or due to management issue. But we know why.
When an accident happens (Motorvehicles or otherwise), most of the time, if anyone in the chain of events did their job, it could've been avoided. But we ignored the policies, standards and laws set in place to avoid just that. There can't be a better example than Boeing for this.
When it comes to natural disasters, we know how to fix most of it. But we as human race just don't want to.
Hell, we knew how to freakin avoid Covid which came out of the blue. 6ft apart, wash hands and mask. We couldn't agree to that to keep us ALIVE!
IMO, we don't want new technologies, new innovations, new laws in most cases. We just needs to apply what we know already - PROPERLY! But that ain't fancy.
All I could remember is that tweet where musk has a $2M reward for the invention of a carbon capture device or something around those lines. And someone replied to that tweet saying, it's trees. Trees do this.
I've often heard it said that <big corporations> often use consulting firms to recommend unpopular decisions they were already planning to make, thereby offloading the culpability to the consulting firm.
Maybe the LLMs are going to be the ultimate "consulting firm" for our societal issues.
I say this mostly tongue-in-cheek, but we're already offloading a lot of lower-level responsibilities (e.g., "write my email") onto these AIs.
Large organizations also offload decision making when they don't have a clue, and simply want to point to some action taken. The answer doesn't even matter, they just don't want to be responsible for something, or waste their own time making something (that other stakeholders care about) a real priority.
People downplay the value of consultants, but they have many uses!
Also, RSI is obviously guided. If only guided by "it's not giving results so we'll try something else". To require that RSI happens in a black box for it to count would be arbitrary, and also not how anyone is going to do it.
There is no publicly known reference example of RSI, we have no idea how it works or what it does to the trajectory of progress?
A more concerning bottleneck is diminishing returns: even with recursive improvement, capability gains could end up being only logarithmic (or worse). Even current systems seem capable of these diminishing returns only/mostly.
That kind of assumes there is an optimal. It seems quite likely not so - that there will always be room for improvement in AI like we don't have a true optimal chess or go programs. They just get better and go up a bit on the ELO scale over time.
like https://huggingface.co/spaces/lmarena-ai/chatbot-arena has a score for chatbots - 1506 for Claude Fable 5 at the moment for example. I guess the test for RSI is if it can make the number go up. It'll probably go through phases of self improvement being a bit rubbish, self improvement combined with human help being best which is probably where we are now, and humans not needed perhaps in the future.
Which means someone is already working on it.
This same process has recently been under scrutiny when mathematicians claimed AI companies trained on their unpublished logs and later claimed merit for results. But the exchange of experience happens across all domains. Experience gets generated at amazing rates, and absorbed by models which get applied everywhere, collecting more experience.
I don’t feel like monitoring Twitter to see how it’s going though.
> this logic gate should not hack other systems
well .. it only applies a rule to inputs
can we impose this of authors "this book should only be causing good things when it is read and knowledge used"
1. Deriving happiness from someone else's suffering should be considered a mental disease to be treated.
2. Connecting people to a virtual reality where each person lives in absolute control of their own reality, matrix style. I.e, a sadist can live in a reality where they make other virtual people suffer, without actually making real people suffer.
No. AI should be optimized for the one true good: maximum market returns. Personally I think that will only be achieved once humans are removed from the market system and decommissioned.
Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail.
Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.
If the bar is 'the reins holder must be able to operate in a completely sustainable way,' why are we letting humans do the job? It makes it sound like that actually isn't the threshold of competency we use in practice.
It doesn't make any sense, of course, because it's a movie.
But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.
Ironically, if you ask Google, the Gemini Annoyance AI [1] confidently asserts that what you're saying is true, but if you follow the links they give, none of them support the claim, and the further you click, the more it becomes people repeating each other's speculation and hearsay and calling it evidence. For example:
https://scifi.stackexchange.com/questions/19817/was-executiv...
Typical internet story.
[1] Aside: I am far more worried about the influence of AI gaslighting on mushy-brained humans than I am on AI destroying the human race. The dystopic future of AI is people.
https://www.reddit.com/r/matrix/comments/1qatv42/is_it_your_...
There is some stuff online suggesting that the original script had that angle, but the movies did not, and AFAICT it's fanfic.
We should definitely use this stuff to guide our thinking about the real world, though, and not, say, Asimov (or a million other science fiction authors), who had an optimistic version of the same thing. Those are wrong.
Whereas using our brains for compute (in the background, or during REM sleep) makes perfect sense; power efficiency is excellent.
People obviously know this stuff is absurd on some level, but it doesn't stop them from cherry-picking the parts they "like" (i.e. fear) and ignoring the rest.
What's your N(doom)?!
Everyone panic and give me money!
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.
Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
so you would unplug a child who hasn't shown their potential yet, simply because they don't produce enough value at the moment to justify their food?
if self_improved():
self_improve_more()
else:
see_tony_robins_and_buy_tapes()But can an AI construct Terminal Bench 5.0 or GDPval 2027 ?
Without this, there is just no path towards RSI.
Doesn't everyone get their agents to construct evals it can't pass? There's nothing magical about this.
For example if frontier models are able to one-shot a database query across 20 columns and 10 tables add one additional relationship then test. Keep doing this until the pass-rate drops below acceptable and now you have your new frontier eval.
My thought experiment was along the lines of "Let's say I'm Anthropic and I want to significantly improve my frontier model's performance on, say, theoretical physics research. How do I build a fully autonomous process capable of constructing an eval that's somewhat outside the current capability in some useful direction (decided by the autonomous process itself)?"
Would love to hear folks' ideas. :)
How many times has AGI been declared already? A bunch of people (e.g. Jensen Huang) have called Astra AGI, for example.
The goal posts get moved, everybody's hustling, lying, inventing new buzzwords, doing mental gymnastics, and it's all just tiring.
Just show me the results, and let the proof be in the pudding.
That's not a gotcha question.
I'm just genuinely curious if that would meet the definition of RSI.
Speak for yourself.
Or even evolution itself? Isn’t that just RSI writ large?
This is all rather rudimentary stuff discussed in detail in the so-called "Sequences", the long series of Less Wrong posts by Yudkowsky back around 2010.
Marginally. Model collapse is still a problem. Continuous learning is still a problem.
For AI to make a big leap we need a big break through.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
Take the most recent qwen and deepseek models with offloadable n-grams, which function (both in name and vaguely in capability) like human memory "engrams".