From what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.
What if the bottleneck is physical chips and data centers and power and only so much can be squeezed out of algorithms?
What if AI seems more impressive than it is, because the things it is good at happens to be hard for us, and it's hard for us because it's new to us (processing lots of abstract information), but the things that we are good at (manipulating physical world) are objectively orders of magnitude harder?
A more concerning bottleneck is diminishing returns: even with recursive improvement, capability gains could end up being only logarithmic (or worse). Even current systems seem capable of these diminishing returns only/mostly.
I doubt the human brain is at global maximum.
That kind of assumes there is an optimal. It seems quite likely not so - that there will always be room for improvement in AI like we don't have a true optimal chess or go programs. They just get better and go up a bit on the ELO scale over time.
like https://huggingface.co/spaces/lmarena-ai/chatbot-arena has a score for chatbots - 1506 for Claude Fable 5 at the moment for example. I guess the test for RSI is if it can make the number go up. It'll probably go through phases of self improvement being a bit rubbish, self improvement combined with human help being best which is probably where we are now, and humans not needed perhaps in the future.
True optimal what? Optimal intelligence? What would that be?
When a severe security bug crops up, one postmortem tells where we messed up. Someone somewhere didn't do his job. Maybe they prioritised speed, money or due to management issue. But we know why.
When an accident happens (Motorvehicles or otherwise), most of the time, if anyone in the chain of events did their job, it could've been avoided. But we ignored the policies, standards and laws set in place to avoid just that. There can't be a better example than Boeing for this.
When it comes to natural disasters, we know how to fix most of it. But we as human race just don't want to.
Hell, we knew how to freakin avoid Covid which came out of the blue. 6ft apart, wash hands and mask. We couldn't agree to that to keep us ALIVE!
IMO, we don't want new technologies, new innovations, new laws in most cases. We just needs to apply what we know already - PROPERLY! But that ain't fancy.
All I could remember is that tweet where musk has a $2M reward for the invention of a carbon capture device or something around those lines. And someone replied to that tweet saying, it's trees. Trees do this.
I've often heard it said that <big corporations> often use consulting firms to recommend unpopular decisions they were already planning to make, thereby offloading the culpability to the consulting firm.
Maybe the LLMs are going to be the ultimate "consulting firm" for our societal issues.
I say this mostly tongue-in-cheek, but we're already offloading a lot of lower-level responsibilities (e.g., "write my email") onto these AIs.
Large organizations also offload decision making when they don't have a clue, and simply want to point to some action taken. The answer doesn't even matter, they just don't want to be responsible for something, or waste their own time making something (that other stakeholders care about) a real priority.
People downplay the value of consultants, but they have many uses!
Elon Musk, obviously
Also, RSI is obviously guided. If only guided by "it's not giving results so we'll try something else". To require that RSI happens in a black box for it to count would be arbitrary, and also not how anyone is going to do it.
There is no publicly known reference example of RSI, we have no idea how it works or what it does to the trajectory of progress?