Hacker News Reader

Sure, it's happening...but, is IT happening? By that, I mean, we can see that the models are able to iterate at a pace and scale that humans can't match, and that provides gains in model performance and efficiency. But, humans are still needed in the loop, and not just because it's necessary for safety/alignment reasons. I don't think any significant discovery has been made by models on their own, and I don't know that LLMs will ever have the capacity to invent. They can synthesize from known data amazingly well, and since they know everything "known data" is extremely broad. But, the leaps, so far, have all come from humans.

So far, I don't think the models are capable of running away on their own. Of course, it would be playing with fire to not at least consider the risks of such a runaway scenario and build in safeguards against it. But, there is no model that can build a better model on its own, thus far, to the best of my knowledge (which is far more limited than the models, so maybe I should ask them).

Recursive Self-Improvement isn't instant, it starts slow and accelerates.

It starts with what they already claim to be doing - increasingly relying on existing models in non-trivial work related to training, evaluating and optimizing the next, more capable generation of models. As long as the proportion of work keeps shifting towards agents doing more and more of it, and humans less and less, that's RSI at play.

It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus, but frankly I find this notion absurd; LLMs already show better judgement than most people. The alternative is, at some point LLMs will show the ability to futz their way into improvement of the next generation of models even without humans in the loop - even if much less efficient at first, if generation N+1 is more capable than generation N, it'll either take off or burn out.

you are really just talking out of your ass here, no offense

just because more and more agents are doing human work, that in no way means the model somehow becomes magically more intelligent, it just means the work will stall and continue on at the same level forever

hell even if they hypothetically have an internal model that can output the entire training data set in a better format, there's no scientific evidence that the newer format has new information that is sufficient enough to train a better AI

as a matter of fact the scientific evidence is on the contrary

It's not the number of agents you should pay attention to, but the scope and nature of work they can perform effectively without being micromanaged by humans.

He talks out of his ass on a lot of topics on here especially those concerning economics and finance.

Even currently, there are clearly diminishing returns throughout the field. While not a skeptic myself, I think the age of LLM's is now over or close to plateauing. GPT-6.1 Sol seems almost the same as GPT-6 Sol(even if the benchmarks are slightly better). It doesn't help that we are running out of real training data, requiring new Synthetic data to be created for the next generation of Models. AI(or SI), has hit its peak, IMO, and now the focus shifts directly to Autonomous tasks. While Fable 5.1 is borderline okay at autonomous tasks, the amount of Slop it generates fundamentally requires a human to correct its mistakes. I was working on a website for a nonprofit with Fable, but the amount of slop and corrections I had to make was astounding/

> It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus

All intelligence, LLM or not, is bound to plateau around the point where the need to operate within physical reality bottlenecks the speed of feedback. AI is progressing swiftly in the digital realm where feedback is nearly instantaneous, but it's unclear whether that would translate into improvements in the physical world where signals are much noisier and intelligence and judgment are less impactful.

Our tried-and-true approach for when the physical world is causing us trouble, is to remake the physical world to be easier to work with.

My go-to example: wheels suck at mobility in the natural world. There's a reason no animal uses wheels as their means of locomotion. They only really work on flat, hard surfaces that give a good grip.

Our solution? We didn't build mechanical legs for all-terrain mobility. We paved the world instead. Dirt roads, then brick roads, then asphalt and concrete - since ancient history, we were forcing the world to adapt, so the one means of mobility we could easily built was effective.

We repeat this pattern every time when the world doesn't agree with us, and adapting ourselves to it is too much of a hassle.

Now, AI will likely adopt the same approach to these kinds of problems, which may not end up very nicely for us.

(Note: we are already adapting our own digital worlds to make them easier for AI. Much like building roads for our wheels, we now have a resurgence of CLI, with new tools exposing high-level operations optimized for agents - not humans - to use comfortably.)

Sure, but is the limiting factor intelligence, here? Give human intelligence to dolphins and they're still not going to be able to pave the world. In order to remake the physical world, we needed certain physical abilities -- we needed the animal mind and body that took billions of years to evolve and embeds advanced priors about the world, and we needed it packaged properly. Irrespective of our intelligence, our body is a general world navigation device adapted to grab and move stuff. We use this innate tool to remake the physical world, but we didn't make it, and we'd be impotent without it.

With AI it's a bit of a chicken or the egg situation. Before it can ablate the complexity of the world, it needs to be able to deal with it, including adversaries such as ourselves. For this it needs a tool as sophisticated as our bodies, but could AI develop such a tool for itself without the help of this tool? I'm not convinced.

The companies whose insane valuation is based on accomplishing thing X say they’re getting closer to accomplishing thing X?

At least they’re led by trustworthy and honest people or we’d need to take their claims with some dose of skepticism.

Those people have a lot of overlap with the LessWrong crowd. They do not have RSI now and probably never will

They absolutely do, unless you believe they are lying about the fact they're using current generation models extensively to develop the next generation of their models.

The letter S stands for "Self" and word "using" is for sure not a superset of the word "self". Basically LLMs are assisting someone who does improving of said LLM, while RSI is a carpal... ahem, RSI is "self" improvement, meaning no intemediary in a human form. PS: it's also not recursive but iterative improvement, even if it ever happens.

At which point is the human using the model as a tool, and at which point is the model using human as a tool?

I can already see the border shift even for mundane tasks I have Claude working on. Increasingly, I'm just setting a high-level goal, and then checking progress and occasionally answering questions or doing something like configuring a system Claude can't easily reach itself (e.g. recording a bunch of traces through my normal use of a system that Claude deemed too fragile to risk operating on its own). Of course, I get detailed instructions to help me - "go there, do this and that, then press this to capture recording, run through this script here to process, attach result to next message". In those cases, Claude is effectively using me as a tool to call.

"improvement" is the word Id put the most emphasis on.

weve seen some improvement from the LLMs unattended, maybe, but will it actually keep improving vs needing a human to bring it back on track?

the recursive part is that it keeps improving on itself, but we really have no example of that. if it does it 30 times with improvements, then maybe, but even then, to actually be relevant it has to do better than paying scientists to do the work for the same cost, consistently.

RSI still means nothing if it costs 1000x the cost to get the same improvements as a human researcher

The physical constraint is money - which could be said anything. Could vehicle factories be more automated if we threw a gazillion dollars at it? Sure. Would it make economic sense? No. ROIC would be disatrous. No investor wants part of that.

This is the nuance that poster doesn’t understand. Given how much money thrown at it - we’re not even close. Who has the appetite to keep throwing more given they continually need to keep raising fresh money?

> The physical constraint is money - which could be said anything.

Money is fake and not a constraint, that's literally the point of capital investments you guys are overindexing on so badly.

> Could vehicle factories be more automated if we threw a gazillion dollars at it?

They already are automated as much as it makes sense. Some of the automation is silicon-and-steel based, some of it is protein-based. Car manufacturers aren't in the business of pushing robotics and nanotechnology, so they prefer to hire protein automatons instead of developing and building their own, so yeah, they "pay people", but think what exactly they are paying them for.

Then consider that this is very much the work the AI is gradually getting as good as, or better, than us.

> This is the nuance that poster doesn’t understand. Given how much money thrown at it - we’re not even close.

What you seem to be missing is that "investing in AI" isn't investing in a chatbot, it's investing in technology that will (and already partially is) sit upstream of every other industry, of everything humans do. Like electricity or the Internet itself.

(The other thing you seem to not understand, in contrast with some of the investors, is that RSI is not a linear walk, it's an exponential curve. X-risk notwithstanding, by the time it's obvious to everyone, it's too late to make money investing in it.)

> keep throwing more given they continually need to keep raising fresh money

Have you heard about R&D?

I'm starting to think that "investors first" thinking that's so common here, that makes people feel they're smart, is actually quite backwards, especially at this scale. Or maybe it's simply people starting with a conclusion and trying to fit reality to match it, no matter how clear of a nonsense that conclusion is?

> if it does it 30 times with improvements, then maybe, but even then, to actually be relevant it has to do better than paying scientists to do the work for the same cost, consistently.

Not necessary. Scientists are capacity limited and supply limited.

> RSI still means nothing if it costs 1000x the cost to get the same improvements as a human researcher

It means you can replace a human researcher with 1000x their salary burned on electricity. It also mean you can get two of them for 2000x of salary of one,

That's a bargain, actually, even with the anomalous, absurdly-overinflated salaries in top-tier ML.

I you knew it was consistent, then these companies would immediately fire their scientists, and burn 10 000x as much as mean researcher salary this month to be able to 10x their virtual headcount overnight, and then use that to make the 1000x be 500x, then 250x, then 125x, then ... and at that point they'd had all the money in the world, because even more skeptical investors would notice what's going on there.

--

TL;DR: what you all seem to miss is that electricity scales better than people.

If I'm learning a human language I can do it in many different ways. But the key distinction would be, if for example I'm using only books to study by myself, or using only some recorded course on the internet - it would be called self-study. But if for example I will go to a class teaching that language, or to a private teacher, or to summer camp, or join online platform pairing students teachers etc. then neither of those will count as self-study.

Is it recursive self improvement if Claude Code writes your pytorch for you?

If the point of that pytorch is to improve the next generation of Claude, then yes, absolutely.