Hacker News Reader

I felt like the scaling laws were magical thinking, but apparently they work. However I still do not understand why we should expect exponential improvements due to this automated process. My intuition is that the first iteration of it should result in a noticeable capability increase (though I think these labs were already using a lot of AI to orchestrate training the current model anyway), and then the second iteration of it should be nearly identical in capability to the first, unless more data is involved, more compute is involved, or the model is bigger.

AI can compress AI nearly losslessly.

You can get exponential growth from completely ordinary feedback loops. You start with some amount of stuff, you do a series of steps and you end up with more of the same stuff you started with. As you keep going through the loop, the stuff you have grows exponentially. That's for example how exponential economic growth works.

Of course data, compute and model size are not held constant. You start with some money and use it to acquire researchers, data and compute, and have the researchers produce a big model and you use that model to get more money, and you use the additional money for more researchers, more data, and more compute to produce a bigger model. This is what has propelled exponential AI progress so far.

Recursive self-improvement is invoked to predict superexponential growth. The idea is that instead of only using the model to make more money, you add it to the researchers to speed up the loop, so not only is the money growing with every iteration, the iteration time also gets shorter, producing growth that is faster than exponential.

The problem with this simplistic prediction is that it assumes additive and multiplicative relationships of the form money = (researchers + AI)×compute_spend, but if doing more research paid off so reliably, you could also just hire more researchers, abstractly money = research_spend×compute_spend and with a balanced allocation of research and compute, you would get a money-squaring machine even without using AI for AI research.

And the reason this doesn't work in reality is that there are diminishing returns everywhere. You can also see this in the OpenAI post, where they write 7 times as much code to run 1.6 times as many experiments, and those additional experiments probably only result in minor improvements to model quality.

Fundamentally the current language-model approach is lacking in any general reasoning ability, so they are trying to mitigate this by using synthetic data and reinforcement learning to bake specific reasoning chains into the model, one domain at a time ... coding, math, hacking, three.js competence ...

The trouble with this is that there is little generalization in the utility of these baked-in reasoning chains from one domain to the next, so in the end this is not dissimilar to the CYC project's decades long attempt to encode all of human knowledge into a giant expert system... the hope is that if you make your collection of jagged narrow intelligences sufficiently large then it will look more like general intelligence, not a bed of nails.

I would assume that the gains from this type of test-time compute (and synthetic RLVR dataset) scaling will level out just the same as gains from human training set scaling eventually levelled out, and basically for the same reason - because you are tapping into a finite data pool, whether language itself, or reasoning steps isolated from that language, so at some point the incremental gains become increasingly small (10->20% is a doubling, 90->95% is just a ~5% gain).

It's not clear where all the different AI companies are currently focusing - on some of these narrow verticals, or on growing the forest of narrow intelligences. OpenAI's chief scientist, Jakub Pachocki, said that their current focus is on RSI(!) - improving the model in ways that will help them iterate faster in order to have a "fire meets fire" tool than can combat enemy AIs. It's not clear what this really means - what skill set makes an LLM more helpful in the process of building LLMs, but it seems to basically be process automation.

“Recursive” is a reasonable term because the generation N AIs will train the Generation N+1 AIs. The term “iterative” doesn’t reflect this nuance as well IMO.

Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool.

I think you’re conflating the direction of definition with the direction of evaluation.

Compare the similarity of:

  AI(n) = improve(AI(n-1))

With:

  Fib(n) = Fib(n-1) + Fib(n-2)

The latter is a classic example of recursion. So why isn’t the former?

Edit: formatting

I think you're confusing a recurrence relationship with recursive execution. Until AI has control over its inputs (including physical inputs - compute hardware, etc.) it's not quite RSI. We'll get there. They'll get there, rather.

It's not a nuance, it's a sequence.

Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function)

>AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light

Sounds like repetitive stress to me.

>loop forever using output as input but at some point the result will stop changing

Running in place will eventually wear you out too. Plus with some things it can be difficult to know for sure if that's where you are at the time.

Even worse may be if you were almost running in place, it could be orders of magnitude more difficult to discern, especially if the scale was massive to an unprecedented degree.

that's not really how eigenvalues work... they specifically also model the case where the result keeps changing exponentially.

The claim is that the RSI operation is just finding a fixed point of improvement,

RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI

As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val

Eigenvectors represent fixed directions, not fixed magnitudes. From Wikipedia:

> More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v .

In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth.

> of (1/val)A

The name you are looking for is "fixed points", not "eingevalues".

The "You can loop forever using output as input but at some point the result will stop changing" idea seems more related to dynamics, feedback and attractor states.

If you have a feedback loop of "recursively improving" by generating synthetic data, to train on, to generate more synthetic data, etc, then this is indeed a matter of looping "using output as input", and without any other system inputs it would indeed "converge" to some attractor states.

I think the limit of what can be achieved with RL and synthetic data generation is better simply described as a leveling off of gains as you extract all the intelligence and knowledge from the original human training data.

Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it.

To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.

What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation?

How do you think why there's this fad of producing general purpose humanoid robots?

> What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation?

Money, regulations, EUV machine lead-times, global helium supply, reality ...

It's funny that we've got the Dwarkesh contingent saying that GPUs will become infinitely expensive, and now another contingent saying that they will become infinitely abundant.

Even if compute were free, and/or the AI was so smart that it picked the right experiments to run every time ("make no mistakes"), you still have to actually train the model, which takes months, and if model Ver. N+1 depends on model Ver. N, then it's iterative regardless of how much compute you have.

Who's saying that compute will become infinitely abundant? "Singularity" is just a way of saying that known models begin to give absurd predictions. Anyway, intelligence is a way of overcoming obstacles. 10 million tonnes of helium is a nice head start and retraining models from scratch is not guaranteed to last forever.

> 10 million tonnes of helium is a nice head start

Yeah, but then you need to refine it to 99.9999% purity, to be able to use it.

AFAIK the notion of a/the technological "singularity" is a point in time where technology is building upon itself (RSI!) so fast, at an ever increasing pace, that the speed of change effectively becomes infinite and incomprehensible to humans.

The word "singularity" is presumably coming from math or space, like a black hole singularity where matter becomes infinitely dense and the known laws of physics break down.

> How do you think why there's this fad of producing general purpose humanoid robots?

For doing physical work?

So a swarm of robots builds the shell of your fab overnight, and then what? Where is the EUV machine coming from?

So far the most we're seen TeslaBot do is serve drinks via tele-operation, and I don't think it's exactly built for construction site work.

For example, TSMC uses behavioral cloning to scale up human-bottlenecked parts of the manufacturing process to meet the growing demand, while automated research laboratories do thousands experiments in parallel to find better manufacturing processes.

The production bottleneck in a fab isn't the human workers - the process is mostly automated. The bottleneck more derives from how many wafers per hour you can process, which comes down to the etching process and EUV throughput.

ASMLs EUV machines are literally the most complex machine that mankind has ever built, which is why no other country, including the US, has yet been able to duplicate it. It's not just the machine itself, but a global supply chain of irreplaceable components such as focusing mirrors made by Zeiss to an incomprehensible level of accuracy - differences in surface height no more than the size of a hydrogen atom (or if you scaled the mirror up to the size of the country of Germany, then surface differences in height of 0.1mm).

Robots are useful to automate things, but they are zero help when trying to build tech like this that you are incapable of building in the first place.

The US has fallen way behind in manufacturing expertise, and no swarm of robots is going to help.

You've missed a part where it's TSMC that does behavioral cloning (to build more EUV machines). The full vertical integration is a bit farther down the line.

Etching a model's weights on silicon is another way to utilize non-top-notch tech-processes, while maintaining or improving performance. (and it suits robotics well)

TSMC doesn't know how to build EUV machines - they are stuck buying them from ASML like everyone else.

Putting a model's weights in read-only memory close to the processor is certainly a way to increase token/sec generation speed, but of course does nothing to increase intelligence. Robots aren't going to help though - semiconductor manufacturing is semiconductor manufacturing regardless of whether you are etching GPUs or memory onto your wafers.

Ah, sorry, it's ASML expertise that needs to be cloned and scaled up. I don't see how it changes things, though.

Robots don't need that much intelligence. High-speed joint control, "hand-eye coordination", the higher level tasks can be delegated to external models. Distillation already works quite well for isolating the required functionality.

I'm pretty sure ASML, and their supply chain, do have plans to increase production, as do the chip fabs - they all see the demand, while at the same time being leary of boom and bust which is the historical reality of the chip business.

But, the production expansion rate of none of these companies is being limited by lack of trained personnel, and if it were it would surely be faster to hire/train more humans since robots are still very far from human dexterity, not to mention intelligence.

Robots and AI are tools of automation, a way to replace humans with machines, but not all the problems in the world are bottle-necked by lack of humans, or the cost of humans.

Investing in training a person gets you one trained person. Investing in training an ML system gets you a cloneable ML system that can be scaled on demand much faster. ROI might change quickly.

Sure, but we're simply not at the point, maybe never will be, where lack of employees is the bottleneck to chip production. A fab takes billions of dollars and multiple years to construct - there are many constraints.