Comment by cs702 | Hacker News Reader

Comment by cs702 | Hacker News Reader

I don't think Sutton's essay is misunderstood, but I agree with the OP's conclusion:

We're reaching scaling limits with transformers. The number of parameters in our largest transformers, N, is now in the order of trillions, which is the most we can apply given the total number of tokens of training data available worldwide, D, also in the order of trillions, resulting in a compute budget C = 6N × D, which is in the order of D². OpenAI and Google were the first to show these transformer "scaling laws." We cannot add more compute to a given compute budget C without increasing data D to maintain the relationship. As the OP puts it, if we want to increase the number of GPUs by 2x, we must also increase the number of parameters and training tokens by 1.41x, but... we've already run out of training tokens.

We must either (1) discover new architectures with different scaling laws, and/or (2) compute new synthetic data that can contribute to learning (akin to dreams).

This is true for the pre-training step. What if advancements in the reinforcement learning steps performed later may benefit from more compute and more models parameters? If right now the RL steps only help with sampling, that is, they only optimize the output of a given possible reply instead of the other (there are papers pointing at this: that if you generate many replies with just the common sampling methods, and you can verify correctness of the reply, then you discover that what RL helps with is selecting what was already potentially within the model output) this would be futile. But maybe advancements in the RL will do to LLMs what AlphaZero-alike models did with Chess/Go.

It's possible. We're talking about pretraining meaningfully larger models past the point at which they plateau, only to see if they can improve beyond that plateau with RL. Call it option (3). No one knows if it would work, and it would be very expensive, so only the largest players can try it, but why the heck not?

> We cannot add more compute to a given compute budget C without increasing data D to maintain the relationship. > We must either (1) discover new architectures with different scaling laws, and/or (2) compute new synthetic data that can contribute to learning (akin to dreams).

Of course we can, this is a non issue.

See e.g. AlphaZero [0] that's 8 years old at this point, and any modern RL training using synthetic data, e.g. DeepSeek-R1-Zero [1].

[0] https://en.m.wikipedia.org/wiki/AlphaZero

[1] https://arxiv.org/abs/2501.12948

AlphaZero trained itself through chess games that it played with itself. Chess positions have something very close to an objective truth about the evaluation, the rules are clear and bounded. Winning is measurable. How do you achieve this for a language model?

Yes, distillation is a thing but that is more about compression and filtering. Distillation does not produce new data in the same way that chess games produce new positions.

Synthetic data is already widely used to do training in the programming and mathematics domains where automated verification is possible. Here is an example of an open source verified reasoning synthetic dataset https://www.primeintellect.ai/blog/synthetic-1

Are they actually producing new data though? This is the sort of thing I called "compression and filtering" because it seems to be new information content is not being produced, but LLMs are used to distill the information we already have. We need more raw information.

Yes this is new synthetic data which did not exist before. I encourage you to read the link.

I think we're talking past each other, I'll try once more. Suppose you train an LLM on a very small corpus of data, such as all the content of the library of congress. Then you have that LLM author new works. Then you train a new LLM on the original corpus plus this new material. Do you really think you've addressed the core issue in the SP? Can more parameters be meaningfully trained even if you add more GPU?

To me, the answer is clearly no. There is no new information content in the generated data. Its just a remix of what already exists.

When it comes to logical reasoning, the difficulty isn't about having enough new information, but about ensuring the LLMs capture the right information. The problem LLMs have with learning logical reasoning from standard training is that they learn spurious relationships between the context and the next token, undermining its ability to learn fully general logical reasoning. Synthetic data helps because spurious associations are undermined by the randomness inherent in the synthetic data, forcing the model to find the right generic reasoning steps.

I agree! DeepSeek has shown this is incredibly powerful. I think their Qwen 8B model may be as good as GPT4’s flagship. And I can run it on my laptop if it’s not on my lap. But the amount of synthetic data you can generate is bounded by the raw information, so I don’t think it’s an answer to the SP.

Yes if you have some way to verify the quality of the new works and you only include the high quality works in the new LLM's training set.

But you don't have a way to do that at scale, other than feed it to another LLM that is trained on that exact same limited corpus. There is no new information being added into the system in loops like that. New information means new measurements, new proofs, new signal or media streams from cameras, new curation/rating data, new books or papers etc.

You can have a look at the DeepSeek paper, in particular section "2.2 DeepSeek-R1-Zero: Reinforcement Learning on the Base Mode".

But generally the idea is that it's, you need some notion of reward, verifiers etc.

Works really well for maths, algorithms, amd many things actually.

See also this very short essay/introduction: https://www.jasonwei.net/blog/asymmetry-of-verification-and-...

That's why we have IMO gold level models now, and I'm pretty confident we'll have superhuman mathematics, algorithmic etc models before long.

Now domains which are very hard to verify - think e.g. theoretical physics etc - that's another story.

> But generally the idea is that it's, you need some notion of reward, verifiers etc.

i dont think youre getting the point hes making.

Simple, you just need to turn language into a game.

You make models talk to each other, create puzzles for each other's to solve, ask each other to make cases and evaluate how well they were made.

Will some of it look like ramblings of pre-scientific philosophers? (or modern ones because philosophy never progressed after science left it in the dust)

Sure! But human culture was once there too. And we pulled ourselves out of this nonsense by the bootstraps. We didn't need to be exposed to 3 alien internet's with higher truth.

It's really a miracle that AIs got as much as they did from purely human generated mostly garbage we cared to write down.

>And we pulled ourselves out of this nonsense by the bootstraps.

Human progress was promoted by having to interact with a physical world that anchored our ramblings and gave us a reward function for coherence and cooperation. LLMs would need some analogous anchoring for it to progress beyond incoherent babble.

True, but LLMs got anchored to reality because we are using them in real world tasks and this connection will only grow richer, wider and faster.

I feel like you’re glossing over some very thorny details that it’s not obvious we can solve. For example, if you just get two LLMs setting each other puzzles and scoring the others solutions how do you stop this just collapsing into nonsense? I.e. where does the source of actual truth come from for the puzzles?

> I feel like you’re glossing over some very thorny details that it’s not obvious we can solve.

Yeah. I tried to be funny. It's not that easy. However AI people already started doing it and AI gains perhaps of the last year come mostly from this approach.

> For example, if you just get two LLMs setting each other puzzles and scoring the others solutions how do you stop this just collapsing into nonsense?

That's the trillion dollar question. I wonder how people are doing it. Maybe through economy? You ultimately need to sell your ramblings to somebody to sustain yourself. If you can't, you starve.

Maybe that's enough for AI as well? Companies with AIs that descended into nonsense won't have anymore money to train them further. Maybe companies will need to set up their internal ecosystems of competing AI training organizations and split the budget based on how useful they are becoming?

Phrasing this in a terminology of "truth" is probably counterproductive because there's no truth. There's only what sells. If you have customers in manufacturing probably things that sell will coincide with some physical truths, but this is emergent, not the goal or even part of the process or acquiring capabilities.

> Of course we can, ... synthetic data ...

That's option (2) in the parent comment: synthetic data.

What about or (3) models that interact with the real world?

To be clear I also agree with your (1) and (2).

Play in the real world generates a data point every few minutes. Seems a bit slow?

What are you basing that statement on?

What exactly are you considering a "data point"?

Are you assuming one model = one agent instance?

I am pretty sure that there is more information (molecular structure) and functional information (I(Ex )) just in the room I am sitting in than all the unique, useful, digitized information on earth.

Humans experience (play in the real world) is multi modal though vision, sound, touch, pressure, muscle feedback, gravitational, etc. Its extremely rich in data. Its also not a data point its continuous stream of information. Also I would bet that humans synthesize data at the same time. Everytime we run multiple scenarios in our mind before choosing the one we execute without even thinking about it is synthesizing data. Also humans dream which is another form of data synthesizing. Allowing AI to interact with the real world is definitely a way to go.

That's true, but still, a single individual or small group living isolated in the real world will, over a lifetime, learn only a tiny fraction of what we can learn from the written knowledge accumulated over millennia.

Having said that, I tend to agree that having AI interact with the world may be key: for one thing, I'm not sure whether there is any sense in which LLMs understand that most of the information content of language is about an external world.

That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.

Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips.

But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis.

So while it's expensive, we can trade scope-constrained robustness (replication and redundancy at many levels of abstraction), for broader cognitive scale and fragility (data centers can't repair themselves and self-replicate).

Going to be interesting to see it all unfold... my bet is on stacking S-curves all the way.

> Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips.

I believe you but I would love to know where this number came from just so I can read more about it

It's napkin math so take it with a pinch of salt, but I am calculating the information stored in genome, assuming 2 bits per base pair, reducing to estimated 88% coding fraction to get the functional bits, and then dividing by cell volume. Did this for a few different types of cells and then averaged the result to around 1–10 Mbit/μm³

# If there are any bioinformaticians around please come eviscerate or confirm this calc #

Then compared it to TSMC 2nm research macro of (38.1 Mbit/mm^2) normalized to cell scale: 0.00019 Mbit/μm³

Living Cells: 1–10 Mbit/μm³

Current best chips: 0.00019 Mbit/μm³

https://research.tsmc.com/page/memory/4.html

You are comparing the fastest writable memory available (SRAM) vs biological non-volatile memory that is essentially read only. Samsung's 280 layer NAND reaches 28,5 Gbit per mm^2. I don't know how you would convert that to a volume, but if we simply multiply by 1000x for simplicity, it would be much closer to 0.19 Mbit/μm³, but even then you have to remember that NAND flash is still writable at pretty high speeds.

This is true, I was only comparing functional information density (unique functional genome bps), not read/write speeds.

I was also taking the information from a genome and then dividing it by the volume of a cell, but there are many instances of the genome in a cell. I didn't count all instances because they aren't unique.

There's a lot to unpack with this comparison and my approximation was crude, but the more I've dug into this comparison the more apparent how incredibly efficient life is at managing and processing and storing information. Especially if you also consider the amino acids, proteins, etc as information. No matter how you slice it, life seems orders of magnitude more efficient by every metric.

I'd like to think there's a paper somewhere where someone has carefully unpacked this and formally quantified it all.

If you find that illusive paper, please post it here!

> The human brain is constrained in size by the width of the female pelvis.

Well, it _was_ until recently.

haha yeah I suppose so, but only barely...

> Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips,

In what sense is this true? That sounds suspiciously like cubic meter of dirt is more advanced than an iPhone because there are 6-7 orders of magnitude more atoms in the dirt.

Well, sort of, but functional-information is a specifically defined term that you can google. "Advanced" is vague and in this context not very helpful.

functional information is basically the amount of data (bits) necessary to explain all the possible functions matter can perform based on its unique configuration (in contrast to random). I am sure I partially butchered this explanation... but hopefully its close enough to catch my drift.

Life is optimized to process and learn from the real world, and it is insanely efficient at it and functional-information dense. (It might even be at the theoretical limit) Our most advanced technology is still 4-5 orders of magnitude behind it.

The capabilities of your iPhone are extremely narrow when compared to a handful of dirt. To you it may seem the opposite, but you are probably mixing up utility to you with functional capability. Your iPhone is has more functional utility to you, but the same amount dirt has way more general functional utility. (Your iphone isn't capable of self-replication, self-repair, and self-nonself distinction aka autopoiesis)

> The human brain is constrained in size by the width of the female pelvis.

I think this is an old belief that isn't supported by modern research.

Thanks for pointing this out. I wasn't aware, and so I just dug into it.

From what I can tell, science used to point to this as the only/primary limit to human-brain size, but more recently the picture seems a lot less clear, with some indications that pelvis size doesn't place as hard of a constraint as we thought and there are other constraints such as metabolic (how many calories the mother can sustain during pregnancy and lactation).

So overall I'd say you are technically correct, even though this doesn't really materially change the point I was making; which is that the size of the human brain is constrained in ways that the size of data centers are not.

> The human brain is constrained in size by the width of the female pelvis.

https://en.wikipedia.org/wiki/Obstetrical_dilemma

While the width is constrained by bipedal locomotion.

This seems so simple but I’m totally not understanding it..

If C = D^2, and you double compute, then 2C ==> 2D^2. How do you and the original author get 1.41D from 2D^2?

If C ~ D^2, then D ~ sqrt(C).

In other words, the required amount of data scales with the square root of the compute. The square root of 2 ~= 1.414. If you double the compute, you need roughly 1.414 times more data.

Thanks for clarification!

Reply on news.ycombinator.com