Hacker News Reader

Comment by 5o1ecist | Hacker News Reader

[flagged]

I'd suggest that a measure like 'density[1]/parameter' as you put it will asymptotically rise to a hard theoretical limit (that probably isn't much higher than what we have already). So quite unlike Moore's Law.

Obviously, there’s a limit to how much you can squeeze into a single parameter. I guess the low-hanging fruit will be picked up soon, and scaling will continue with algorithmic improvements in training, like [1], to keep the training compute feasible.

I take "you can't have human-level intelligence without roughly the same number of parameters (hundreds of trillions)" as a null hypothesis: true until proven otherwise.

[1] https://arxiv.org/html/2602.15322v1

Why don't we need them? If I need to run a hundred small models to get a given level of quality, what's the difference to me between that and running one large model?

You can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any decent throughput.

Ah interesting, I didn't realize MoE doesn't need to all run in the same place.

Doesn’t the widely accepted Bitter Lesson say the exact opposite about specialized models vs generalized?

Bitter Lesson is about exploration and learning from experience. So RL (Sutton's own field) and meta learning. Specialized models are fine from Bitter Lesson standpoint if the specialization mixture is meta learned / searched / dynamically learned&routed.

The corollary to the bitter lesson is that in any market meaningful time scale a human crafted solution will outperform one which relies on compute and data. It's only on time scales over 5 years that your bespoke solution will be over taken. By which point you can hand craft a new system which uses the brute force model as part of it.

Repeat ad-nauseam.

I wish the people who quote the blog post actually read it.

Reply on news.ycombinator.com