Hacker News Reader

Comment by YeGoblynQueenne | Hacker News Reader

>> This is not "brute-force" though. It's an iterative search algorithm. You learn things at each iteration. You also don't search blindly. You use "something" (heuristics, experience, intuition) to come up with "ideas" at each iteration. You don't try 650 random programs. You try 650 different ideas each learning from the results of previous trials.

But, learn what? All those ideas where wrong. How does an LLM "learn" from incorrect proofs that it has generated itself? What does it learn? Can you explain how this mechanism works?

View on HN

>> In this case, by happy accident, what stuck was an improved bound of the zeroes of the zeta function etc.

I’m not familiar with this specific example (or Mathematics) but I assume they keep intermediate results (python functions, lemmas, computations, intermediate proofs etc.) and formulate and explore adjacent ideas. Even with a failed attempt you can learn things. LLMs make a difference here because they can evaluate an experiment and hypothesize what went wrong or what should stick. So the search space is dynamically evolving unlike pre-LLM algorithms.

Karpathy’s Autoresearch provides a proof of concept for this method.

Thanks, but I can't see that in the article by Anthropic. AFAICT the "650 ideas" that were wrong were all generated independently of each other.

I don't agree that LLMs can evaluate an experiment. There's nothing in LLM training that makes them capable of telling what is e.g. a correct hypothesis from an incorrect one. I know that is a common claim particularly encouraged by AI companies but whenever that claim has been studied systematically and carefully the result is that self-verification doesn't work. For example, see:

On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

https://arxiv.org/abs/2402.08115

Note also that basically all the mathematical results published so far have to be checked by an external verifier, either human mathematicians or a proof assistant like Lean, or both, and some systems explicitly couple an LLM generator to a traditional solver, like e.g. AlphaProof. None of this would be needed if LLMs could really evaluate their own results in any reliably correct manner.

> On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

An empirical paper from 2024 that doesn't give a principled reason why LLMs will always be bad at self-verification.

"An empirical paper that doesn't give a principled reason" is a great description for 99% of machine learning papers since the 1980's. I don't see anyone batting an eyelid e.g. that "Attention is All you Need" or "Language Models are Few Shot Learners" have no theoretical results.

Which is to say, I think your criticism is unfair and an attempt to avoid engaging with the arguments in the paper.

Ask Edison about the light bulb

I beg your pardon?

Reply on news.ycombinator.com