Comment by ozgung | Hacker News Reader

Comment by ozgung | Hacker News Reader

Parent Full thread

This is not "brute-force" though. It's an iterative search algorithm. You learn things at each iteration. You also don't search blindly. You use "something" (heuristics, experience, intuition) to come up with "ideas" at each iteration. You don't try 650 random programs. You try 650 different ideas each learning from the results of previous trials.

Yes this is the "Universal Problem Solving Algorithm". It's actually the same algorithm used by Evolution.

Also this algorithm is vastly different than "monkeys with typewriters". Monkeys don't learn or evolve their writing. There is no memory, no constraints, no learning-curve. At each iteration they freshly sample from a Uniform Distribution. Expected time for a solution is infinitely long.

"The Universal Algorithm" on the other hand is incredibly fast. Humans (designers, researchers) also use the same algorithm but they are much slower to iterate than computers. Instead of trying 650 different ideas at a single run, we have 100s of researchers each try few different ideas independently.

>> This is not "brute-force" though. It's an iterative search algorithm. You learn things at each iteration. You also don't search blindly. You use "something" (heuristics, experience, intuition) to come up with "ideas" at each iteration. You don't try 650 random programs. You try 650 different ideas each learning from the results of previous trials.

But, learn what? All those ideas where wrong. How does an LLM "learn" from incorrect proofs that it has generated itself? What does it learn? Can you explain how this mechanism works?

Ask Edison about the light bulb

I beg your pardon?

>> In this case, by happy accident, what stuck was an improved bound of the zeroes of the zeta function etc.

I’m not familiar with this specific example (or Mathematics) but I assume they keep intermediate results (python functions, lemmas, computations, intermediate proofs etc.) and formulate and explore adjacent ideas. Even with a failed attempt you can learn things. LLMs make a difference here because they can evaluate an experiment and hypothesize what went wrong or what should stick. So the search space is dynamically evolving unlike pre-LLM algorithms.

Karpathy’s Autoresearch provides a proof of concept for this method.

Thanks, but I can't see that in the article by Anthropic. AFAICT the "650 ideas" that were wrong were all generated independently of each other.

I don't agree that LLMs can evaluate an experiment. There's nothing in LLM training that makes them capable of telling what is e.g. a correct hypothesis from an incorrect one. I know that is a common claim particularly encouraged by AI companies but whenever that claim has been studied systematically and carefully the result is that self-verification doesn't work. For example, see:

On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

https://arxiv.org/abs/2402.08115

Note also that basically all the mathematical results published so far have to be checked by an external verifier, either human mathematicians or a proof assistant like Lean, or both, and some systems explicitly couple an LLM generator to a traditional solver, like e.g. AlphaProof. None of this would be needed if LLMs could really evaluate their own results in any reliably correct manner.

> On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

An empirical paper from 2024 that doesn't give a principled reason why LLMs will always be bad at self-verification.

"An empirical paper that doesn't give a principled reason" is a great description for 99% of machine learning papers since the 1980's. I don't see anyone batting an eyelid e.g. that "Attention is All you Need" or "Language Models are Few Shot Learners" have no theoretical results.

Which is to say, I think your criticism is unfair and an attempt to avoid engaging with the arguments in the paper.

Thank you. I've never seen 'brute force' and 'type writing monkeys' abused so much than these LLM discussions.

Reply on news.ycombinator.com