Hacker News Reader

> 100 instances of the same LLM may converge on the same result as 10.

Not in the highly verifiable domains. There you can take it from say 80-90% maj@x to 99% pass@n. Math, some parts of programming and cybersec are examples of highly verifiable domains. (e.g. if you're searching for a linux LPE, that's expensive to search but easy/cheap to verify - just have a token in /root and have the model retrieve that token)

View on HN

Verifiability makes it easier to understand how well the LLM works, but this doesn't counter my hypothesis. If X number of instances get 99.0% on an objective, verifiable metric, is there any guarantee that 10X will get 99.9%? The fact that we are reliant on new model releases to push capability in big ways, and that people running gigantic clusters of LLMs end up beaten by new models implies that the capabilities of a given model have a hard upper limit, and that it may not even take much to reach it.

Reply on news.ycombinator.com