Hacker News Reader

> You can sample GPT-4 not once, but 10, 100, 1000 times.

Is there a study on improved outcomes based on simple repetitions of GPT-4? I would be very interested in that study. I don't think gpt hallucinations are like human hallucinations. Where if you ask someone after a temporary hallucination they might get it right another 9 times, but I could be wrong. That would be an interesting result.

View on HN

This is called self-consistency: sample multiple times and then select the answer that appears the most. You can read Alphacode's paper, which uses a similar method. I think we can do even better, though. Over 1000 runs, it's likely that the correct answer appears at least once. Instead of selecting based on the majority, we could use the LLM to judge the answer individually. Since it's easier to verify than to generate, I think this method would work very well. I don't know if any lab has tried this, though.

Reply on news.ycombinator.com