Hacker News Reader

It will certainly decrease. Also, there are multiple ways to deal with hallucinations. You can sample GPT-4 not once, but 10, 100, 1000 times. The chances of it hallucinating the same things asymptotically reaches 0. It all depends on how much money you are willing to invest in getting the right opinion, which in the field of medicine, can be quite a lot.

View on HN

> You can sample GPT-4 not once, but 10, 100, 1000 times.

Is there a study on improved outcomes based on simple repetitions of GPT-4? I would be very interested in that study. I don't think gpt hallucinations are like human hallucinations. Where if you ask someone after a temporary hallucination they might get it right another 9 times, but I could be wrong. That would be an interesting result.

This is called self-consistency: sample multiple times and then select the answer that appears the most. You can read Alphacode's paper, which uses a similar method. I think we can do even better, though. Over 1000 runs, it's likely that the correct answer appears at least once. Instead of selecting based on the majority, we could use the LLM to judge the answer individually. Since it's easier to verify than to generate, I think this method would work very well. I don't know if any lab has tried this, though.

Reply on news.ycombinator.com