Hacker News Reader

How can you know which LLM is better for every specific question?

That is, how do you quantify "better"?

Imagine I asked: Write me a sad poem about "Yorick" in Hamlet?

You have to predict which LLM is going to be produce the "best" poem, but by what criterion do you evaluate that?

Currently, we simply use GPT4-as-a-judge, with a general system prompt we've written which is task agnostic. This is then used to train the neural scoring function, which predicts quality ahead-of-time. However, it's on our roadmap to add make the judging more flexible, potentially task-specific judge prompts and in-context examples, also perhaps using a jury [https://arxiv.org/pdf/2404.18796].

Reply on news.ycombinator.com