Hacker News Reader

Comment by dspillett | Hacker News Reader

> If I recall correctly, there were some papers which suggested that LLMs favor LLM-generated passages over human written ones

That makes sense. What an LLM does is output what the model thinks is the best set of tokens in response to a given input, so when you ask it to judge the best response to that input it is going to conclude that the best one is the one that must closely matches what it would output, which is what it did output.

Of course you aren't giving exactly the same context+input, but close enough that any difference doesn't push the output it made far from what it is going to say is ideal.

View on HN

Does an LLM have any idea of what "best" is?

I think it doesn't, and just predicts the range of most statistically likely next tokens based on its training data, and picks one of those.

In ML, "best" is what minimises the loss function, so in a certain (very basic) sense it does.

You are mixing up levels.

When you are asking it a question (like which of these two texts is the best), the output is also just picked by minimising that loss function. There's no guarantee that answering "Text B is better" aligns with text B minimising the loss function.

(And they aren't really minimising loss functions during inference. They sample from a distribution. During training they minimise the loss function of the distribution.)

Point taken, I was possibly overly curt in my response, leading to it being ambiguous, both re when the loss function is being minimised and that it is a stochastic process.

So to OPs question: I guess LLMs do have an "idea" of what is best (conditioned on minimising a loss function during training), however they may not always output that (because stochasticity), which maybe represents a degree of uncertainty in that "idea"?

My point is that when you asked them to 'judge' two texts, they are just going to output the tokens that have the highest probability (or a sample from that distribution).

But depending on their training data, these tokens might say 'correct horse battery staple', and not necessarily 'text A is better' or 'text B is better'.

Even if text A would have been more likely to be produced by the LLM.

> Does an LLM have any idea of what "best" is?

"Best" here isn't being used to imply a conscious decision, but at each stage which token has the best score coming out of the model, so the overall best is the sequence of those best tokens. When judging another output it is essentially running the numbers the same way.

It is a bit more complicated than that as the output tokens become part of the context for the next choice, but I think that simplified way of thinking about it holds water.

> That makes sense.

No, it doesn't if the "which one do you prefer" was just a prompt continuation task. LLMs can't see their own evaluation of a given text. If you ask them to continue

  Which one you prefer
  
  > option 1: human text
  
  > option 2: ai text

And they continue with

  option 2: ai text is the better one because [reasons]

then it is not because they evaluated these 2 texts on themselves, observed the evaluation numbers and reported which one is better.

Also, if you instruct humans to come up with the best text they can, and you show them an even better text, they will prefer the better one written by someone else.

Reply on news.ycombinator.com