Hacker News Reader

Comment by croes | Hacker News Reader

Isn't your input your confidence that GPT-4 gives the correct answer, like I'm 100% sure GPT-4 gives the right answer for the capital of France but I'm only 50% sure it gets the ASCII art correct, and 0% for another question which means 100% GPT is wrong?

Because how could he know the probability of getting the correct answer? He just tried and then it's a yes or no.

And if it's right/wrong shouldn't GPT right/wrong 100% of the time for the same question and the same version?

View on HN

Regarding your last question, ChatGPT isn’t deterministic. It can certainly give a correct answer one time and an incorrect answer another time, for the same prompt.

For the same prompt and same version?

That would be deal breaker if the same prompt gives different results.

Would also make the whole prompt engineering thing pretty useless.

Yes, same prompt and same version. GPT-4 is especially stochastic, even with temperature=0.

So all these comments claiming that an article about the GPT's responses is wrong are useless.

There are also such comments on this article.

https://news.ycombinator.com/item?id=37361521

I wonder why that is. Maybe they’re A/B testing assistant prompts, or there’s noise built into the architecture somewhere.

There was a post about it a while back. From what I recall, it’s an efficiency/cost reduction side effect, but it could be made deterministic, if they wanted to eat the cost.

I firmly disagree with your latter two statements, but yes it is non deterministic

> Regarding your last question, ChatGPT isn’t deterministic. It can certainly give a correct answer one time and an incorrect answer another time, for the same prompt.

It's supposed to be deterministic if you set temperature to 0.0, but seems that doesn't work well in GPT4 compared to earlier versions...

It is not deterministic since text-davinci-001.

https://152334h.github.io/blog/non-determinism-in-gpt-4/

Good to know, I thought it was deterministic, especially because of all the comments on other articles where people wrote that the author was crazy for getting a different answer to the same question.

> Isn't your input your confidence that GPT-4 gives the correct answer

You may be right that that’s the intent, however what’s the point (other than collecting data about user confidences)? If I enter 0.3 and GPT provides a correct answer, then that doesn’t mean that the 0.3 was somehow wrong.

In that case 0.3 would be more wrong than 0.4 and less wrong than 0.2. The closer your predictions are to reality over a bunch of questions, the better you understand reality.

You can’t really say that for a single data point. The 0.3 may be completely correct. Now, if you try ten times, things might be different.

Isn't the whole point to show you how right your confidence is about GPT's capabilities?

At least the results are about the quiz taker and his confidence.

It’s a noisy measurement of how right you were.

You're right. But if you enter 0.3 on average over 28 questions and the actual number of true answers differ by a lot from 8 then you have learned your general sense of GPT-4s abilities is uncalibrated.

Reply on news.ycombinator.com