Hacker News Reader

Comment by feanaro | Hacker News Reader

> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them".

The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer?

Am I missing something?

[1]: https://en.wikipedia.org/wiki/Almond#Production

View on HN

Human can't even properly agree on what "majority" means in all contexts, in some it's "One option have more than half of the total" but for others it'd be "difference in votes between the first-place candidate in an election and the second-place candidate", as just one silly example.

https://en.wikipedia.org/wiki/Majority has a bunch of variations and contexts listed, where it might differ what "Majority" is actually referencing.

Weird, this page says 80%

https://en.wikipedia.org/wiki/Almond_cultivation_in_Californ...

The difference is share of global production vs exports

Since the agents were instructed to not explain their answer, you can't know if their answer was reasonable or not.

The reason for the "No explanations, no qualifiers" in the prompt was to force the models to put the claim in one of the four buckets and answer with the bucket name only. It's a pure quantitive analysis (first in a series) and it does indeed lack the qualitative aspect.

Sure, but people are drawing conclusions beyond "LLMs said different words" and trying to use it to analyze whether LLMs were wrong about the underlying facts, but that information isn't available to us.

structured output { "answer" : "Misleading", "reason" : "Almonds..." }

Have reason be optional and instruct it to only provide reason for the middle "Mostly True" or "Misleading".

It’s misleading because it’s false. But yes, I think false is quite plainly the better answer there.

The 51% is US, the question was about California.

The statistic is about commercial production, not number akmonds grown.

Looks safe to say that even majority of almonds are not grown in California.

Here (https://en.wikipedia.org/wiki/Almond_cultivation_in_Californ...) I have

> California produces 80% of the world's almonds and 100% of the United States commercial supply

But regardless of which number we use, California represents a large portion of US almond production, so much so that misleading could be an acceptable answer if the LLM interpreted the prompt as an exaggeration. I think the example was apt

"All almonds are grown in the U.S. state of California." implies "No almonds are grown outside the U.S. state of California."

You find one almond tree outside of California that grows almonds, where such almonds are grown intentionally, and the claim is false.

Nobody is saying the claim is true. This is a discussion of whether misleading could be a valid answer. I've been arguing if the model interprets the claim as an exaggeration, then misleading would be an acceptable answer, and due to California's dominance in the industry one could reasonably interpret a claim of this nature as an exaggeration.

It's fine if you disagree, but I have never claimed the question was true.

A statement that is obviously false cannot reasonably be “misleading”.

An exaggeration can. If I said "the C language was a million times faster than python" that would be an exaggeration. It would both be obviously false (most things are only trivially faster) and misleading.

If the LLM interpreted the original statement as an exaggeration, then misleading could be an acceptable answer to a false statement.

Reply on news.ycombinator.com