Comment by edouard-harris | Hacker News Reader

Comment by edouard-harris | Hacker News Reader

Parent Full thread

I just tested GPT-4 on the Winograd schemas below, which I made up and (as far as I know) are novel.

It scored 3/4, incorrectly answering "the pouch" instead of "the baby kangaroo" for Question 2 of Schema 1. And now that I look at the question, one could easily imagine the kangaroo being afraid to jump into a pouch that's too large! In hindsight, if a human complained to me that this question was ambiguous, I'd have a hard time disputing them.

Granted these are just two examples, but they aren't cherry-picked. Moral of the story: before one makes a strong claim such as "there isn't any evidence a model can do X", it's often worth making a minimal effort to investigate one's own claim.

========================================

Winograd Schema 1:

Sentence 1: The baby kangaroo was afraid to jump into the pouch because it was too small. Question 1: What was too small? (Correct answer: the pouch)

Sentence 2: The baby kangaroo was afraid to jump into the pouch because it was too large. Question 2: What was too large? (Correct answer: the baby kangaroo)

Winograd Schema 2:

Sentence 1: The gardener couldn't grow the flowers in the greenhouse because it was too hot. Question 1: What was too hot? (Correct answer: the greenhouse)

Sentence 2: The gardener couldn't grow the flowers in the greenhouse because they were too delicate. Question 2: What were too delicate? (Correct answer: the flowers)

I have tested them on novel Winograd schemas, they didn't pass; that's why I said what I said. Your second schema was incorrectly constructed, in both the first and second sentence the pluralisation of the pronoun in question gives away the answer. "It" won't apply to "flowers" because they're plural, and "they were" won't apply to "the greenhouse" because it's singular. As a result, those sentences aren't Winograd schemas because they're not testing common sense pronoun attribution specifically, they're testing the ability to maintain plural status consistency, which is doable by much less intelligent systems. It's also standard to ask "What does the pronoun '[pronoun]' refer to in that sentence?" rather than "What was too small?" or similar, as using descriptive words is also giving away information that you're meant to be testing the model on. But plenty of people make that mistake, so it's not that important.

Summing up, out of your four (not an acceptable sample size to form evidence) attempts to subject a SOTA LLM to Winograd schemas, two of them were actually Winograd schemas and only one of those got answered correctly. That isn't evidence that LLMs can pass novel Winograd schemas. If you actually come across such evidence, you should post it publicly on at least a blog page if you can't access the infrastructure to publish the results in a paper, even though the finding would certainly be novel enough to deserve a paper. As I said: there is no evidence that LLMs can pass novel Winograd schemas as of yet. We could get said evidence tomorrow, but it doesn't exist right now.

I would put a condescending moral of the story here, but I don't feel any particular need to belittle you, and instead hope you just learn from the information as presented.

Hey — you're quite correct and I appreciate this feedback. Thank you.

Moral of the story: before one makes a strong claim such as "there isn't any evidence a model can do X", it's often worth making a minimal effort to investigate one's own claim.

Do you think this was a strong refutation ? I just flipped a coin for fun and also got 3/4 correct.

I understand your point, but I mean, something that's intelligent really should've got the second question in that test correct.

Reply on news.ycombinator.com