Wait so they tested if it detects sarcasm, it failed, and then they were like "let's pretend this never happened" and wrote "We did not test for the ability to understand sarcasm"?
The first notable point is that the model caught that (on its first read, FWIW), even though the original human doing the labeling didn't.
And the second is that the researchers discovered this, and presumably discussed it. And yet when they wrote up the paper they not only dropped the content but denied that the analysis had been done.
> I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless.
Mocking irony, in the context of a negative review.
What I find fascinating here is how generative AI has inverted all our sci-fi tropes. I mean, sure: you're right! That's the way "sarcasm" is defined in most dictionaries. But you and I both know that as the language is actually used, the term means a whole host of techniques used to convey negative emotional content in language that is not directly negative. Your (correct!) dictionary pedantry isn't interesting to me. We've been here before.
But GPT-4 wasn't trained on dictionary rules. It was trained on actual language. And it's actually better at inferring this stuff than the pedants are. Our introvert brains have trouble teasing meaning like this and have to hide behind rules and structure. The computer doesn't.
To wit: the robot is us.
There is another message that the reviewer doesn't like the genre, but that isn't a comment on the movie.
There is no way a human reading this review would think its positive.
"I'll just tell you that it's slightly reminiscent of U Turn by Oliver Stone but is a way down in all artistic properties."
Oh and UTurn only scores 6.7 so its a bit of a crap movie to start with.
So I guess maybe it’s saying the human was wrong after all?
Maybe more like appreciation with some humour thrown in?
Are we all a bit confused and second guessing our own use and interpretation of language now ?
"he must have never in his life seen a flick about any small towns"
That's probably the least sarcastic sentence for me, because it's just a reinforcement of the opening statement:
"I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless."
I don't think it's a "positive review" I think it's neither, it's fairly neutral, and the author kind of suggests the reader watches the movie.
Out of interest I asked someone who has no idea about ChatGPT-4 and it's apparent sarcasm detection abilities and they didn't think it was sarcastic, albeit a bit 'weird' and poorly written. Confirmation bias?
We could say more, however, there are more important things to do...
He gave it a score of 7 which IMO is pretty high?
And yes it absolutely is subjective. That’s exactly the point and the power of these LLMs, to be able to handle the vagueness of human communication.