Comment by tracerbulletx | Hacker News Reader

Comment by tracerbulletx | Hacker News Reader

Parent Full thread

In every post about LLMs there is someone to blindly say something like this.

When in reality if you ask ChatGPT for 10 good movies from this year you will get this.

Anora - Directed by Sean Baker, a compelling drama about the life of a sex worker in Coney Island.

Challengers - A provocative tennis drama directed by Luca Guadagnino, starring Zendaya.

Dune: Part Two - Denis Villeneuve's continuation of the epic science fiction saga.

Furiosa: A Mad Max Saga - An action-packed prequel exploring the origins of Furiosa, directed by George Miller.

Inside Out 2 - Pixar's sequel that dives deeper into the complexities of human emotions.

Wicked - A musical fantasy adaptation directed by Jon M. Chu . The Zone of Interest - A thought-provoking film about Auschwitz, directed by Jonathan Glazer.

The Idea of You - A steamy romance starring Anne Hathaway.

Hit Man - A comedy thriller starring Glen Powell.

The Outrun - A powerful drama about a recovering alcoholic, starring Saoirse Ronan.

Let me know if you'd like more details about any of these!

Which is a great list.

Those descriptions are less detailed than the information you will see on basically any streaming interface and yet it still manages to not being very good. For example, no person who had actually seen Anora would describe it as "a compelling drama about the life of a sex worker in Coney Island".

I haven't seen Anora so I'll give you that one, but you cited that as if it was just one of many examples, when in fact I think it's the only one, as all the other descriptions seem reasonable.

Originally the problem was supposedly that it would hallucinate complete and utter gibberish, but now here we are quibbling over one example and insisting that maybe it's not quite as good as alternative descriptions.

The gap between what was produced and what you're looking for is small enough that I think it could be covered with some slightly tweaked prompt instructions.

I'm not saying you're wrong but want to note how the goalposts keep seeming to shift whenever we talk about these capabilities.

I'm not Alupis. I can't and am not trying to speak on their behalf. I'm therefore not moving the goalposts they established. I'm making my own related point.

That point is that the information provided above about these movies is worthless. It does not add any new value beyond what would already be available in the streaming interface. Several of the descriptions are nothing but the genre and one person involved in the making of the movie. And yet even with these descriptions being incredibly short and vague, they still manage to contain at least one misleading summary.

I'm aware that you're a different commenter but you are addressing yourself to a comment that was in reply to them and therefore not necessarily appropriate to measure such a comment against entirely new criteria that you want to bring into the conversation.

Despite your protestations to the contrary, these descriptions seem perfectly fine in that they're accurate and meaningful. And it if you want to start getting fast and loose with all kinds of new extra criteria and requirements for what it's supposed to do, they all seem squarely within the reach of the capabilities on offer, with some prompt tweaks.

>these descriptions seem perfectly fine in that they're accurate and meaningful

The description of Wicked doesn't mention either The Wizard of Oz or the Broadway musical. So yes, the descriptions don't contain obscene mistakes like calling Wicked a courtroom drama. If that is enough for you to call these "accurate" while ignoring the vagueness or the 1 in 10 failure rate on the Anora description, fine by me. But you must have some weird definition of the word "meaningful" to apply that to descriptions like the one of Wicked. That simply isn't a helpful way to describe that movie.

The comment thread you're at the end of started with this:

> I would expect nothing but hallucinations and nonsense coming out of any LLM regarding recently-released movies (aka. the ones you often find on flights).

The comment that replied to it (the one that you're arguing against) provides evidence that proves it wrong. You are correcting someone who isn't incorrect, and I think the person you're responding to is very justified in saying you're moving the goalposts here.

A reply downthread is not an endorsement of everything said upthread. I'm happy to discuss the points I made, but I’m not going to be made to defend something I didn’t say.

Well if you're not endorsing what was said upthread, then your comment is a complete non-sequitir. The parent comment said "LLMs can't give movie recommendations for recent movies because they'll hallucinate or spout nonsense", the next comment responds with a list of accurate movie recommendations, and then you come in and say this:

> Those descriptions are less detailed than the information you will see on basically any streaming interface and yet it still manages to not being very good.

The points you made were not relevant to the discussion at hand. It's like if people were having a debate about where to find the best tacos in town and you stepped in to say "tacos aren't as good as hamburgers, you know" and then got upset that nobody wanted to debate that point with you. It's not everybody else's fault if you don't understand how conversations work!

I don’t know why you are letting that one reply define the bounds of this conversation. My comment was directly relevant to the first comment in this thread and the comment I was replying to.

> I haven't seen Anora so I'll give you that one

It was literally the first movie in that list.

You tried making a counter example and the first part of it was already wrong.

That’s the point. Not that it _cant_ give good answers, but whether it does or not is a crap shoot.

Now to analyze how correct it was we need to verify each movie it gave… It’d be faster just to read the movie descriptions.

I _have_ seen Anora and I think that description is perfectly fine. It certainly isn't "hallucinations and nonsense" which is what the parent comment is claiming. What part of that description do you consider "wrong"?

You're still trying to imply that the list as a whole is as inaccurate as that one particular example.

And I think that's quite obviously not the case, most, probably every other example on the list is just fine.

I’m saying that because of the one incorrect example, I can’t just assume that the rest are accurate.

I now need to either trust a machine that I know gives incorrect information (as demonstrated by the first example) or I need to verify each example.

> probably every other example on the list is just fine.

Why don’t you check IMDb and let me know?

While you’re at it, don’t think about how much faster it would’ve been if you just looked up popular recent movies on IMDb or rotten tomatoes.

For comparisons, Wikipedia's opening paragraph for Anora reads:

"Anora is a 2024 American comedy-drama film written, directed, and edited by Sean Baker. It follows the beleaguered marriage between Anora (Mikey Madison), a young sex worker, and Vanya Zakharov (Mark Eydelshteyn), the son of a Russian oligarch. The supporting cast includes Yura Borisov, Karren Karagulian, Vache Tovmasyan, and Aleksei Serebryakov."

I haven't seen the film, but it doesn't seem incompatible with ChatGPT's briefer description.

You see the issue here, right?

Instead of just checking with a first party source, you ask a statistical guessing machine for an answer.

There was a disagreement about the answer, so we needed to dig deeper.

You bring up Wikipedia, a 3rd party source of information. That description could also be wrong (it’s probably not, but stick with me)

Instead of just checking with a first party source (IMDb is very easy to search on), we went through several layers of obfuscation.

This was an issue for Wikipedia early on, but it has citations, at least. AI doesn’t and doesn’t have an army of people constantly fact checking every answer generated either.

There’s no benefit to asking AI for information like this. Especially since the in flight summary has accurate information that’s more than “drama, sex worker, cony island”

Maybe something like perplexity is better, since it has citations, but I haven’t tried it for very long yet.

These details are already available in the in flight entertainment interface

They were responding to a commenter suggesting it would produce completely unusable results, the question was never about whether the results produced would be redundant.

I know that any mention of fallacies, valid or otherwise, causes instinctive eye rolls, but in this instance I agree with them that this amounts to moving the goalposts.

This type of response is called moving the goal post. When someone responds to one claim, the claim is changed to something different which was not part of the original argument. This is debating in bad faith.

Great! Now show me a system that can verify that list for accuracy as well. Not to be flippant, but this is the complaint. You can't approach outputs uncritically. And no I don't want it to be as unreliable as a person who also forgets how English or basic knowledge works at random intervals.

They were responding to a comment that suggested that this was a category where the only thing you would get is unintelligible gibberish.

You don't even seem to be disputing the actual results here, just gesturing towards a kind of philosophy class exercise of whether we can ever "really" verify its accuracy. I see Wittgenstein's name increasingly tossed around in these parts (a good thing!), so I'll just note that one of the reasons he's hailed as one of the great philosophers of the 20th century is because he felt these puzzles about "really" knowing were frivolous.

I don't think I agree that what's needed here is some new and extra process of verification. I think the same usual quality control criteria that are already being used are good enough in this case.

Yes, like, how are corporations (like movie productions in this example) supposed to control their message?

> Great! Now show me a system that can verify that list for accuracy as well. Not to be flippant, but this is the complaint. You can't approach outputs uncritically.

In general you can't, but surely it's not that big a deal if ChatGPT offers an inaccurate summary of a movie you're about to use to kill time on a flight? I suppose it becomes important if, e.g., you're relying on it to tell you whether a movie is appropriate for children, but, if you're just asking it whether a movie is worth watching, that's a question that doesn't have an objective, factual answer anyway, so a hallucinated answer is probably about as useful as that of a not-previously-known reviewer.

If I invested money into a film, I would want its representation in the world to reflect what the movie is about at the very least.

> If I invested money into a film, I would want its representation in the world to reflect what the movie is about at the very least.

Sure, but that's the filmmaker's interest. As someone sitting on a plane trying to decide whether to watch a movie, I care about my interest, not that of the person who made it. I'm not particularly arguing for the use of ChatGPT here (I wouldn't use it), just that the risks it usually poses are fairly minimal in this case.

You're forgetting the information hazard of five years from now someone mentioning a movie and you saying "oh I didn't want to watch that because of the car chase" and everyone looks at you funny because it is a film set in the 1700s about a carriage driver.

Movie reviews are amongst the most subjective things, how do you define “verify” and “accuracy”?

If I run a business, I guess I would have the ability to bring a libel suit would be one way?

The system is I'm a movie nerd who knows all the best movies of the year and have seen them all.

Reply on news.ycombinator.com