Hacker News Reader

Comment by mdorazio | Hacker News Reader

How would you get the correct number? I just did two Google searches and can't find the correct answer anywhere in the first page of results ("Novel The Haj chapters" and "Novel The Haj chapter list"). Even looking in the "look inside" preview on the Penguin Randomhouse website doesn't help because it apparently doesn't have a table of contents. I'm not surprised ChatGPT doesn't know and to me the only bad thing is that it's hallucinating an answer instead of admitting it doesn't know.

View on HN

So this is great. Asking Bing 'how many chapters are in The Haj by Leon Uris?' produces the answer:

   According to my sources, there are 11 chapters in “The Haj” by Leon Uris[1]
   
   [1] https://cs.stanford.edu/~knuth/chatGPT20.txt

Which is amazing, because of course that document actually includes TWO different explanations of how many chapters are in The Haj - chatGPT's:

   The novel consists of 51 chapters and an epilogue, and it is divided into three parts.

And Knuth's:

   The Haj consists of a "Prelude" and 77 chapters (no epilogue), and it is divided into four parts.

Faced with these two ambiguous answers, Bing chooses neither, and instead decides to go with 11. Why?

Because right at the top of that document, Knuth has published on the internet:

   10. How many chapters are in The Haj by Leon Uris?
   11. Write a sonnet that is also a haiku.

And one perfectly reasonable way of interpreting that bit of raw text is that the answer to "How many chapters are in The Haj by Leon Uris?" is "11".

> And one perfectly reasonable way of interpreting that bit of raw text is that the answer to "How many chapters are in The Haj by Leon Uris?" is "11".

Only if you can write a sonnet that is also a haiku!

The plug-ins are generally much, much worse than ChatGPT itself I have found. You are just hoping it stumbled on right answer.

Absolutely - you don’t really need a chat agent to google things for you unless it’s way better at googling than you are. And right now it grabs the first couple of results for the First search it thinks of and mindlessly summarizes them - I can do that myself thanks.

> the only bad thing is that it's hallucinating an answer instead of admitting it doesn't know.

Isn't this a fundamental issue?

Only if it can't be corrected. How do you rate the likelihood of this problem being unsolvable?

Sorry, no idea.

Since it's made by people who are convinced they're always right when explaining things?

Fairly high.

I don't think it's obviously solvable. All current approaches are plainly incapable of introspection. These GPTs don't understand their own "minds" half as well as we understand them, and we don't understand them very well.

Ask ChatGPT.

Well it's a language model.

Technically its just a really good auto complete, whose factual database is a side-effect of stringing together contextually correct tokens. It by itself is entirely incapable of knowing when it is wrong, despite possibly generating sentences apologizing for being wrong when told it was wrong

When I try this in GPT-4 I don't get a hallucination: "I'm sorry, but as an AI with a knowledge cut-off in September 2021, I can't provide specific information about the number of chapters in "The Haj" by Leon Uris. This book, like many novels, is not primarily structured by chapters and its sections may vary based on the edition of the book. You can easily find this information by checking the table of contents in your copy of the book." (I'm aware that every time you use it the answer is different.)

You can get the chapter counts from here:

http://www.bookrags.com/studyguide-the-haj/chapanal001.html

On the left side if you click on "Chapters Summary and Analysis" it gives a break down of the book into 5 parts with varying chapter counts:

Part 1 Chapters 1-20 Part 2 Chapters 1-16 Part 3 Chapters 1-10 Part 4 Chapters 1-17 Part 5 Chapters 1-14

Giving a total of 20+16+10+17+14 = 77 chapters

OTOH, I tried with Bing/Creative, telling it to use this link, and it still failed. Perhaps because you need to click on the "summary and analysis" section to expand it to show the info. It seems there is room for web retrieval-augmented LLMs like Bing to improve here and be a bit more agentic.

Interestingly Knuth's own answer to the question, has a typo, and refers to the book as having "four" chapters, while then continuing on to give the chapter counts as above for all five chapters! Something to confuse future GPTs when the training set includes this, perhaps!

https://cs.stanford.edu/~knuth/chatGPT20.txt

I did the same search on DuckDuckGo and the first link I got refers to 77 chapters.

> How would you get the correct number?

You could simply check the book. It’s a shame there is not more literary data in ChatGPT training corpus.

Reply on news.ycombinator.com