Hacker News Reader

Comment by paxys | Hacker News Reader

I love it. It may not objectively be on par with GPT-4, but uploading a 100 page document and getting a summary in seconds is nothing short of miraculous.

View on HN

How do you know if it is correct or a hallucination?

Investigate, same as you would a paralegal etc. If it makes an assertion, contest it and ask where it found supporting evidence in the document for the claims made. Ask it to make the counter-argument, also with sources. Verify as needed.

That's what prompting is all about. Ask it to prove its statements. Ask it to quote passages that support its arguments. Then double and triple check it yourself. It isn't going to do the work for you, but can still be a pretty great reference tool.

Presumably one can test it with documents one has already read and knows before. If the summaries of the test documents are good, future summaries will probably be OK too.

> If the summaries of the test documents are good, future summaries will probably be OK too

But that is exactly what is problematic with hallucinations. It's a rare / exceptional behaviour that triggers extreme departure from reality. So you can't estimate the extremes by extrapolating from common / moderate observations. You would have to test a lot of previous documents to be confident, and even then there would be a residual risk.

As with anyone, if you are going to take action on something, double check it.

Maybe having it summarize a fiction book (outside of training data)?

is it an accurate summary tho?

In my experience it is very hollow. It skips details unless you force it to.

Gpt4 is still way better

Reply on news.ycombinator.com