Hacker News Reader

Comment by nutanc | Hacker News Reader

The question was about movie scripts and children stories. Not about paragraphs. Sure it works ok for paragraphs(even then it fails many times) but the moment you cross the 2000 token limit, it does not work. When GPT3 was first released we did a lot of experiments on summarisation. We discussed a lot in the OpenAI slack. But no one could come up with a right prompt for summarisation. Yes, it works for toy paragraphs and is fun to show off. But I wouldn't build a summarisation startup on top of GPT3. Yet.

View on HN

Do it in stages. Chop the text up into sections (using GPT-3 if you have to) and then summarize each page/section/chapter in isolation. Then concatenate the summarizations and summarize again. 2048 tokens is like >5KB of text — it’s not that limiting.

The context is lost. We have tried all that. We tried that when there was free usage. Now it is too expensive. If you believe it can be done please do go ahead. There is a huge market out there for summarizing novels. No one has cracked it yet.

The GPT-3 you had access to when it was free is quite different than what’s deployed today, and its ability to handle long-form inputs is its most apparent change. I’ve gotten it to give good summaries of ~5000 character texts but I admit I haven’t gone longer than the context length.

Edit: Here’s an example summarizing 8,294 characters of Harry Potter fan-fiction: https://twitter.com/goodside/status/1561213457374011392?s=21...

Update: I got this working for all 14,410 characters of the first chapter of that fanfic. See reply in same link above.

Maybe if you've read the story it'd seem good but as someone who hasn't, I consider it a poor summary, especially the latter half.

The goal wasn’t to get the post possible summary, which could be done with a more elaborate prompt detailing exactly how the summary should go. I was just demonstrating that it can get to the end of 14K chars of text and still remember both the task at hand and enough information to solve it.

And it failed at that. As the text gets longer and longer, the lack of synthesis across the contexts you've glued together becomes ever more glaring. It's a decent hack but not a solution. More research is needed for how to forget properly across long contexts.

I assume you’re familiar with this, but if not: https://openai.com/blog/summarizing-books/

Familiar with it. But as they themselves say,the fine tuned model(not released) achieves 5/7 rating for only 15% of the time. So 85% of the time the results are not satisfactory.

Wow, didn’t notice it was that low. Thanks for walking me through this — I might try to tackle this problem next. Very interesting.

To be fair, one of the earlier comments said "I wouldn't build a summarisation startup on top of GPT3". But it seems like GPT3 is more than capable of producing summaries that are passable, and would be far cheaper than humans. It does seem feasible that one could build a startup based on that

Reply on news.ycombinator.com