The Killer Use Case for LLMs Is Summarization | Hacker News Reader

The Killer Use Case for LLMs Is Summarization

sebastianmellen.com

58 pointssebmellen65 comments

Summarization fits into my mental model of LLMs as "calculators for words". https://simonwillison.net/2023/Apr/2/calculator-for-words/

I think it's one of a small set of killer related use cases:

- Summarization

- Data and fact extraction, which is strongly related to summarization. Find all factual assertions in this news article. Extract the names of every person and location mentioned.

- Question answering: given this context, answer this question. That's key to implementing Retrieval Augmented Generation, eg https://til.simonwillison.net/llms/embed-paragraphs#user-con...

Aside from these summarization-adjacent applications, the thing I find most exciting is LLMs incredible aptitude at generating code.

Code is especially interesting because it can be automatically executed (in a sandbox), which means many hallucinations can be automatically detected and fixed!

Summarization is all I have ever used Claude for, and it is excellent.

They're incredibly good at data extraction. You can start from zero and have a decent pipeline in hours.

I never coded but I know what's needed and how things should be arranged to work. Result: I've developed and deployed a fully fledged web service both for human and machine use. Backend, front-end, endpoints, db, integration with our company systems, authentication, testing and deploy. In 5 sessions of 4 hours each. It was incredible while it was happening. I pay openai.

> This Connection is Untrusted Go Back The owner of costantini.pw has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

I tried to see what it built for you, and this was the only url in your bio.

That's a personal profile page which has an expired certificate, sorry for that. If you're (or anyone really) interested in what I did and would like to try it out for curiosity and feedback, please feel free to reach me at j@costantini.pw as I'd be pleased to share it. Thanks

> many hallucinations can be automatically detected and fixed!

how? you just set off my halting-problem neuroses with that claim, since how do you know what inputs to test with, in your idea?

Have you tried ChatGPT Code Interpreter (now renamed to Advanced Data Analysis)?

It's quite something to watch. It writes code, then executes it, gets an error, adjusts the code to fix the error, runs it again...

I've seen it loop like that 4-5 times before it gets to code that works.

The halting problem is one matter (and LLMs will often write terrible code performance wise, though you can often determine this automatically with a myriad of approaches and either fix it or discard it and start anew) but in terms of "will it run" this is more akin to the old "will it blend" youtube channel than any complicated mathematical problems which require Church or Turing to intervene on.

This is a simple smoke test and the biggest problem I've found is that it simply writes code which probably worked in the past in its training data (or is very close to what would work) but is just so ever so slightly off.

This was just a front page article today because flask3.0 broke a login module, so it's still going to be a problem no matter how recent your training data is.

You don't even need a python interpreter fired up for the first checks, in fact, you can generate the tree of valid libs, functions, their sigs, etc within your interpreter, save it as a json and and directly parse incoming python markup to evaluate if it is valid or not (at least, for your particular environment)

I'm actually a bit surprised that openAI themselves doesn't do this since they have a fixed sandbox and it would save them compute cycles to handle the "little fix" directly with either non-LLM code or a much smaller "LLM call" to figure out how to call a function for lib X version Y which is similar to this.

Or you could probably just do some non-insignificant double digit percentage of the time with Levenshtein distance (or similar fuzzy matching) and having an internal db of every libraries functions and function signatures for every version and automatically identifying where something changed and reducing your problem space to those.

The Halting problem is only relevant if you want to be completely correct. If you also reject code that looks weird, then you have bypassed the problem entirely.

Yes, I'm assuming the need to be completely correct.

But you have to nail down a definition of “looks weird”?

In a sense, LLMs are already lossy compressors. Summarization is sort of just turning up the lossy part so it seems to be a natural use-case for them.

I would agree that summarization is a killer use case, but would actually argue for three:

1. Condense (summarize, synthesize, extract, etc.)

2. Expand (explain, copilot, suggest, etc.)

3. Transform (translation, style transfer, switch modality between text/image/audio, etc.)

All of the major consumer use cases of LLMs at this point boil down to one of these three core operations. To the extent that LLMs allow you to perform "magic", that magic is comprised by a chain of {condense, expand, transform} operations, each of which could be performed in half an hour by a reasonably informed college graduate with access to Google.

Even if chain length maxes out at 3 or 4 for production-grade applications, that solves a frightening number of problems.

Chatting seems very popular too.

I don't understand why this got downvoted, but there's absolutely a huge unserved(or, rather, anti-served) and somewhat-looked-down-upon market here, and it's actually seemingly more women then men when i ran some numbers.

If LLMs can be a cheap and instant therapist (even if they are just somebody who listens) then they could do a massive amount of good in this world.

I'm an early engineer at a startup that has nothing to do with AI. I use LLMs for 2 and 3 all the time. A lot of times we'll release a new feature and I'll write my own fairly technical summary of it, then I'll ask an LLM to either target it for a specific channel (e.g. create a version of the post targeting execs vs creating one that's focused on users) or condense it down if we're posting a summary onto a social media channel. In all cases I check the LLM's work, but I rarely have to do much.

When I led teams at a big tech, I generally had a human whose job it was to do this kind of copy refinement/editing when my teams released new features. I find the LLM to be just as good, though obviously our reach is much smaller and the consequences of a mistake are much milder given our size. This is the kind of productivity boost I think of that LLMs offer; pre-LLM we would probably contract out a copy editor for this kind of work but now we do it all in-house and don't have to deal with working with a contractor's hours.

„However, LLMs appear to be quite good at knowing what they do and don’t know“ Really? I think LLMs have not the slightest idea what they know and you have to go through great lengths to stop them from making stuff up.

They certainly can be: https://huggingface.co/docs/transformers/model_doc/t5, and moreso if you plug in data to them.

If you paste text into ChatGPT and then ask about specific information within the text, it's pretty good at answering.

I think the "killer use case" is slightly more general: operations in linguistic space, of which summarization is one. Some other useful uses include:

- the inverse of summarization; given a bulleted list of facts, write a sensible set of paragraphs presenting those facts

- semantically searching in large documents

- make some text sound more professional / casual

- translate some text between human languages (they're not perfect at this yet, but they're surprisingly good, and I think they will be the way to do translations in the future).

- expand abbreviations given some context

The key here is that none of these depend on the language model injecting factual information into its output. All of the facts are encoded in your input - the language model just does some operation in linguistic space. If you don't give it the opportunity to give you incorrect assertions, it won't give you incorrect assertions.

Another application is to use the LLM to build an embeddings database, and then do semantic search only within that database. Don't just let the LLM talk about whatever it wants to hallucinate up, force it to use a set of known good answers while accepting natural language input and summarizing documents in the known good dataset.

everything there sounds great (though i suspect for actual human-passing linguistic register-transfer we will need something different than the bog standard GPT models because everyone has this six sense for things being written by it and, particularly in English, register is one of oldest forms of 'discrimination' in the language/culture, and certainly persists to the current day. And there's a huge competitive advantage to being able to have an LLM write things for you but make it sound like you yourself wrote it.)

For translations considering how 'mature' the field is relative to everything else I'm a little surprised we haven't seen any rigorous benchmarks on round-trip translations, especially given that you aren't even limited to natural languages. You can take text, flip it into Klingon, and back, if you so chose.

>- the inverse of summarization; given a bulleted list of facts, write a sensible set of paragraphs presenting those facts

Except this one, this sounds unintentionally evil for some reasons i can't entirely put my finger on. Perhaps the inevitable conclusion of countless GPU farms spamming terabytes of text at each other predicated upon some short bullet point lists and the fact that "those are our marching orders"

I guess Fundamentally I would like to live in a future where LLMs can only decrease the length of text written by humans, never increase it or fabricate it.

> The key here is that none of these depend on the language model injecting factual information into its output.

This feels like an under-appreciated insight. There are lots of valuable (and in some cases previously difficult, expensive, or impossible) linguistic manipulations that fall under this lower-risk end of spectrum.

Kind of .. depending on how sensitive you are to accuracy.

Also also, we've had powerful summarization techniques for decades now (deep-learning, or otherwise) - and they aren't susceptible to the hallucinations of present day Generative LLMs.

Which summarization models aren't susceptible to hallucination? Is there any good writing out there about this issue?

I found an 'old' (2016) reddit thread describing how auto tldr works, which doesn't suffer from hallucination (because it's only a statistical model that outputs original sentences from the text):

https://old.reddit.com/r/askscience/comments/4s5b5q/how_exac...

https://huggingface.co/docs/transformers/model_doc/t5 is quite good for summarization!

>we've had powerful summarization techniques for decades now (deep-learning, or otherwise) - and they aren't susceptible to the hallucinations of present day Generative LLMs.

do you mean BERT, Seq2Seq and similar?

My biggest takeaway from summarization tasks was that the benchmarks/evals seemed rather.. lacking.

Like many use cases for LLMs, this seems like it would benefit a lazy student trying to avoid work. Rolling in some statements that are related but not mentioned in the input isn't a bad thing when you're studying for a test.

However, for most use cases, falsely attributing statements is a terrible failure case for a summarizing tool.

I couldn't see a date on this piece but the Internet Archive first captured it on May 14th: https://web.archive.org/web/20230514232943/https://www.sebas...

(Relevant here because even five months is a long time in LLMs!)

Update: turns out there's a date on https://www.sebastianmellen.com/post/ - this post is from 18th March, so seven months ago.

Thanks, my initial response to summarization as killer feature: "is it though?" but the "historic" nature of this post makes a bit more sense now. Unlike crypto, there are many diverse use cases for LLM. While summarization is useful I don't think it's killer.

I would argue it's translation with summarization as a close second, but not for people who spend much time on HN since their command of English is almost always good enough.

Tools like DeepL already were cutting into the field of translation/localization, but now we can have something which can basically translate ANY language to ANY language (even things like fictional languages from e.g. LoTR, although that may be very hallucinogenic due to the paltry amount of training data) rather rapidly (gpt3) or rather slowly but more accurately (gpt4) including understanding of things like metaphors and idioms.

Not to mention you can probably fix some of the choice of vocab and style just by specifying (once converted to target language) "reword in the stylings of X" or countless other multi-invocation approaches to the previously very expensive and artistic task of converting human thought from one variation to another.

The biggest problem is probably over-training on english (and being dependent on openai), but even Microsoft made a LLaMA[!] finetune just for translation which iirc was significantly better than the non-translation versions from everyone else

It's one of the things LLMs do really well. Summarization, translation, and guided blithering for generating ad copy work very well. Sometimes a bit of understanding seems to appear. Sometimes that's illusory.

Here's a video where someone tries to use GPT-4 to read a drawing and generate G-code for a milling machine.[1] He puts in a simple textbook problem drawing, and reasonable G-code comes out. The trouble is, GPT-4 plagiarized the answer from a Chegg homework assignment which has the same design of 4 holes in a square. This is clear because the problem didn't specify the material thickness, and the GPT-4 result used the material thickness from the Chegg homework assignment.

For harder, but still easy drawings, GPT-4 does worse. It's clear there's no underlying geometric model.

[1] https://scanalyst.fourmilab.ch/t/programming-a-computer-nume...

I really think LLMs are great for translation between programming languages. I don't trust them to write complicated code, but when I am learning a new language or framework, the fastest way for me is to write a function in X and ask it to translate it to Y. It doesn't get fancy, it almost always compiles and if it makes a mistake it's not too bad.

I'm not familiar with the summarization or NLP space really but I remember ~2011-2015 I signed up for a couple of daily email services that summarized a number of news articles and the summaries were fantastic. I don't even remember what they were called, they eventually sold out with ads and worse formatting/summaries to make money I guess. I often use them as an example of 1) why LLMs are a bit old news for the summary use case and 2) how various LLM use cases will probably also be ruined because for a lot of people tools like that seem novel and useful but all I can see is onboarding to more advertising.

So to someone who is actually knowledgeable in this space, are LLMs really that much better than whatever we had 10 years ago? Is this tech the key to some features we truly didn't have before?

Not an NLP expert but the biggest difference in my experience is guided focus, so to speak. When summarizing something huge like the US Code, for example, you can tell the LLM to focus on specific topics and anything adjacent to them so that it ignores irrelevant details (which is usually >99.9% of the text in my use case). The word relationships encoded in the LLM are really good at identifying important adjacent topics and entities.

LLMs are also really good at the harder NLP problems like coreference resolution, dependency parsing, and relations which makes a huge difference when using recursive summarization on complex documents where something like "the Commisioner" might be defined at the beginning and used throughout a 100,000 token document. When instructed, the LLM can track the definitions in memory itself and even modify it live by calling OpenAI functions.

Might I ask how you use OpenAI's function calling here? That's the one bit of their functionality I haven't really explored.

I use OpenAI function calling most of the time I use the OpenAI API since that's the easiest way to get structured data and implement retry logic.

The simplest implementation is "retrieve_definition(word_to_lookup, word_to_replace)" with some number of tokens at the beginning of the prompt dedicated to definitions. You can use a separate LLM call with a long list of words (without their definitions) to do the actual selection since sometimes there might be ambiguity, which the LLM can usually figure out itself (it can also include both definitions when it's too uncertain if instructed).

A more complex variant does multiple passes: first pass identifies ambiguous words in each chunk, second pass identifies their definitions, third pass does actual summarization using the output of the previous passes to craft the prompt.

Interesting so maybe not my trivial "summarize an article" example but clearly the upper bound on what's possible is higher and more interesting.

Just because the model is unlikely to come up with novel insights doesn’t mean it’s not a useful conversationalist. I have all sorts of ideas I like to bounce off of ChatGPT and it’s good at coming up with reasonable sanity checks in the meantime.

Don’t forget classification. Even the zero shot classification can be quite decent, but put a solid prompt together, or fine tune a set of examples and it’s way easier with an LLM than previous methods in my experience.

My favourite extant example of this GPT4’s 18-way rule-based reward model classifier, appendix A https://cdn.openai.com/papers/gpt-4-system-card.pdf

LLM summaries are in the sweet spot: more specific than typical website short glossaries, easier to read quickly than Wikipedia, and well-prioritized for industry professionals.

LLM summaries are providing excellent results (IMHO) for team training for technology groups, including for UI/UX, product management, innovation partnerships, software programming, and the like.

If you want to see some of these, my GitHub has a six of these LLM summary guides for tech people: https://github.com/joelparkerhenderson

Is there any work being done on summarization based on loading content in and doing some sort of manipulation on the internal representation? It seems to me that this ought to be a better solution for summarization than simply asking the model to summarize and hoping its training set taught it how to do what you want. Seems like it would generally fix the potential confabulation issues as well w.r.t. summarization at least.

For me, the killer user case is having it as an AI assistant. A few months ago, I've started building a machine learning model for a keyboard iOS App as a solo dev, without python or deep learning knowledge.

The model I wanted to build, needed text correction, word completion and prediction in one pass. I'm pretty confident, it was an impossible task without the help of chatGPT, first GPT-3.5 and then GPT-4.

Oh my I really hope that you are building the app of my dreams.

I’m a an awful typer on my phone and on my desktop. Autocorrect is ok but I honestly don’t understand how we have all this amazing ML in the wild but we still seem to have the same shitty autocorrect.

My dream autocorrect (powered by ML) learns how you type and how you speak. It learns what mistakes you always make (due to muscle memory) and corrects based on your past mistakes. Furthermore my dream autocorrect actually understands what you’re trying to say and corrects words based on the context of the conversation rather than how close the mistake you made is to some other word.

For example, if someone texts me to say “I’m at the restaurant” and I type with a mistake “I’ll be right theyre” the autocorrect should understand the context of the conversation and rather than correct to “they’re” which is thinks is the correct spelling for this mistake, or autocorrects to “there” because it’s looking at the wider conversation. This is a poor example but I hope you get the point. I really don’t think what I’m asking for is hard. Apple should/could easily improve autocorrect

I would love to build that keyboard, but unfortunately there is a RAM limit for keyboard apps. My current model is not so smart as I would like. For fun, I just tested your example, using only 8 tokens (current limitation): "restaurant I’ll be right theyre" and it's corrected to "restaurant I’ll be right there", there is hope :) I'm pretty sure that I can expand it a few more tokens without much impact in memory and latency.

That’s awesome is this for iOS or Android though? I thought iOS doesn’t support other keyboards (like Swiftkey)

it's iOS, there is an app category about 3rd party keyboards in both App Stores. Not a listed app category, but a type of app.

I recently went to a talk by Don Syme, the creator of F#. He was talking about the LLM tools he is working on at GitHub Next for summarising PR and code base content. He seemed pretty upbeat about the impact these tools will have.

Obligatory GPT4 summary of the article.

"LLMs: The Summarization Superheroes of the Modern Era

LLMs, like the one you're reading from, shine brightest when tasked with summarization. They may not craft unique insights, but they excel in distilling vast amounts of information into digestible bites. As information overload continues to plague our modern age, a tool that can swiftly and effectively condense info is invaluable. Bing Chat's emergence hints at a future where LLM-driven search engines index and summarize vast content troves, from books to blogs. But it's not just about search. Inspired by a recent Hacker News discussion likening the LLM movement to the early days of the web, it's clear that these models will reshape how we handle information, from helping CEOs extract corporate insights to tidying up the sprawling mess of unstructured data in enterprises. While LLMs have diverse potential applications, summarization stands out as their killer use case, at least for now."

Does anyone have any experience for how the generalized LLM's like llama/gpt4 compare to much smaller, purpose built summarizers like some of the FLAN variants?

the killer use case for me is quickly finding specific commands involving parts of tools that i don't use often. how do i turn the last three commits into a patch using magit? how do i grep for this weird pattern and exclude this directory? give me an elisp snippet that runs my last "compile" command that i used without prompting me, and bind it to f5.

It's pretty amazing how great it is at bash/coreutils/etc in addition to everything else.

The fact that it can output quite valid ffmpeg commands was the commandline cherry on the cake for me

What is the most capable LLM for summarising stuff like long educational articles perhaps?

GPT-4

GPT-4 is usually the answer to “which llm is most/best…”

But if the article is longer than the context window, you can try either RWKV or Claud-200k

Has Claude 200k ever been made available? I've only found ways to access the 100k version.

GPT-4 if it's less than 8,000 tokens (or 32,000 if you have access to that via the API).

Claude for up to 100,000 tokens.

Anthropic's Claude 2 is great for pasting large text files in, and Flan T5 seems to be excelling at summarization benchmarks: https://huggingface.co/docs/transformers/model_doc/flan-t5.

I'd be interested to hear more about companies that are successfully doing this now.

We use https://read.ai for every meeting, and it's quite good (not affiliated, just a customer :)

I find using the word "regurgitate" when talking about LLMs pretty ironic since the word itself is almost never used except by people who "regurgitate" arguments from other people.

Also, I wish people would reduce the use of such derogatory terms in arguments because it distracts from the core reasoning.

I don't think "regurgitated" is a derogatory term at all, at least in the sense that I'm using the word — "[to] repeat (information) without analyzing or comprehending it." [0]

[0]: https://www.google.com/search?hl=en&q=regurgitated%20definit...

Related thought - sometimes I ask for a "tldr" of long articles and people shame me for being lazy or something like that. I think this is really illogical. There's nothing wrong about reading a summary before or instead of reading the full article

Reply on news.ycombinator.com