Hacker News Reader

Comment by simonw | Hacker News Reader

Summarization fits into my mental model of LLMs as "calculators for words". https://simonwillison.net/2023/Apr/2/calculator-for-words/

I think it's one of a small set of killer related use cases:

- Summarization

- Data and fact extraction, which is strongly related to summarization. Find all factual assertions in this news article. Extract the names of every person and location mentioned.

- Question answering: given this context, answer this question. That's key to implementing Retrieval Augmented Generation, eg https://til.simonwillison.net/llms/embed-paragraphs#user-con...

Aside from these summarization-adjacent applications, the thing I find most exciting is LLMs incredible aptitude at generating code.

Code is especially interesting because it can be automatically executed (in a sandbox), which means many hallucinations can be automatically detected and fixed!

View on HN

I never coded but I know what's needed and how things should be arranged to work. Result: I've developed and deployed a fully fledged web service both for human and machine use. Backend, front-end, endpoints, db, integration with our company systems, authentication, testing and deploy. In 5 sessions of 4 hours each. It was incredible while it was happening. I pay openai.

> This Connection is Untrusted Go Back The owner of costantini.pw has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

I tried to see what it built for you, and this was the only url in your bio.

That's a personal profile page which has an expired certificate, sorry for that. If you're (or anyone really) interested in what I did and would like to try it out for curiosity and feedback, please feel free to reach me at j@costantini.pw as I'd be pleased to share it. Thanks

They're incredibly good at data extraction. You can start from zero and have a decent pipeline in hours.

> many hallucinations can be automatically detected and fixed!

how? you just set off my halting-problem neuroses with that claim, since how do you know what inputs to test with, in your idea?

Have you tried ChatGPT Code Interpreter (now renamed to Advanced Data Analysis)?

It's quite something to watch. It writes code, then executes it, gets an error, adjusts the code to fix the error, runs it again...

I've seen it loop like that 4-5 times before it gets to code that works.

The halting problem is one matter (and LLMs will often write terrible code performance wise, though you can often determine this automatically with a myriad of approaches and either fix it or discard it and start anew) but in terms of "will it run" this is more akin to the old "will it blend" youtube channel than any complicated mathematical problems which require Church or Turing to intervene on.

This is a simple smoke test and the biggest problem I've found is that it simply writes code which probably worked in the past in its training data (or is very close to what would work) but is just so ever so slightly off.

This was just a front page article today because flask3.0 broke a login module, so it's still going to be a problem no matter how recent your training data is.

You don't even need a python interpreter fired up for the first checks, in fact, you can generate the tree of valid libs, functions, their sigs, etc within your interpreter, save it as a json and and directly parse incoming python markup to evaluate if it is valid or not (at least, for your particular environment)

I'm actually a bit surprised that openAI themselves doesn't do this since they have a fixed sandbox and it would save them compute cycles to handle the "little fix" directly with either non-LLM code or a much smaller "LLM call" to figure out how to call a function for lib X version Y which is similar to this.

Or you could probably just do some non-insignificant double digit percentage of the time with Levenshtein distance (or similar fuzzy matching) and having an internal db of every libraries functions and function signatures for every version and automatically identifying where something changed and reducing your problem space to those.

The Halting problem is only relevant if you want to be completely correct. If you also reject code that looks weird, then you have bypassed the problem entirely.

Yes, I'm assuming the need to be completely correct.

But you have to nail down a definition of “looks weird”?

In a sense, LLMs are already lossy compressors. Summarization is sort of just turning up the lossy part so it seems to be a natural use-case for them.

Summarization is all I have ever used Claude for, and it is excellent.

Reply on news.ycombinator.com