Hacker News Reader

Comment by mabedan | Hacker News Reader

I use it for summarization constantly. I made iOS/mac shortcuts which call Gemini for various tasks and use them quite often, mostly summarization related.

View on HN

How do you know its summaries are correct?

You already know that they aren't. Yesterday my wife and I were discussing Rønja Røverdatter. When we were kids it used to have danish talk over, so you could still hear the original swedish sound as well. Now it has been dubbed, and we were talking about the actor who voices Birk. Anyway, we looked him up and found out he was in Blinkende Lygter, which neither of us remebered. So we asked Gemini and it told us he played the child flashback actor of the main character... except he doesn't, and to make matters worse, Gemini said that he played Christian a young Torkil... So it even got the names wrong. Sure this isn't exactly something Gemini would know, considering Rønja Røverdatter is an old Astrid Lingren novel that was turned to film decades ago, and Blinkende Lygter is a Danish movie from 20ish years ago where Sebastian Jessen plays a tiny role. Since they are prediction engines though, they'll happily give you a wrong answer because that's what the math added up to.

I like LLM's, I've even build my own personal agent on our Enterprise GPT subscription to tune it for my professional needs, but I'd never use them to learn anything.

Because they mostly are, and even if not, it doesn't usually matter.

For example - you summarize a YouTube link to decide if the content of it is something you're interested in watching. Even if summarizations like that are only 90% correct 90% of the times it is still really helpful, you get the info you need to make a decision to read/watch the long form content or not.

For most things it doesn't matter, as long as its usually correct enough, and "enough" is a pretty low bar for a lot of things.

Can you give an example? And how would I know the LLM has error bounds appropriate for my situation?

> Can you give an example?

Recipe pages full of fluff.

Review pages full of fluff.

Almost any web page full of fluff, which is a rapidly rising proportion.

> And how would I know the LLM has error bounds appropriate for my situation?

You consider whether you care if it is wrong, and then you try it a couple of times, and apply some common sense when reading the summaries, just the same as when considering if you trust any human-written summary. Is this a real question?

Most recipe blogs have a "skip to recipe" button because they know you don't care.

Enough don't.

DuckDuckGo has a great tool for dealing with those ones: "Block this site from all results".

That doesn't get me their content.

Why would you reward those sites with clicks? Just go to the sites that actually respect your time.

Because I can get content I want there, and with a summarisatin option, it is irrelevant to me if they don't "respect my time" because it doesn't take any more time for me to get at the actual recipe.

"Get me the recipe from this page" feels like a place where I do really care that it gets it right, because in an unfamiliar recipe it doesn't take much hallucination around the ingredients to ruin the dish.

I guess I never come across that situation because I just don’t engage with sources that fluff. That is a good example, but presumably, there should be no errors there because it’s just stripping away unnecessary stuff? Although, you would have to trust the LLM doesn’t get rid of or change a key step in the process, which I still don’t feel comfortable trusting.

I was thinking more along the lines of asking an LLM for a recipe or review, rather than asking for it to restrict its result to a single web page.

Doesn't matter if they get it wrong sometimes. So does human writers.

How do you know they want a correct summary? AI slop is good enough, acceptable for many people.

What is the use of such a summary?

Determining whether something is worth reading doesn't require a good summary, just one that contains enough relevant snippets to give a decent indication.

The opportunity cost of "missing out" on reading a page you're unsure enough about to want a summary of is not likely to be high, and similarly it doesn't matter much if you end up reading a few paragraphs before you realise you were misled.

There are very few tasks where we absolutely must have accurate information all the time.

I've done some summarizing with my own small Tcl/Tk-based frontend that uses llama.cpp to call Mistral Small (i.e. all is done locally) and i do know that it can be off about various things.

However 99% of the times i use this isn't because i need an accurate summary but because i come across some overly long article that i do not even know if i'm interested in reading, so i have Mistral Small generate a summary to give me a ballpark of what the article is even about and then judge if i want to spend the time reading the full thing or not.

For that use case i do not care if the summary is correct, just if it is in the ballpark of what the article is all about (from the few articles i did ended up reading, the summary was in the ballpark well enough to make me think it does a good enough work). However even if it is incorrect, the worst that can happen is that i end up not reading some article i might find interesting - but that'd be what i'd do without the summary anyway since because i need to run my Tcl/Tk script, select the appropriate prompt (i have a few saved ones), copy/paste the text and then wait for the thing to run and finish, i only use it for articles i'm in already biased against reading.

It's a good question. I'm not the OP, but I'd like to add something to this discussion.

How do I know what I'd be reading is correct?

To your question: for the most part, I've found summaries to be mostly correct enough. The summaries are useful for deciding if I want to dig into this further (which means actually reading the full article). Is there danger in that method? Sure. But no more danger than the original article. And FAR less danger than just assuming I know what the article says from a headline.

So, how do you know its summaries are correct? They are correct enough for the purpose they serve.

You can make a better decision if you have the context of the actual thing you are reading, both in terms of how it's presented (the non-textual aspects of a webpage for instance) and the language used. You can get a sense of who the intended audience might be, what their biases might be, how accurate this might be, etc. By using a summarizing tool all that is lost, you give up using your own faculties to understand and judge, and instead you put your trust in a third party which uses its own language, has its own biases, etc.

Of course, as more and more pieces of writing out there become slop, does any of this matter?

What are you constantly summarizing?

Articles. Some articles I fully read, some others I just read the headline, and some others I want to spend 2 minutes reading the summary to know whether I want to read the full thing.

Reply on news.ycombinator.com