Hacker News Reader

Comment by simonw | Hacker News Reader

I got gpt-4.5-preview to summarize this discussion thread so far (at 324 comments):

  hn-summary.sh 43197872 -m gpt-4.5-preview

Using this script: https://til.simonwillison.net/llms/claude-hacker-news-themes...

Here's the result: https://gist.github.com/simonw/5e9f5e94ac8840f698c280293d399...

It took 25797 input tokens and 1225 input tokens, for a total cost (calculated using https://tools.simonwillison.net/llm-prices ) of $2.11! It took 154 seconds to generate.

View on HN

interesting summary but it's hard to gauge whether this is better/worse than just piping the contents into a much cheaper model.

It’d be great if someone would do that with the same data and prompt to other models.

I did like the formatting and attributions but didn’t necessarily want attributions like that for every section. I’m also not sure if it’s fully matching what I’m seeing in the thread but maybe the data I’m seeing is just newer.

Good call. Here's the same exact prompt run against:

GPT-4o: https://gist.github.com/simonw/592d651ec61daec66435a6f718c06...

GPT-4o Mini: https://gist.github.com/simonw/cc760217623769f0d7e4687332bce...

Claude 3.7 Sonnet: https://gist.github.com/simonw/6f11e1974e4d613258b3237380e0e...

Claude 3.5 Haiku: https://gist.github.com/simonw/c178f02c97961e225eb615d4b9a1d...

Gemini 2.0 Flash: https://gist.github.com/simonw/0c6f071d9ad1cea493de4e5e7a098...

Gemini 2.0 Flash Lite: https://gist.github.com/simonw/8a71396a4a219d8281e294b61a9d6...

Gemini 2.0 Pro (gemini-2.0-pro-exp-02-05): https://gist.github.com/simonw/112e3f4660a1a410151e86ec677e3...

Thanks for sharing. To me, purely on personal preference, the Gemini models did best on this task, which also fits with my personal experience using Googles models to summarize extensive, highly specialized text. Geminis 2.0 models do especially well on Needle in Haystack type tests in my experience.

At a glance, none of these appear to be meaningfully worse than GPT-4.5

Seeing the other models, I actually come away impressed with how well GPT-4.5 is organizing the information and how well it reads. I find it a lot easier to quickly parse. It's more human-like.

I noticed 4o mini didn't follow the directions to quote users. My favourite part of the 4.5 summary was how it quoted Antirez. 4o mini brought out the same quote, but failed to attribute it as instructed.

It's fascinating, but while this does mean it strays from the given example, I actually feel the result is a better summary. The 4.5 version is so long you might just read the whole thread yourself.

I actually think the Claude 3.7 Sonnet summary is better.

yeah I liked it too, especially for 10x less the price lol

Interesting, thanks for doing this. I'd say that (at a glance) for now it's still worth to use more passes with smaller models than one pass with 4.5

Now, if you'd want to generate training data, I could see wanting to have the best answers possible, where even slight nuances would matter. 4.5 seems to adhere to instructions much better than the others. You might get the same result w/ generating n samples and "reflect" on them with a mixture of models, but then again you might not. Going through thousands of generations manually is also costly.

Compared to GPT-4.5 I prefer the GPT-4o version because it is less wordy. It summarizes and gives the gist of the conversation rather than reproducing it along with commentary.

Didn't seem to realize that "Still more coherent than the OpenAI lineup" wouldn't make sense out of context. (The actual comment quoted there is responding to someone who says they'd name their models Foo, Bar, Baz.)

Wonder if there’s some pro-OpenAI system prompt getting in the way of that.

It'd be a silly move considering how fast system prompts leak.

Seems to have trouble recognizing sarcasm:

"For example, there are now a bunch of vendors that sell 'respond to RFP' AI products... paying 30x for marginally better performance makes perfect sense." — hn_throwaway_99 (an uncommon opinion supporting possible niche high-cost uses).

? You think hn_throwaway_99's comment is sarcastic? It makes perfect sense to me read "straight."

That is, sales orgs save a bunch of money using AI to respond to RFPs; they would still save a bunch of money using a more expensive AI, and any marginal improvement in sales closed would pay for it.

It maybe excessively summarized his comment which confused you-- but this is the kind of mistake human curators of quotes make, too.

I don't think they are being sarcastic. Maybe you are the bot /s

I don't know why but something about this section made me chuckle

""" These perspectives highlight that there remains nuance—even appreciation—of explorative model advancement not solely focused on immediate commercial viability """

Feels like the model is seeking validation

As expected, comments on LLM threads are overwhelmingly negative.

Personally, I still feel excited to see boundaries being pushed, however incremental our anecdotal opinions make them seem.

I disagree with most of the knee-jerk negativity in LLM threads, but in this case it mostly seems warranted. There are no "boundaries being pushed" here, this is just a desperate release from a company that finds itself losing more and more mindshare to other models and companies.

Huh. Disregarding the 4.5-specific bit here, a browser extension or possibly website that did this in general could be really useful.

Maybe even something that just noticed whenever you visited a site that had had significant HN discussion in the past, then let you trigger a summary.

My site https://hackyournews.com does this!

Been keeping it alive and free for 18 months.

Wow I find this very useful, thanks! Bookmarked.

there are literally hundreds of extensions and sites that do this

the problem is that they are competing each other into the ground hence they go unmaintained very quickly

getrecall.ai has been the most mature so far

Hey, check this one out with all the different flavors that existed out there. I think I made something better. https://cofyt.app

As far as I am aware, feel free to test it head-to-head. This is better than gecall, and you can chat with a transcript for detailed answers to your prompts

I tried it out, looks nice and clean.

But as I mentioned, my main concern is what will happen in 6 months when you fail to get traction and abandon it. Because that's what happened to previous 5 products I tried which were all "good enough" .

Getrecall seems to have a big enough user base that will actually stick around.

Thank you for getting back to me.

I understand your perfectly reasonable argument to make from your position (user).

First let me tell you that I saw a lot of things out there including getrecall before starting building this and felt there was nothing out there that had a good UX/UI that actually makes it a enjoyable product (nice and clean).

I’m confident in the direction and committed to seeing it through by building something better for me and maybe for you to by doing it with more care.

Appreciate your feedback and while no one can control the future I´ve added this thread to my calendar do come back here in 6months.

Hundreds that specifically focus on noticing a page you’re currently viewing has been not only posted to but undergone significant discussion on HN, and then providing a summary of those conversations?

Or that just provide summaries in general?

Thanks, it's amazing how much stuff is out there I don't know about.

What I want is something that can read the thread out loud to me, using a different voice per user, so I can listen to a busy discussion thread like I would listen to a podcast.

The headline and section: "Dystopian and Social Concerns about AI Features" are interesting. It's roughly true... but somehow that broad statement seems minimize the point discussed.

I'd headline that thread as "Concerns about output tone". There were comments about dystopian implications of tone, marketing implications of tone and implementation issues of tone.

Of course, that I can comment about the fine-points of an AI summary shows it's made progress. But there's a lot riding on how much progress these things can make and what sort. So it's still worth looking at.

Maybe it's just confirmation bias but the language in your result output seems higher quality that previous models. Seems more natural and eloquent.

$2.11! At this point, I'm more concerned about AI price than egg price.

you know what? it would be damn nice to do this to literally every post in HN and give people a summary so that they dont have to read 500 comments

Reply on news.ycombinator.com