Hacker News Reader

Comment by dexwiz | Hacker News Reader

Does anyone else hate reading AI summaries of code? Code can be pithy, but at least its terse compared to prose. When you add how verbose LLMs can be, I often end up reading a paragraph to explain a few lines. Or the opposite happens where the summary skips important edge cases or criteria. "You're right, X also does Y. I missed that in my initial analysis," is much too common of a phrase.

I like the idea of using LLMs to transform code into something more readable, and vice versa. I am not sure if meandering paragraphs and linear lists are the best targets.

View on HN

I wonder if a productive avenue might be "doing aspect-oriented programming in reverse": asking the LLM not to "summarize" the code per se, but rather to "clarify" it by transforming it into what a programming blog post would call a "toy example" of what the code is doing, by stripping out all the (non-semantic) error-handling, logging, metrics incrementing, etc — all the things that you might treat as their own "aspects" under AOP.

So..pseudo code?

AI summaries are great for learning which parts of the codebase are load bearing.

One thing worth flagging: summaries can provide context that is important, but you should always remember to thoroughly comb through the information in case there are caveats in the text that may act as a heads up, which is important.

And exactly where the seams are

And what was the smoking gun

Now I have a full picture

I've found that I don't mind getting a lot of text back from an LLM, when I was the one who prompted it. I can easily enough let my eyes flit around in the text and figure out what I need to, and I expect that putting all that text in the context window will help with the rest of the conversation.

It's when that text gets copied and pasted into a blog, or a PR, etc. that it really galls.

Yeah. Human psychology is weird.

I don't like the default explanations. But I prompt for small code snippets with explanations of a problem and the solution. This is in the context of extending features, fixing bugs, reviewing new code etc. I still skim the code, but it's nice to have a somewhat thoughtful overview of the key points like database schema, API spec, algorithm or abstraction. It makes it easier to skim a large diff without feeling lost. It also quite often catches some weird choices that might slip through it not carefully reasoning about the code. For me it is hard to understand code I didn't write myself so I have landed on this workflow.

The "language model" aspect shows IMO, at least for someone who grew up with the material of 90s and 2000s where we describe things more in mechanical, engineering terms. When gpt4 landed it was somehow amazing to see the output, but yeah nowadays I keep skimming through the explanation. It's like an intermediate dev who has nothing else to do but create long sentences to describe what could be simpler.

Right, the value is not so clear of having multiple people wade through lengthy LLM responses. Part of the value that a human in the loop adds is figuring out what's correct and important in the response and passing on a more polished result.

There is a reason Caveman (https://github.com/JuliusBrussee/caveman) is as popular as it is

I thought Caveman was debunked in the token saving argument? In real sessions it doesn’t really save much longterm and made sessions more forgetful/take longer.

I wish LLM summary of code was put aside the actual code, like the Jasmine documentation[0].

[0]: https://jasmine.github.io/tutorials/your_first_suite

I'm a bit confused by your use of 'pithy, but at least its terse'. Don't pithy and terse mostly mean similar things? Here it seems like you're contrasting them. Not trying to be pedantic, apologies if it comes off that way.

"Bob can be blunt but at least he's honest." Words can have positive or negative connotations for the same notion.

That's insightful! In a negative way.

Thats a good point, however, you wouldn't say "Bob can be honest, but at least he's blunt" which is I think more faithful to the initial quote. But point taken.

I think he’s saying that’s it’s dense and _too_ full of meaning. So, while terse, it can also hide extra meaning in its density.

Gnomic, then (on my mind because I saw it used yesterday, thanks flir).

I use AI explainers a lot in my own projects and have skills that give outputs I like. I'll even pipe the output into NotebookLM to make a video sometimes. This is actually a decent way to grok complex subsystems (YMMV).

What I don't care for is the pretty-yet-verbose HTML outputs that I see from some teams. They can be way too long and it's often the case that the engineer who made it didn't read the whole thing.

Those HTML reports are almost always filled with fluff.

As someone who likes a nicely designed report document, slapping a coat of paint on slop output isn’t helpful

Whenever I read any AI-generated documentation of any kind, it feels like my brain immediately blanks and I can't comprehend what I'm reading. It's like my brain blocks out the particular combo of word AIs love to use and 0 information manages to make it in there.

I think it's because it's basically always a high-level, rough overview of a topic that isn't quite correct. It's using 10x the words necessary to say nothing at all. It is a large, featureless wall of words that ultimately tells you absolutely nothing. It's simultaneously too verbose while also not actually saying anything useful.

Maybe my attention span has just gotten worse or something, but I don't really get the same sensation reading good quality docs that I know were human-generated so it must be something specific to how AIs form their paragraphs of noise.

We need a new specification language for summaries. I think just a (pseudo?-)programming language’s API (modules, interfaces, method signatures), but more concise and descriptive than Java, maybe Haskell or OCaml minus quirks (…the academics were this close to an ideal syntax, maybe they should’ve gave up those languages’ LR parsers…)

Your AI summary is raw output, like uncooked mince meat.

Ask for a rewriting pass! Get it to edit for word count, remove redundancies, remove run-on sentences, and change sentence structure to use active voice.

You wouldn't ship your first draft - why let AI do it?

I mean... the problem with that, as with all the "you have to go back and forth with the AI" scenarios, is that by the time I finish with that I may as well have just written it myself to begin with. Would take a similar amount of time, actually have my voice in it, and not cost money for tokens.

If all you need is a readable summary you can understand you should definitely try asking. You might get a surprise!

Worst case you lose 10 seconds of your life. Not too bad!

I agree, I also sometimes do stuff manually when it's easier to do manually than to ask AI. Took me years before I got it though.

Yeah, I have a rule in my agents MD that essentially says the code doesn't need a comment because it should be readable. And ask me in the exceptionally rare instances where it needs a summary or a comment. I am a Ruby and JavaScript developer, so I suppose that is a lot more readable than some other languages, but broadly, I've found code summaries and code comments to not be necessary if the code is very elegantly written.

It’s an LLM, it does what you want. If you don’t like it verbose tell it you want it to be succinct and it’ll do exactly what you ask of it. What sort of complaint is this?

Ask for a walkthrough of the highlights (with '...' omissions so that the most important parts are quickly visible) instead of a verbal summary.

One thing i hate more is the tendency of people now to rely on such summaries over the code and copy-pasting 400 line summaries of something fundamentally wrong.

Code is more precise than english language. If it wasn't everyone would just use english to write programs.

Yes this is why I ask the agent to summarize the code as a comic of a lizard explaining it to a cat.

I like to ask for a code tour using snippets of the most important types and functions. Much better to read through

I use caveman to handle this. Not because to I want to spent less tokens, I want to spent less time reading.

I hate fighting with the AI to get a good summary. Reading the good summary is great. Getting to it is the challenge.

First attempt: mountains of meta conversation and not answering the original question.

Second attempt: shorter and more concise, cuts out some of the nonsense, but terribly written.

Third attempt: goes off the rails, misunderstands what you are asking and tries pushing the work onto you.

Fourth attempt: finally something that's concise, reasonably well written, to the point and passable.

What agent are you using? I have much better experiences fortunately

You're absolutely right!

Reply on news.ycombinator.com