Hacker News Reader

Comment by afro88 | Hacker News Reader

Another cherry-picked example of an LLM doing something amazing, written about with a heavy dose of anthropomorphism.

It's easy to get LLMs to do seemingly amazing things. It's incredibly hard to build something where it does this amazing thing consistently and accurately for all reasonable inputs.

> Analyzing authentication system files:

> - Core token validation logic

> - Session management

> - Related middleware

This hard coded string is doing some very heavy lifting. This isn't anything special until this string is also generated accurately and consistently for any reasonable PR.

OP if you are reading, the first thing you should do is get a variety of codebases with a variety of real world PRs and set up some evals. This isn't special until evals show it producing consistent results.

View on HN

I also think there's some exaggeration. Annotating files with a feature tag system is both manual and not scabale. Custom prompting for each commit or feature a lot more so. You do a decent bit of specialized work here.

And I think he left out the most important part, was the answer actually right? The real value of any good dev at all is that he can provide reasonably accurate analysis with logic and examples. "Could have an error" is more like a compiler warning than the output of a good engineer.

Side note: "broke the benchmark script?" If you have an automated way to qualitatively evaluate the output of an LLM in a reasonably broad context like code reading, that's far bigger a story.

>Annotating files with a feature tag system is both manual and not scabale.

Wouldn't you have the AI annotate it?

Isn't the entire point of the post the fact that the AI is missing this information in the first place?

It's incredibly hard to get __ HUMANS __ to do this amazing thing consistently and accurately for all reasonable inputs.

Some humans can do it consistently, other humans can't.

Versus how no publicly-available AI can do it consistently (yet). Although it seems like a matter of time at this point, and then work as we know it changes dramatically.

To be fair, most senior developers don't have any incentive to put this amount of analysis into a working codebase. When the system is working, nobody really wants to spend time they could be working on something interesting trying to find bugs in old code. Plus there's the social consideration that your colleagues might not like you a lot if you spend all your time picking their (working) code apart while not doing any of your tasks. Usually this kind of analysis would come from someone specifically brought in to find issues, like an auditor or a pen-tester.

The right incentives would motivate bug hunting, it entirely depends on company management. Most competent senior devs I’ve worked with spend a great deal of time carefully reading through PRs that involve critical changes. In either case, the question is not whether humans tend to act a certain way, but whether they are capable of skillfully performing a task.

Unfortunately some senior devs like myself do care. Too bad no one else does. Code reviews become quick after a while, you brain adapts to being to review code deeply and quickly

Humans are fully capable of protracted, triple-checked scrutiny if the incentives align just right. Given the same conditions, you cannot ever compel an AI to stop being wrong or consistently communicate what it doesn't understand.

That's exactly what I want to do next.

Any tips on how should I get codebases and real world PRs? Are the ones on popular open source repos on GitHub sufficient? I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias.

Secondly, after getting all this, how do I evaluate which method gave better results? Should it be done by a human, or should I just plug an LLM to check?

“…should it be done by a human?”

Sigh.

I'll do it personally in the beginning but was thinking about doing it on scale

> doing it on scale

Like cloud-scale, no-code scale, or NoSQL scale? You are confused, which shows that, maybe, you should not be using such tools with the experience that you don't have.

‘Like cloud-scale, no-code scale or NoSQL scale’.

That is the dumbest statement I have heard this week. You should perhaps refrain from commenting, at-least until you gain the modicum of intelligence that you currently don’t have.

Very first thing you can tell us (or try if you haven't) is that if you re-prompt, does it give the same answer? Second can you get it to generate (consistently and repeatedly) the text that gp pointed out?

Don't need to switch to a different repo for quick test, just make it reproable on your current repo.

Not the exact text, but still decent quality. I'll play around with temperature and prompts a bit.

If you could get an LLM to check, you could just spam solutions with any assortment of models and then use your checker to pick the best.

Except LLMs are usually really bad at checking.

Perhaps if they're some day augmented by formal methods, that might change.

Sorry, I was unclear- my statement was supposed to be proof that LLMs are bad at checking, because the approach I listed is well known to not work

Got it, now it makes sense :)

I may accidentally have been inspired from your message when I wrote the following piece, yesterday: https://yoric.github.io/post/formal-ai/

> I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias.

As opposed to what, yet another beginner React app? That’s what everyone seems to be testing with but none of the projects I’ve seen are reflective of a production codebase that’s years old and has been touched by a dozen developers.

Throw it at a complicated non-frontend mixed language repo like cxx-qt [1] or something, preferably where the training data doesn’t include the latest API.

[1] https://github.com/KDAB/cxx-qt

"preferably where the training data doesn’t include the latest API"

That is the reason LLM's in their current shape are pretty useless to me for most tasks.

They happily mix different versions of popular frameworks, so I have to do so much manual work to fix it, I rather do all by myself then.

Pure (common) math problems, or other domains where the tech did not change so much, like bash scripts or regex are where I can use them. But my actual code? Not really. The LLM would need to be trained only on the API version I use and that is not a thing yet, as far as I am aware.

Have you tried RAG on the docs?

Not yet, I can imagine it can improve things, but was too sceptical that it is worth the effort. Have you had good succes with it?

https://www.kaggle.com/competitions/konwinski-prize

Given how eager Microsoft is to steal other people's code, perhaps the leaked Windows source code would be an option. Or perhaps Microsoft will let you train on their internal issue tracker.

Still, the findings in the article are very valuable. The fact that directing the "thought" process of the LLM by this kind of prompting, yields much better results, is useful.

The comparison to how a senior dev would approach the assignment, as a metaphor explaining the mechanism, makes perfect sense to me.

What I want to know is how accurate was the comment? I've found AI to frequently suggest plausible changes. Like they use enough info and context to look like excellent suggestions on the surface but you realize with some digging it was so completely wrong.

This is exactly the behaviour that I would expect from a human teleported into a project and given a similar task.

After some time humans would gather some background info needed to be more productive and we need to find out how to copy that.

You're saying a human would also offer bullsh*t suggestions that seem right but are in fact wrong?

Humans who make lots of mistakes with confidence that they aren't mistakes usually get fired or steered into a position where they can do the least amount of damage.

It's not that AI needs more background info for this type of of thing. It needs the ability to iteratively check it's own work and make corrections. This is what humans do better.

Ok, maybe that was exaggerated - there are differences, but they are shrinking.

What i'm learning is just because something might be hard to you or I, doesn't mean it's not possible or not working.

LLMs can generlaly only do what they have data on, either in training, or instructions via prompting it seems.

Keeping instructions reliable, is increasing and testing, appears to benefit from LLMops tools like Agenta, etc.

It seems to me like LLMs are reasonably well suited for things that code can't do easily as well. You can find models on Hugging face that are great at categorizing and applying labels and categorization, instead of trying to get a generalized assistant model to do it.

I'm more and more looking at tools like OpenRouter to allow doing each step with the model that does it best, almost functionally where needed to increase stability.

For now, it seems to be one way to improve reliability dramatically, happy to learn about what others are finding too.

It seems like a pretty nascent area still where existing tooling in other areas of tech is still figuring itself out in the LLM space.

The people who claim it's that hard to do these things have never heard of or used constrained/structured generation, and it shows big time!

Most other related issues of models these days are due to the tokenizer or poor choice of sampler settings which is a cheap shot on models.

No need for the pessimism, these are new tools that humans have invented. We are groking how to utilize them.

OP brought a rational argument, you didn't. It sounds like you are defending your optimism with emotion.

> We are groking how to utilize them.

Indeed.

It’s tautologically the case that these are new tools under constant improvement.

The fact that these tools have extremely weird and new to the world interfacial quirks is what the discussion is about…

This post talks as if the results are a worthless pile of trash while obeying the HN rules of not directly insulting the results. I agree with everything under the first paragraph.

Let me spell it out for you. These results. Are. Not. Worthless.

Certainly what you said is correct on what he “should” do to get additional data, but your tonality of implying that the results are utter trash and falsely anthropomorphizing something is wrong.

Why is it wrong? Imagine Einstein got most things wrong in his life. Most things but he did discover special and general relativity. It’s just everything else was wrong. Relativity is still worth something. The results are still worthwhile.

We have an example of an LLM hallucinating. Then we have another example of additional contextual data causing the LLM to stop hallucinating. This is a data point leaving a clue about hallucinations and stopping hallucinations. It’s imperfect but a valuable clue.

My guess is that there’s a million causal factors that cause an LLM to hallucinate and he’s found one.

If he does what he did a multitude of times for different topics and different problems where contextual data stops an hallucination, with enough data and categorization of said data we may be able to output statistical data and have insight into what’s going on from a statistical perspective. This is just like how we analyze other things that produce fuzzy data like humans.

Oh no! Am I anthropomorphizing again?? Does that action make everything I said wrong? No, it doesn’t. Humans produce correct data when given context. It is reasonable to assume in many cases LLMs will do the same. I wrote this post because I agree with everything you said but not your tone which implies that what OP did is utterly trivial.

They didn't say worthless, they said amazing.

Their comment is "do it consistently, then I'll buy your explanation"

lol “seemingly amazing” means not amazing at all.

He didn’t literally say it but the comment implies it is worthless as does yours.

Humans dont “buy it” when they think something is worthless. The tonality is bent this way.

He could have said, “this is amazingly useful data but we need more” but of course it doesn’t read like this at all thanks to the first paragraph. Let’s not hallucinate it into something it’s not with wordplay. The comment is highly negative.

You seem very emotionally involved in this. It says "an LLM doing something amazing". That's the sentence. Later the term "seemingly amazing" is used. Implying that it _seems amazing_. Anything beyond that is your personal interpretation. Do you disagree that there is an excess of cherrypicked LLM examples getting anthropomorphized? Yeah, it did a cool thing. Yes, llms doing single cool things are everywhere. Yes, I well be more convinced of its impact when i see it tested more widely.

Reply on news.ycombinator.com