Hacker News Reader

Comment by OtherShrezzing | Hacker News Reader

We’ve seen this in a few open source repos we voluntarily manage security on. They’re not massive repos, but big enough they get attention from security researchers.

Most of the real low hanging fruit was picked up by humans years ago. When doing automated scanning, the majority of stuff is overly-verbose nonsense which takes hours of expert human labour to understand, test, and discard.

Reading through a Claude generated false positive is absolutely excruciating, because it is absolutely determined that what it’s found is justified. Often you’ll receive very long accompanying “proof of concept” code which demonstrates absolutely wild scenarios. It’s especially frustrating when you’re volunteering your time for a project, and a well-meaning contributor submits the report without the technical nous to understand why you’re rejecting it.

View on HN

Before:

There is a vulnerability in library X when you call Y with these specially crafted parameters; as seen in the attached logs it can overflow buffer Z and clobber memory potentially leading to an RCE.

After:

Honest take: the load-bearing constraint violation is real. The log documenting the exploit gate is the ledger which weaves the story.

Blast radius!

Now you see the whole picture!

Right now, all top tier LLMs are as eager, bright 20ish year old interns.

Very gung ho, full of energy, loads of book learning, no real world experience or understanding of why things are as they are.

Leave them to their own devices at your peril. Trust nothing they do.

Yet directly guide them, monitor everything they do, some value emerges.

Have you used a frontier model since 2025? You are underplaying their strength.

Okay so now they're like a top percentile fresh grad on meth. Still a lack of real world experience plus some bizarre failures that illustrate gaping holes in the mental model. Does that description work for you?

Well, so basically supercharged fresh grad. CS/math implied.

In human terms, that's already at least a standard deviation above average person.

Kroah-Hartmann has used the closed frontier++ model, and it made up 37 out of 76 vulnerabilities.

And the bugs that actually were real rely on a setup so contrived it’s unlikely anyone in the world is impacted.

It’s still good that some real bugs are being patched but what is being reported to the media is so overblown.

We've been seeing "But you're not using the latest model!" over and over again, with every new model supposedly "groundbreaking" and "a game-changer" - just for the general population to conclude a few months later that it once again doesn't live up to the crazy marketing hype.

Anthropic claimed that Mythos was so good at finding vulnerabilities that it was too dangerous to release to the public. As this post clearly shows: that (only again) simply isn't true. If you believe your favorite flavor of frontier model is the exception, it is up to you to provide proof to back up that claim.

I sure hope they stop training them to be spaghetti throwers, please stop and ask questions when there is ambiguity

Well let's see. You sell to management. Does management buy:

1) nuanced tools that talk back, question assumptions, take over decisions, ... oh and expose just how much management knows about the business. Or how little)

2) a tool that can provide the excuse "we've had our source checked and dealt with the remarks"

We all know the answer.

I wish people stopped equating LLM to interns or junior engineers. They are tools, and as good they may be at some specific things people do, they also really suck at many more which we wouldn’t find acceptable in humans.

It's merely a way to frame things, in terms of experience and trust. And it highlights how an LLM can code very well, but not truly understand the ramifications of that code.

An "eager, bright 20ish year old interns" will grow as you guide them, while LLM will not, they'll only grow when their owner (definitely not us) update them. So, trying to humanizing some tool with human emotion is wrong way to framing it. Treat tool as tool.

Indeed they are tools. But if people/companies treat them as tools that can take a problem a human used to solve, and make them solve it from start to end, then the comparison begins to be necessary.

I wonder if the same was once said about calculating machines and calculators.

> Right now, all top tier LLMs are as eager, bright 20ish year old interns. > Yet directly guide them, monitor everything they do, some value emerges.

That's simply not true. I've had some very talented interns, and they are leagues ahead of what the LLMs can do. Not that it matters though, because the point of having interns wasn't to have them produce value. What made the investment worth it was that 12 months down the line I would have a competent colleague that I could have an interesting conversation with. A human person that could challenge some of my blind spots. A person that could take responsibility of something. Maybe not my most important work, but some of it. You don't get ANY of that from the LLM.

In some ways the reality is worse: the same version of an LLM won't get better at its job, even though you might get better at prompting it. Newer versions are trained on more code, which has obvious benefits, and their harnesses are better at taking advantage of tools that were created to keep human coders out of trouble.

The other side of the coin is that coding agents are not maximally productive unless you give them enough rope to potentially hang themselves. Over roughly the past year, the coding agents I use have gone from hot garbage to pretty consistently useful, especially if I find tasks where I can give them a lot of running room. On the other hand, last week I found a case where the coding agent was looping and flailing like it was doing every third try a year ago.

They fail less often, but they fail in the same way.

I think the way to handle this is to just feed it into another AI agent (a better one) and ask it how serious the issue actually is. Fight fire with fire!

That's somehow what https://sashiko.dev is.

Reply on news.ycombinator.com