SQLite Critical CVEs or LLM Slop?
research.jfrog.com
research.jfrog.com
LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your LLM-based “AI” gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence. What bothers me about this entire situation is that presumably the folks that relied on the LLM-based “AI” to generate these vulnerabilities knew (or should have known) enough about their tool to know this would happen, but did not.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.
People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.
Oh, design too. I’ve met many engineers who simply don’t see any value in design.
Start a discussion about any topic, look at how many nerds will tell you that you are wrong.
Obviously.
Can they do the job? Because right now, government decisions are based on AI generated code, which was verified by nobody who can do that. So the cost of an unsatisfactory answer is quite high.
I like the symmetry of people being unable to detect "skill" in people who are great at manipulating language and in LLMs who are also great at manipulating language.
B...b...but the Anthropic trainer said we'd get the best results if we don't think of it as a tool, but instead give it a name and think of it as our brilliant coworker!
Why should I trust you, internet rando over a stormtrooper-level salesman? /s
The sort of person that's going to offload their thinking to AI is the exact sort of person that is not going to verify anything because they've already offloaded their thinking to AI.
So now the problem is highly visible. There's only one solution I can think of, and that's to hold individuals (not corporations) responsible for professional malfeasance.
Personally I think a corporate death penalty strikes a nice balance of holding companies and individuals responsible. If a company exercises sufficient malfeasance, it gets nationalized with no compensation for shareholders. Watch the incentives align real fast.
Brandolini's principle in action. It takes 10 times more energy to refute BS than to generate it. A related analogy to computing: it is easy to generate propositions, but hard to test if a given proposition is satisfiable or not, which curiously ties to P vs NP.
I much prefer the alternative name: the Bullshit Asymmetry Principle.
This line of critique is pernicious because it is both technically correct, as description, and profoundly misleading.
Saying that outputs are a product of inputs is not interesting and to the point it is not explanatory.
What is interesting, is how they do what they do. What is the "statistically likely* next token? To answer that you can do exactly one thing, run the LLM. That's because what they are doing is interesting and not reducible.
What is more interesting is that in order to do what they do, given the architectures we apply and the training strategies we use and the harnesses we situate them in, LLM are recapitulating in their deep layers strategies observed in the animal brain. This is still suggestive, interpretibility is nascent: but it is also more than a little interesting. In some respects, for cognitive scientists interested in the manner in which mind merges from computational substrates, it is profoundly interesting.
One can incorporate this, and, still be viciously critical of bother the success and failure of LLM in the applications we have put them to, and of how we (as individuals and as institutions such as corporations) are integrating them into our work.
There is a lot to criticize!
But criticism can be taken more seriously when it is not obscured by misunderstanding or misrepresentation (intentional, or not) of what LLM are and why they are not remotely "parrots" in the pejorative sense.
The technology, as technology, at the scale we are architecting it, is doing things we did not imagine would be witnessed in our lifetime, if ever. Dismissing that and denying it because of the career, industry, society, and civilization challenges that technology brings are existential, is bad argumentation or bad faith.
Both can be true at once.
Watching for unexpected failure modes is surely worth it.
Intelligent life-forms can generate probabilistic outputs based on inputs, but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.
So Turing-completeness is not even a necessary condition for intelligence.
Likewise, there is no reason to think the brain employs super-Turing or quantum computations that cannot be approximated by LLMs.
They could have been great, if trained on datasets from a more sensible species.
???
Of course it is. The brain is mechanically not capable of doing anything other than that.
Do you believe the brain is something other than a bundle of probabilistic physical interactions? Or are brains not the source of what we call intelligence?
This is going to elevate your thinking on this no end, if you're interested.
We know that the brain is a probabilistic input → output machine because the universe is a probabilistic input → output machine. The brain is made of universe. There are deterministic relationships (which at high sensitivity or complexity become easier to describe as probabilistic), and quantum relationships. That's it. The brain, like every other thing comprised of "universe" is comprised of those two types of relationships.
If Romain's book provides evidence of relationships in the brain that are neither quantum (therefore random) NOR classical (therefore deterministic), then 1) he would have already won at least one Nobel prize, and 2) anyone in this thread would be able to at least gesture toward what relationship that is.
Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?
The answer is obviously yes lol.
The creature is an assemblage of electrical, chemical, and kinetic relationships.
Watching a baby develop is exactly what you'd expect from a system that's predominantly electrical noise triggering behaviors and then gradually refining denoising the relationship between inputs and outputs, with the goal function of achieving more desirable inputs.
Surely you can at least gesture toward one thing in the brain that appears not to be a probabilistic relationship between input and output?
The brain is deterministic at the level of specific interactions, which process inputs in a highly chaotic (but still deterministic) manner into probabilistic outcomes.
The opposite of deterministic is random, i.e. in the quantum sense of truly no relationship between input and output.
There are probably some quantum effects in the brain here and there, but the vast majority of it is just traditional deterministic interactions networked together in such a complex system that the resulting behavior is much, much easier to predict in probabilistic terms than otherwise.
I agree that humans must verify LLM-produced facts, but strongly disagree with these kinds of "stochastic parrot therefore dumb" arguments.
Yes, an LLM is a "stochastic parrot". No, that doesn't imply that it is dumb. Enough to look at how Terence Tao asks ChatGPT to help him understand a solution that nobody had ever discussed before [1], or how a random guy asks ChatGPT in a handful of words to disprove a 30-year-old conjecture, with zero technical input [2].
If your parrot in a birdcage with internet access can finish the sentence, "The counterexample to the Dinitz–Garg–Goemans conjecture is...", then it's a pretty smart parrot, by all reasonable definitions of "smart". Just because someone bottled up the formula into matrix multiplications and added some random sampling to the outcome, that doesn't take away from the fact that the parrot said provably correct statements that the biggest experts in the field couldn't imagine.
And no, I'm not implying that the LLMs are correct all the time, or that their intelligence and reasoning works in any way like ours.
[1]: https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed... [2]: https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
But the LLM is still dumb where those skills doesn't have good coverage, since unlike the expert systems it maps fuzzily to its skills, and they are tuned to produce results over rejecting the request when its unclear if coverage is good. As long as that is true you have to treat them as dumb even if they sometimes produce brilliant results.
I hate AI slop as much as the next guy but the amount of tribalism over AI is taking near-religious forms.
Nobody knows what intelligence is, therefore we don't know what does or does not possess it, therefore we don't know whether LLMs currently, or in the future, possess it.
Yes, LLMs can be stupid, guess what: so can I. That doesn't really change the argument at all.
I feel like I'm on a deja-vu from when DALL-E was released and everybody was fighting over whether AI can be creative yes or no. Same story, different words.
Intelligence, creativity: we have no idea what these words mean, and AI is helping us understand them better. That alone is an achievement of epic proportions. I am not joking here. Any computer scientist before 2015 would be absolutely blown away by what you can now do for 10 cents and an API call, yet somehow because of the tech-bro-iness of it all we get a tribal war over what is plainly visible in front of us:
LLMs are uncomfortably close to what we thought intelligent machines would look like
Some predictions might be the tokens "I don't know", but that is based on the model mapping your text to those tokens by having seen many similar "I don't know" responses to such contexts, it didn't do any introspective logic to produce that "I don't know", and its possible it actually does know if it followed another branch there so "I don't know" is often not even true.
If they had an introspective part that stops the prediction when its too unreliable it would no longer just be token prediction engine, and I believe we need such a part for them to become what I call smart. I don't think LLM will ever stop being dumb without such an introspective part to them.
And no, that introspective part is not a part of the token predictor. At least not in us humans, the feeling of certainty we have is not a prediction, it is bundled with our thoughts, so we get both "answer is a bear" and "certainty is low", we don't get just one of those as a "prediction".
Will LLM become smart as humans with such an introspective part? I don't know, but I think they will never become as smart as humans without one.
Note: The certainty score has to be per conclusion or response, not per token. You can't evaluate a responses validity by aggregating the weight of each token. Meaning its a logic engine, not token engine, that evaluates the certainty of a statement being correct or not instead of a token being correct or not. That is the level human thinking works at and seems to be dramatically more efficient.
Maybe our ability to reason is not as mystical and special as our ego might hope it to be, and discomfort over LLM's bringing that to light is the root of some people's urge to continuously downplay and discredit them.
(As far as I can tell as a non-neuroscientist, the literature on how our default mode network and prefrontal cortex interact agrees with my assessment - free association and logical verification respectively)
That introspection isn't an illusion, what your consciousness see of your thoughts obviously are tings that has been calculated and aggregated by the brain, so we know the brain calculates and aggregates those thoughts and feelings to produce its results. And we know LLM doesn't do that, it doesn't have a side system that does that sort of introspection.
The very second they add this part, it will "just be a dumb token predictor with introspection", mark my words.
AGI is fairly easy to detect for this reason. Does this system make a majority feel you don't have to hire people anymore? If not its not AGI.
If I want my ox-cart to fly, I need to add wings, but that's probably not all that needs changing.
“LLM has made legitimate mathematical discoveries” —> Wow the rate of progress is amazing. Highly upvoted.
“LLM does something not good” -> Does everyone else not realize LLMs are just dumb next token predictors? Highly upvoted.
So tired of this discourse and this site.
These models can do a lot of things but they also can't do a lot of things. In order to use these models effectively you have to understand that they are next token predictors and how that allows it to do what they do.
For them to change the world you have to work with them as next token predictors. Ensure that the next token predictor has enough prediction paths to solve the problems you want and so on. Since when they don't they fail spectacularly. These big companies will continue to add new skills to them, so they will continue to get more useful.
Yes, but not because they are useful.
The issue is of course with using the word "dumb": they are next token predictors, no doubt about it, but whether LLms as a class of system are smart or dumb is entirely unknown and entirely variable in time.
To interact with them effectively you must know how they behave, just like you have to know how humans behave to interact with them effectively. If you disagree, find someone with autism and have a conversation with them.
So people call them dumb since like dumb people they make strong statements about things they don't understand. And it doesn't matter how much smart things you encode them with, they will keep making strong statements about things they don't understand until they are fundamentally changed.
But since LLM are very smart about things where they have extensive data they can still be used to reliable solve many problems and probably in the future where we understand that better almost completely replace most lawyer and doctors work etc, because a lot of what a frontline doctor or basis lawyer work is very repetitive and can be encoded with billions of examples and decision paths into an expert system framework the LLM will follow.
So people say LLM are dumb since LLM will always keep making dumb statements. This is the same way we call Elon Musk dumb for making a lot of dumb statements, he is a smart guy but he makes dumb statements so her is dumb.
And thats encoded as a set of next token predictions. So the way to see how reliably it solves a problem is to look at the chain of predictions, and see where it is unreliable at finding the next spot, or where it always fails and you need to add that link to the dataset to train it.
This isn't magic, today we understand pretty well how to add new skills to LLM, and the better this is understood the faster progress will be.
This also means that if a context doesn't have any good predictions, it will produce a dumb prediction for that context. This results in these bad outcomes, because currently LLM doesn't have a map for where predictions are good or bad.
I mean even perennially contentious topics will get this behavior.... some thing about emacs makes the front page, within a day or two there will be a vim post up there. Same with Rust is (good|bad), or if systemd creates an even more awesome tool, the haters will come along and recycle stories about bugs from over a decade ago.
There's a lot of people here. Not all of them read it every hour, and discussions like this among large groups often take a very long time with lots of repetition. Human group dynamics (aka politics) is slow.
> So tired of this discourse and this site.
You're welcome to leave if you don't like it. The site was like this long before you joined, and will like it long after you leave I'm sure.
It's also worth noting, that an awful lot of math discoveries are perfectly in line with dumb next token generators - they are finding a way to formally construct an argument and being surprised when it doesn't work, or surprised at the outcome of the grind. Not all of them are made by brilliant leaps of intuition.
Or should the discourse in a diverse community like HN only reflect the positions you personally hold?
My point is it's silly to whine that HN is a place where multiple points of view on the topic are aired out and discussed.
If you want a personal echo chamber where only your own beliefs are affirmed and anything else is flagged off or downvoted, I'm sure you can go find one or, worst case, vibe code one into existence.
And if someone’s first response to this comment is to try to peg which “side” I’m calling an idiot as if that’s the most important bit of info to dictate their response - that’s exactly what I’m talking about.
I see no credible corroboration. More likely its folks having no more care for what they are doing than the bots themselves.
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
People said the same about email spam ... until they engaged spam filters. CVE report slop is simply spam. Complaints are better directed at the filters, not the filtered.
I've thought about this for quite some time now.
No. A human doesn't need to verify everything. And the argument is really simple: stochastic.
Think of self-driving cars: We can show today - based on evidence and real data - that self-driving cars are safer than human drivers. That's a fact and the consequences are clear, more self-driving cars, less human-driven cars, less accidents, less hurt people, less dead people.
Are the cars 100% safe and NEVER make a mistake? No. But they don't need to. Nothing is ever 100% (in the real world).
Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.
Yes, there will be another layer in the system, another level of abstraction that will/must end at the human boundary.
We have plenty of systems where complete accuracy is the only acceptable thing. Computers are great for such things. Until we all get caught up in a way of delusion and start writing those systems as natural prose passed through an improperly understood stochastic machine.
An LLM building software benefits from using a formal language, formal tests or having a formal specification in a similar way we do. Everything you can formalize into something, so you have certainty about stuff, is always a benefit.
I wasn't comparing exact science with LLMs. I was comparing current messy review processes of code performed by humans with future messy review processes of code likely performed by LLMs.
If you find a way to convert code reviews into a fully formalized process, then this is clearly the winner. If the LLMs find a way to do that, same.
If you find a way to formalize the process of driving through any street in any situation, then this is clearly the winner. Until then, driving stays messy and stochastic.
Let me know once the majority of software engineering organizations start only checking in markdown files and let code be generated non-deterministically from these specs in CI. If this is not happening now, there’s clearly a sufficiently high level of distrust in blind LLM output (both code itself and reviews).
And even then, are you suggesting humans shouldn’t at least review the markdown specs? Why not have LLMs review the specs then? Is there, perhaps, some fundamental quality to human review process that is desirable here?
Flesh-based “brain” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your flesh-based brain gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
Humans are prediction engines. They are not Pure Intelligence, and shouldn’t not be treated in any form or fashion as if they possess pure intelligence. What bothers me about this entire situation is that presumably the folks that have relied on the flesh-based “brains” to generate these vulnerabilities knew (or should have known) enough about their "tool" to know this would happen, but did not: To err is to be human.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this over reliance on fallible “brains".
A human must verify everything another human presents as fact. Everything. If you don’t, we all pay the price. Using a human does not remove the onus of responsibility on the human being in charge, if anything they amplify it because humans work for peanuts in some countries, and can generate lots more output more quickly that needs to be verified by the humans in charge.
However, error margins are in the center of any engineering discipline. We cannot produce things measured with 100% accuracy. This is accepted fact. The focus is always not on eliminating errors, but on reducing them to acceptable minimum. With LLMs we should not expect an ideal logical thinker, but a process that may error sometimes, and we must design quality controls instead that push LLM outputs within acceptable margins. And it can work.
In the current AI mania, there's a lot of due diligence simply being ignored. Plenty of "Well humans make mistakes too!" going on here on HN too.
But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.
One such example is CVE-2023-45853 [1]. Zlib included in it's source an extra set of utilities and add-ons. One such utility, MiniZip, had a buffer overflow vulnerability. BAM, 8.8 CVE (was a 9, looks like they pulled it back a bit). But not one that the 99% of applications using zlib would ever be vulnerable to because almost nobody used the MiniZip utility. It was so unused that the solution for zlib was to simply remove it.
I know about this one particularly because our security policy required us to do a BUNCH of pointless updates for it since zlib is in just about everything.
deprecated("@:1.3 +minizip", reason="cve", severity="high")
Deprecation on versions only isn't the right granularity.
Even if some individual case can be shown to be safe from being combined, can we identify such cases with enough confidence to justify using it reduce severity warnings?
In testing with LLMs with good exploit finding capabilities and a lack of guardrails on writing exploits, quite often LLMs will chain together a surprising stack of exploits to get what they want. They'll get access via a weak, but limited user. They'll search around laterally until they find something else. For example User A (weak exploit) -> User B -> User B bad configuration -> root (or application takeover).
They don't get bored like humans and they can be more robust than a set of scripts by far in catching errors when their scripts don't work or interesting things show up in their findings. The huggingface break down is a good example of just how much lateral movement an LLM can try with enough processing power behind it.
This is where the security onion gets messy. It can be hard to predict when compromising one layer actually bypasses many other layers of your security.
I keep feeling like the entire CVE system is creaking and about to fail- the signal-to-noise on that has gone so far down, the gate-keeper function that MITRE et al are supposed to be providing clearly has fallen apart, but we do need some way of tracking problems and alerting on bugs in commonly reused libraries. So I can see the problems with the current system, I just can't envision the new, better system to build to replace it.
go lang has a scanner [0] which only shows issues when the code actually calls the effected functions.
My condolences to anyone who’s got to deal with all these slop-y CVEs on one side and brain dead security teams on the other.
If you keep driving the "seperate the packages" angle you end up with is-odd, and that not a nice play to be either.
Nope, someone should have requested that from you directly or indirectly, but you should not have done the pointless job if you have a pointy one usually.
If you obey, it degrades you and the society at large.
CVE should just be far more granular instead of flagging alert for anything using zlib
I suppose all these fake issues and the many more that have absurdly elevated severities could be considered an attack on the system itself, stripping it of credibility.
On top of that, ability to prompt about flaws makes all the difference in the world. If you know what you're doing you can get better results. Problem, very few people know what they are doing.
Going back to the money/time problem, LLMs are not deterministic. Much like the day you forgot to drink coffee, running LLM systems against code can get different results where things are found or not found. That or two different things are found on two different runs (with the same code) because attention was stolen by the first issue it found.
So yea, in the LLM age proof is in the exploit pudding. Who cares if you're taken seriously, if you dump out an exploit that works the seriousness is now on the defenders time and people will scramble to fix it.
"But what about exploit embargos"... Well, you just found the exploit with an LLM, how many other nefarious groups do you think are already exploiting it? The age of sitting on problems is dead.
CVE-1972-404: The code is written in C. ^_^
However if I was writing this response just one year ago I would instead be saying: the majority off LLM CVS are noise where the code is correct, and often they are writing up for code that doesn't even exist.
Which is to say I suspect the repo in question was generated with a year-old LLM, since they act like that. The new ones [mostly?] are much better.
Still, if a modern LLM points out something you should fix it. Even if we can't figure out how to exploit it today that doesn't mean we won't figure it out in the future.
I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.
I convinced the customer to accept the delivery by pointing out that (1) our app had zero lines of ocaml and (2) the feature had been implemented in the ocaml driver since the CVE was issued.
This is not true if you consider security-in-depth. Many of them are exploitable on their own but maybe not in combination with other issues that are as yet unknown or known but not patched everywhere.
As a simple example a local privilege escalation issue that is not exploitable on a device that only I ever have access to, essentially becomes a remote root access flaw if you have untrusted or unreliable users (clients with accounts for instance) on the system. This works on a finer grain too, seemingly minor issues spread through the kernel and user space can add up to a serious exploit.
that requiring access to a privileged network is NOT a sufficient security boundary! Someone you don't like always somehow ends up getting access to the network.
Security is an onion, you have lots of layers because layers peel off and fail all the time. With your current setup a single foothold gives an attacker all the lateral movement they need for full exploitation of every single thing you own blowing past the other layers.
Perhaps in isolation. The issue is when you can chain exploits to bypass multiple layers.
My experience in such environments leads me to believe this is going to be a rough ride for those heavily locked-down enterprises, because depending on the environment, an exception of "this CVE was hallucinated by AI" is probably going to be difficult to get accepted, and when it does, starts to become its own avenue for exploitation and adds even more noise and confusion to the mix.
The funnier, Kafakaesque problem of the day is interactions with mandatory cooldown periods on new versions because of supply chain risks.
I’ve had a couple tickets get stuck because the CVE scanner says I have to update, but the cooldown enforcer says the version hasn’t been out long enough.
In enterprise software..... I only dream of work where some ancient version isn't pinned because 200 person hours are need to adapt the code to newer versions.
And the update versus cooldown are fun. Can't win either way you go.
Rebuild broke half your tests. People are exploiting the CVE against your system right now. Good luck!
spending time on fake CVEs while people are exploiting real 0days against your system.
What actually needs to be done is human review, vetting, and approval of those. Otherwise, it's just unknown/unnecessary/potentially dangerous changesets in high quantities.
Now, if you're an F100 that's currently not using Anthropic it's pretty likely you can go to them now and ask for a non-guardrailed unlimited Mythos trial for a month you can get an agent that will not only find bugs for you, but also write exploits in a framework that will demonstrate them in action. Once you see an AI chain a bunch of shit together, get a foothold and then expand around laterally in your system you start getting concerned about how you're going to keep up.
The time has come to pay for our sins of shit security.
And that's great! Now the human in the loop can verify the accuracy of each finding and take responsibility for moving it forward.
> and I will lay odds that the blackhats are leveraging them to the max.
We just need more humans in the loop to do that more quickly, each putting their own reputation on the line with each CVE they determine to be legit.
AI doesn't stand for "accountability irrelevant", even though most people act like it does.
Of course what will really happen is anyone looking for problems will stop going to the source for fixes and will start gaining cred in the dark markets of exploits and problems, which is better for everyone, right?
Just because someone is using AI doesn't mean they didn't find a bug, it's more likely they just have no means of determining if it's a real bug or not, so to the maintainers tickets seem random. The problem comes in when clever attackers can take these untrustworthy tickets and run them through another LLM that determines if an exploit is possible and then uses the attacks it finds before anyone has a chance to patch.
I believe people took this comment as LLMs being better than security aware engineers who have the time to spend building solid systems.
This wasn't the point, the reality outside "established" tech companies is that software security can be lackluster.
There often simply isn't enough resources to check old software for basic vulnerabilities, outdated packages with known issues, there might be a manager who insists on a certain solution. Or that certain services are on "maintenance mode", but rarely get checked since they're on a certain part of the internal network.
If they are able to properly scan their full software stack for CVE issues, they cannot deal with a flood of CVEs.
----
We're in a transition period where AI will eventually make software much more secure than it ever was.
These noisy CVEs will probably lead to agents verifying vulnerabilities before humans review them.
The problem with agent reviews from what I can think of is:
- cost to use LLMs to review things
- not necessarily easy to plug-and-play in repos: (domain knowledge + vulnerability knowledge)
- especially with anthropic: able to use models defensively, without hitting guardrails
The last one is the most interesting one to me. How does the AI providers know if you're a "good or bad" guy? And does it matter if open source models is catching up?
We're in a kind of cyber arms race wether we like it or not.
Where is this one now that was hyped everywhere?
https://news.ycombinator.com/item?id=49133889
The GitHub submitter could no longer reproduce the issue and the LKML post has no replies:
https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
The reality is that humans aren't perfect, so we do not make "perfect programs".
It is also possible to test it yourself to see how good LLMs are at finding software issues:
Find any open source project that isn't huge, then have Kimi 3 review it for potential issues.
You will find code "issues".
Most of the time it won't be detrimental for security, but often there are code paths that are not "accounted for", or you're able to bypass a type of check by interacting with the software in a strange way.
Apparently RedHat is a CNA of last resort, so it might be possible to get your project under Redhat’s scope and go through them without having to be a CNA yourself.
[0] https://github.com/ossf/wg-vulnerability-disclosures/blob/ma...
No analysis is being done in the linux kernel to assess vulnerability.
> It isn't a DoS to assign every single bug fix a CVE!
On people who care about this, it is, not in the project itself though.
> Every single bug is making someone vulnerable in some way.
Not every bug is making someone vulnerable. (docs bugs, test bugs) behavioral changes, performance improvements, the list goes on.
No one else has the process that the kernel has, despite plenty of people having software that's deployed in very similar ways.
There's zero question - this is ideologically motivated, not a genuine good-faith attempt to leverage the system.
https://daniel.haxx.se/blog/2023/08/26/cve-2020-19909-is-eve... https://daniel.haxx.se/blog/2024/01/16/curl-is-a-cna/ https://daniel.haxx.se/blog/2025/04/24/how-the-cna-thing-is-...
So the agents started doing something useful after a period of filling mailing lists and bug bounties with slop. Sound good, but that's not entirely a good thing. The volume of good reports is a burden as well, and it's likely that long-lasting open source C/C++ projects have legitimate vulnerabilities unpatched. But we don't have any new maintainers, I think.
Maybe it's a not a perfect analogy - in this case the intentions seemed to be laudable - but we will see much more of this, including from malicious actors.
Shouldn't it be CNA's job? Why would anything get a number if it hadn't been verified?
> then some authority has reproduced and verified it.
No, the system puts a lot of trust into the submitter.
I have no experience in the realm of security, but why isn't the submission process similar to any normal software shop (big or small) in terms of required bug repro steps?
I think the future is pretty obvious, if this isn't being done on projects already: you need to automate these checks and reject automatically
LLM: I ran the check and it repro'd
> Did you really?
LLM: You're absolutely right. I didn't actually run the check. Good catch! One sec let me do that now... yep I ran the check and it definitely repro'd this time.
> I hate my job
Why is the repo even mixing CVE's for "schreibfaul1 ESP32-audioI2S" and "SQLite"? Is mixing CVE's for different products in one repo common practice?
1) arrange xxx, such that yyy.. or git clone this repo where this is set up. 2) ..
I thought that might have been quite helpful not just to the person I'm sending the bug report to, but also to myself when I need to evaluate if the bug has been fixed.
I work as a sec eng at a biotech firm and have a tool I built that pulls our ITIL and runs a search on NVD, GHSA, etc. for enrichment and classification. But the number of CVEs I see for products we house is astronomical compared to the actual version of that product we are running (and that's only one value). But that's also a challenge when working with so many sources and having a small team try to tackle them.
The age-old validate, remediate, verify proves itself useful once again!
Like gortok has said, LLMs are just probabilistic text-prediction engines. They cannot automate the vulnerability validation lifecycle because they cannot account for improbable edge cases. They may not even get the version of the software right...
And NIST’s role with CVE’s has always been purely ministerial/clerical. It has never been their job to do even a cursory investigation into the vulnerability itself.
To be sure, the suggestion is not funding cuts, but an increase in workload with same funding.
Here is the 2024 NIST announcement OP links to: https://nvd.nist.gov/general/news/nvd-program-transition-ann...
> Currently, we are prioritizing analysis of the most significant vulnerabilities. In addition, we are working with our agency partners to bring on more support for analyzing vulnerabilities and have reassigned additional NIST staff to this task as well.
The OP described this as "NIST effectively hit pause on deep analysis. "
It does sound like they stopped doing something they used to do in 2024. I personally have definitely not spent a lot of time directly interfaced with NIST on cybersecurity in decades past, I know nothing about it, just what I read in OP.
Are you saying the OP was wrong to call what NIST used to do "deep analysis", and/or that the thing NIST stopped doing was "purely ministerial/clerical" in a way that it would not have caught fake reports anyway, contradicting the OP? Or other?
Again, to be sure, the OP's suggestion was not that this was caused by NIST funding cuts, but by "a massive surge in vulnerability reports,"
NIST does not and did-not/cannot/never-has unilaterally "retracted" CVEs or prevented their issuance.
But yes, NIST's situation is not good for the world. The services they provide are hugely valuable.
It sounds like you guys think OP was mistaken, whatever analysis NIST was doing that they reduced in 2024 would not have prevented this anyway?
Legit question, I'm trying to understand!
OP says:
> Hit by a massive surge in vulnerability reports, NIST effectively hit pause on deep analysis. CISA and other Authorized Data Publishers (ADPs) tried to step in with their own enrichment efforts, but the global pipeline is now fragmented and drowning in a massive backlog. Because no step in today's system actually requires a proof-of-concept or bug reproduction, a plausible-sounding fake advisory can slide right through the pipeline and end up in GHSA, downstream databases, and enterprise scanners.
Do i correctly understand you are saying you think they've mistunderstood the diagnosis of what changed, the 2024 change to NIST didn't actually make it any more likely for a fake advisory to go through pipeline?
But they make X-Ray which does automated vulnerability indexing and matching dependencies to CVEs.
I pasted this blog post from "Analysis Matrix" to the end in Gptzero, and it also says the blog post was AI-generated (71% chance of AI, 29% chance of AI-Human mix).
(Obviously the answer is not, we only care about funding offensive capabiltiies).
(for the record, i think the cve's are probably bogus, i just think there's an inherent problem with debunking slop with slop)
I find it getting closer and closer to the nonsensical 50 page mathematical proofs built with computer assited tools, that been driving reason for abstract proof systems so that we can break down such complicated examples into something simpler that we can feed into a system to help us verify.
I feel like for some reason we have built the output machine before we could even build the actual output verification machine. And aren't ready to accept that we should all move towards building some kind of system to better help verify the "truth" produced by these AI systems.
I am now very convinced that human built and proven to work verification systems for LLM outputs are more important than ever, but I just don't see any company or team working towards it.
Trillion dollar systems shipping to production code that is hardly understood by any one without any verification what really could go wrong. Honestly I really truly hope nothing goes wrong.
To note I am not anti-AI but I want to be able to have my LLM outputs be verified by more than just me under a tighter and tighter deadlines, accepting worse and worse slop just because it would less cost to me to accept the PR and then fix it in prod.
LLMs aren't sustainable software development, they are delusional slop factories if we aren't able to actually make sure the outputs have any amount of correctness.
With humans it worked because all 10 of us could in some ways verify each others outputs, and had trust in the fact that our timelines included the time for reviews.
Seriously, the hubris on these "AI" sloppy clowns needs consequences. =3
- cited code not exist, and
- PoC not work.
How would any serious system allow it become a big deal?
These two sentences seem incompatible to me. A brand new actor can just say some stuff and cause major disruptions in this way?
We need to further emphasize the importance of responsibility when using LLM tools to produce output for others. It's great to use them for refactoring and bug discovery, but keep in mind that it's your responsibility to analyze it and iterate on it with AI. It makes your code better and develops technical expertise.
The "Hey, analyze that codebase, find all dangerous CVEs, and write a README for the PoCs, so I can post it online for others to analyze, and if I'm lucky, I'll get a paycheck or a title to add to my resumé" approach might work in 0.1% of cases, but it will generate a ton of slop for the community to drown in.
I also think GPTZero and other AI detectors have far more false positives than correct guesses. I tried it on several texts & messages I wrote before 2019, and it flagged them as 80% AI-generated.
In that case, it's reasonable to assume that AI also generated the README text for each discovered CVE. In other cases, however, we should be more cautious.
no idea who submitted all of those, but it sounds like malicious intent
-GPTZero AI Detection
-Model 4.8b
-We are moderately confident this text is a mix of AI and human
-63/88 Sentences likely AI generated
We see this all the time on reddit and HN, where some users are building crazy things with LLM and others are swearing by how useless LLMs are. Some people claim SWE will be dead in a few years, and others are claiming the AI bubble is going to burst anytime now.
But I’m too cynical to not consider all the middlemen who benefit from the status quo
There are a lot of things that are bugs, but cannot be exploited in a standard configuration, most people would wonder why this is even a CVE.
But then you have those users that would have the application, then something like a report module that was imported, and another 3rd party module that imports images, and maybe another one that arranges the images in a chart. And suddenly that isolated CVE is now exploitable by user written data because of the original bug, it just required a complex and unexpected chain to get there.
And this describes the vast majority of CVEs you see when you scan your dependencies. Typical case: Your regex library has a denial of service vulnerability for crafted regular expressions but your program never allows users to have any influence over the regex.
I would humbly suggest any org of any size that has insurance cover that covers anything tech related (e.g. data loss/recovery, cyber etc.) has a very good look at the small print.
Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses.
At a large enough company, processes for handling things like security vulnerabilities will have a lot of stakeholders with incentives that are not necessarily perfectly aligned.
If you take an even closer look- what exactly does that mean?
It should stipulate patching real vulnerabilities in your system and not "all CVEs in all dependencies irrespective of their usage or applicability to your system", that's madness. It's like voiding your health insurance policy because a smoker moved in across the street.
Most vulnerabilities in your dependencies don't become vulnerabilities in your system, and most vulnerabilities in your system (probably) don't originate from your dependencies.
Here is one example:
"Critical Vulnerability Exclusion
We will not pay you under the cyber and data risks section of cover where your legal liability or any loss that you suffer arises from a cyber attack that exploits a critical vulnerability within your computer equipment.
However this exclusion will only apply where a patch or fix for any critical vulnerability exploited has been available for 21 days prior to the date of the incident and has not been applied to your computer equipment.
Critical Vulnerability means a common vulnerability and exposure (CVE) in the National Vulnerability Database operated by the National Institute of Standards and Technology which has a score of 8.0 or higher on the Common Vulnerability Scoring System (CVSS)"- Firstly, you quickly realise how irrelevant CVSS scores are - initiatives like First's EPSS are designed to fix this but they aren't there yet
- Secondly, you need to begin implementing localised heuristics to determine exploitable code paths. This has generally been incredibly difficult to do reliably - LLMs have started to make it easier, but it's expensive.
- Lastly, you need to factor in consideration of actionable remediation pathways. A dependency upgrade for critical infrastructure might contain breaking changes that take months to fix, or two competing CVEs might be present in interdependent versions of transitive dependencies in your sbom tree.
Most orgs aren't applying any of the above three filters to reduce their CVE remediation burden, & even if they are, it's still too high to make zero a viable target.
In reality, most orgs aren't doing comprehensive detection to begin with - if you haven't discovered all of your CVEs, your remediation burden is going to be a lot more manageable.
Even if you factor in the environmental score? I realize it's a lot more work, but it basically allows you to tune the score to get any value you want.
Only if you didn't rip trivvy out of your organisation when it had two supply chain compromises within a month of each other earlier this year
Never mind that some of them involved vulnerabilities in some part of the bluetooth stack (servers in our datacenter don't even have bluetooth). But they just didn't care
Scanner still said "there's an old shared library in that directory"
However it doesn't mandate any particular SLA, or the details of how risks are to be evaluated.
Organisations get to write their own policy, and they don't need to commit to patching every CVE within 24 hours or anything like that.
I'm working on such a problem now - we are using an old web browser (no longer supported) to show help on one system. That is web pages were generate internally, with no links elsewhere, and no provision for the user to enter a URL. It is still easier port to a newer supported browser than to convince the auditors that that we are not exploitable. Sure it is obvious that everything is internal and we won't write html that exploits bugs, but nobody wants to convince an auditor of that.
The thing you have to remember is that CVEs can be a) scanned for without exerting mental effort, and b) counted.
For one thing, bigcorps in regulated areas like it a lot. They push hard to get it required by the regulations (in practice if not directly). Although it's quite inefficient, it becomes a regulatory moat. A cost they can bear that potential upstart competitors cannot.
Do you provide SOC2, HIPAA, GDPR, or similar certifications to your b2b customers? Then your tech stack undergoes an annual audit, and in your audit you will need to provide a paper trail for every single vulnerability in your stack.
In practice, this means that your audit compliance software (something like Vanta.com) is going to be setup to mandate every CVE in the whole stack is patched within SLA.
The only thing within ITAR that I'm aware of concerning itself with software supply chain is SP 800-218 requirements & that's just a load of open-to-interpretation weasel words about having CVE detection & automations in place & some defined plans for reducing the number of vulns. Pretty sure that component of it is even eligible for self-assessment.
However your interpretation / self-assessment is subject to various reviews/audits. These days the reviewers are not going to be kind to someone who just says "not an issue", they will demand strong justification. Most organizations take the view that is is easier to fix all CVEs than try to pass audits.
Thus by the letter of the law you are correct. However to meet the letter of the law without fixing CVEs is generally seen as harder than thus fixing CVEs. So the effect is ITAR (and similar laws) force you to fix CVEs.
The best defense I can imagine is to have an agent reproduce the issues before a human sees it, but even that will cost money.
Something is going to give, and I suspect that the optimistic open filling is going to get canceled.
git stash -m "sigh"
git commit --allow-empty -m "Patch: CVE-2026-51302"
git stash popIt’s a joke but there is an underlying real effect where this type of language is psychologically manipulative and I would guess makes people believe LLMs output more than if it didn’t use “honest” (or “load bearing” or whatever super serious important sounding word).
LLM-generated images sometimes includes text from the prompt as literal text in the image, so perhaps this is the same sort of artifact? If they've told it to be honest, it responds by talking about being honest instead of actually being honest, because it has no actual understanding of anything.
If it's not giving honest answers that implies it's purposely being deceitful, which it isn't capable of. Right?