The world is so not ready for the impact of LLMs on security issues. If true, congrats to the Calif team. It’s likely too technical for me to understand in details but looking forward to reading the 55 pages report
I agree, but it's the people I'm worried about.
I'm hearing anecdotes from all over about devs pushing LLM-generated code changes into production without retaining any knowledge of what it is they're pushing. The changes compound, their understanding of the codebase diminishes, and so the actions become risker.
What's worse is a lot of this behavior is being driven by leaders, whether directly (e.g. unrealistic velocity goals, promoting people based on hand-wavy "use AI" initiatives, etc) or indirectly (e.g. layoffs overloading remaining devs, putting inexperienced devs in senior rolls, etc).
The world's gone mad and large swaths of the industry seem hellbent on rediscovering the security basics the hard way.
juniors have been writing code forever that is imperfect and not memorized by the people reviewing
isnt the important thing the mechanisms for maintaining the code?
When Sundar Pichai announces that 75% of all new code at Google is AI-generated, their stock price goes up. If he were to announce that 75% of all new code at Google is now written by junior engineers, this would trigger a massive sell-off and a lot of employees would resign.
Seniors are only part of the picture as team leads, or when it escalates after big screwups.
Seniors are only part of the picture as team lead, or when it escalates after big screwups.
Will we now have leetcode of prompt writing?
I don’t think so.
An LLM can produce higher-quality documentation than most humans. If it's not already happening, when a new developer joins a team, they're going to have an LLM produce any documentation a new developer needs, including why certain decisions were made.
It could also summarize years of email threads and code reviews that, let's face it, a new person wouldn’t be able to ingest anyway; it's not like a new developer gets to take a week off to get caught up on everything that happened before they got there. English not their first language? Well, the LLM can present the information in virtually any language required.
As the models continue to improve, they'll spot patterns in the code that a human wouldn’t be able to see.
Can bears some heavy weight.
LLM generated documentation has so low level of information density, that it’s useless. Yes, it writes nice sentences… or even writes. But it contains so much noise that currently, reading code is a better documentation than what I’ve seen from every single LLM generated documentation.
The same with LLM generated articles. I close them after the second sentence because at least about 90% of it is useless filler.
Now compare that to this: https://slate.com/technology/2004/11/the-death-of-the-last-m...
I almost closed it when I read the first few sentences because these kinds of articles are useless time wasting nonsenses. But this was different. This was old. Most sentences contained something new. Something worthy. (Of course, people also write unnecessary long articles… looking at you Atlantic)
You can throw out almost everything by volume from LLM generated documentation without loosing any information.
Currently, if I smell (and it’s very easy to smell) LLM generated documentation or article, then I close it immediately, because it’s good for only one thing: wasting my time, for no good reason.
I should clarify: the documentation I’m talking about is not generated using a generic LLM prompt, which would mostly suck.
With the proper context and additions (skills, plugins, MCPs) LLMs can produce high-quality documentation. You'd also have subagents doing QA of the documentation.
But it does require effort; it’s not magic.
If stuff really goes wrong, you need people who deeply understand the codebase so that they know where to look and how to diagnose the issue. It might be the case in the future that LLMs become so powerful they'll diagnose any issue (I doubt it), but until then, we need people in the loop.
No anecdotes needed, it's entirely happening.
But it's also devs, being devs.
So much out of date software with known exploits left running for years. The only reason there hasn't been total disaster is no one has tried to hack it yet.
The root problem is the world runs on C code that is riddled with vulnerabilities.
I'd imagine this set is very similar to just "the set of software on the world". Even before the AI stuff, it was a pretty good bet at any given software had some vulnerability; it was just a question of how easy to was to find it.
I mean we are literally in a thread about how the 4 trillion dollar company, literally the 3rd most valuable company in the world, with a core competency in software has, yet again, released a core product riddled with security defects for the 50th year in a row.
Commercial IT security is a industry that is incapable to a fault and has, so far, faced basically zero consequences for it.
This is true in America in many industries now, but most of the rest of the world (even the rest of the OECD) is still far behind.
Even more so in the future when a software company can be launched by a farm of AI Agents with a founder at helm with no clue about computing or security.
What's debateable is how many of those companies actually need irontight security, because they are never realistically going to be targets of criminals and/or they have nothing valuable to steal/corrupt in the first place (other than the owner's pride).
I still have nightmares about the contact form on my low-stakes personal website getting hijacked to use as a spam sender (because I used unsanitized input in mail headers).
I was pointing out how even Apple, a entity who by all rights should have top-notch security, is still absolutely hopeless in the face of commonplace commercial, profit-motivated attackers.
Massive, extremely well-resourced divisions supported by management in a technically competent organization that is actually trying to solve the problem struggle to produce at best middling security that is inadequate against commonplace threats. This is not a prioritization problem; even if you do “everything right” you are still vulnerable to run-of-the-mill commercial attackers. This is a fundamental capability problem, like how we can not make a net positive fusion reactor right now.
It is actually unfair to blame these companies for not having a fusion reactor because they “were not trying hard enough”. Actual security is not a easy problem, and it is a great disservice to portray it as one that is only unsolved due to dunderheads being in charge since it leads to underestimating what actually needs to be done.
That is not to say that you can not do dramatically worse than the “gold standard” and also that most organizations are actually incompetent; but the “gold standard” is still objectively grossly inadequate. You need to be dramatically better than the 4 trillion dollar software company to reach adequate against prevailing threats.
Exploits are BAD!
Then you have the many companies in the UK, US, Canada, EU that have compliance and regulatory laws that require them to exist in some capacity in house. Though that is changing with MDR services, but someone still has to interface with the MDR.
[1]: https://www.elastic.co/pdf/sans-soc-survey-2025.pdf [2]: https://github.com/jacobdjwilson/awesome-annual-security-rep...