Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In that case, this would be as true for random people submitting malicious PRs that someone accepts without reading, even without an LLM involved at all? Seems like an odd thing to call a "security nightmare".
Cognitively, these are fairly distinct tasks. When creating code, we imagine architecture, tech solutions, specific ways of implementing, etc., pre-task. When reviewing code, we're given all these.
Sure, some of that thinking would go into prompting, but not to such a detail as when coding.
What follows is that it's easier to make a vulnerability pass through. More so, given that we're potentially exposed to more of them. After all, no one coding manually would consciously add vulnerability to their code base. Ultimately, all such cases are by omission.
A compromised coding agent would try that. So, we have to change the lenses from "vulnerability by omission only" to "all sorts of malicious active changes" too.
An entirely separate discussion is who reviews the code and what security knowledge they have. It's easy to dismiss the concern once a developer has been dealing with security for years. But these are not the only developers who use coding agents.
The agent decides to import a bunch of different packages. One of them is a utility package hallucinated by the LLM. Just one line being imported erroneously, and now someone can easily exfiltrate data from your internal DB and make it very expensive. And it all looks correct upfront.
You know what the nice thing is about actually writing code? We make inferences and reasoning for what we need to do. We make educated judgments about whether or not we need to use a utility package for what we're doing, and in the process of using said utility can deduce how it functions and why. We can verify that it's a valid, safe tool to use in production environments. And this reduces the total attack surface immensely; even if some things can slip through the odds of it occurring are drastically reduced.
Consider business pressures as well. If LLMs speed up coding 2x (generously), will management accept losing that because of increased scrutiny?
If they don't then they're stupid
This has been the core of my argument against LLM tools for coding all along. Yes they might get you to a working piece of software faster, but if you are doing actual due diligence reviewing the code they produce then you likely aren't saving time
The only way they save you time is if you are careless and not doing your due diligence to verify their outputs
They’re just not experienced because this has never happened before. But when you call them stupid, they’re not going to listen to you because they won’t like you.
I don't believe people can actually be persuaded by words in today's age
The CTO of my company has pushed multiple AI written PRs that had obvious breaks/edge cases, even after chastising other people for having done the same.
It's not an experience issue. It's a complacency issue. It's a testing issue. It's the price companies pay to get products out the door as quickly as possible.
At that level, it's the combination of all the power and not that much tech expertise anymore. A vulnerable place.
A lot of famous hacks targeted humans as a primary weak point (gullibility, incompetence, naivety, greed, curiosity, take your pick), and technology only as a secondary follow-up.
An example: Someone had to pick up that "dropped" pendrive in a cantina and plug it into their computer in a 100% isolated site to enable Stuxnet.
Were I a black hat hacker, targeting CTOs' egos would be high on my priority list.
I am not a luddite. I see great potential in this tech, but holy mackarel will there be price to pay.
The nature of code reviews has changed too. Up until recently I could expect the PR to be mostly understood by the author. Now the code is littered with odd patterns, making it almost adversarial.
Both can be minimised in a solid culture.
i refuse to review PRs that are not 100% understood by the author. it is incredibly disrespectful to unload a bunch of LLM slop onto your peers to review.
if LLMs saved you time, it cannot be at the expense of my time.
Expect this to become the norm. I hate it.
let's see how that holds up against "author does not understand own PR"
If insecure code makes it past that then there are bigger issues - why did no one catch this, is the team understanding the tech stack well enough, and did security scanning / tooling fall short, and if so how can that be improved?
Hasn’t this been the case for entire categories of bugs? Stop me if you’ve heard this one before but we have a new critical 10/10 cvs that was dormant for the last 6 years…it was introduced in this innocuous refactor of some utility function and nobody noticed the subtle logic flaw….
AI can write plausible code without stopping. So, not only you get sheer volume of PRs going up at the same time you might be asked to do things "faster" because you can always use AI. I am sure some CTOs might even say - why not use AI to review AI code to make it faster?
Not to mention previously the random people submitting malicious PRs needed to have some experience. But now every script kiddie can get LLMs to out the malicious PRs without knowledge and scale. How is that not a "security nightmare"?
When management wants to see dollars, extra reviews are an easy place to cut. They don’t have the experience to understand what they’re doing because this has never happened before.
Meanwhile the technical people complain but not in a way that non technical people can understand. So you create data points that are not accessible to decision makers and there you go, software gets bad for a little while.
It's been going on since Stack Exchange copypasta, and even before that in other forms. Nothing new under the sun.
The telling thing is they never mention this "threshold" in the first place, it's only a response to being called on the bullshit.
Increasing the quantity of something that is already an issue without automation involved will cause more issues.
That's not moving the goalposts, it's pointing out something that should be obvious to someone with domain experience.
Every post like this has a tone like they are describing a new phenomenon caused by AI, but it's just a normal professional code quality problem that has always existed.
Consider the difference between these two:
1. AI allows programmers to write sloppy code and commit things without fully checking/testing their code
2. AI greatly increases the speed at which code can be generated, but doesn't nearly improve as much the speed of reviewing code, so we're making software harder to verify
The second is a more accurate picture of what's happening, but comes off much less sensational in a social media post. When people post the 1st example, I discredit them immediately for trying to fear-monger and bait engagement rather than discussing the real problems with AI programming and how to prevent/solve them.
Allowing people who have absolutely no idea about what they're doing to create and release a software product will produce more "code slop", just like AI produces more "article slop" on the internet.
I don't understand the distinction you are trying to draw between your two examples. Instance #1 happens constantly, and is encouraged in many cases by management who have no idea what programmers do beyond costing them a lot of money.
You can internally discredit whomever or whatever you like, but it doesn't change the fact that LLMs currently add very little value to software development at large, and it doesn't appear that there is a path to changing that in the foreseeable future.