Ask HN: Is AI going to ruin FOSS? | Hacker News Reader

Ask HN: Is AI going to ruin FOSS?

4 pointsaf3d9 comments

I almost feel like this deserves a blogpost, but I'll just go ahead ask the question directly.

The fact that AI could be used by bad actors to create (or even subvert existing) open-source software seems to me to pose a fairly imminent threat. Advanced obfuscation techniques, super-humanly complicated and/or subtle penetration methods, the ability to imbue a "legitimate" or "authoritative" looking documentation, et cetera, all of these things will only tend to lead toward more vulnerabilities (as if there weren't enough!). And most likely a paranoia in developers when it comes down to cloning an running a repo! The (as it were) old fashioned approach aka "reading through the code" just isn't going to cut it anymore, is it?

The recent XZ backdoor comes to mind, actually. I have a sneaking suspicion that this too may have been constructed in such a way. This could really stall FOSS projects imo. After all, the harder it is to trust a code base, the less likely one is to bother even participate (much less use the software, for that matter).

I’d argue the opposite. This happened, as how I understood it when it was explained, is that while code gets lots and lots of eyeballs, non- trivial make files get little to no real review as it’s often hard to reason about them. The exploit took advantage of going where the security was weakest. The more AI is involved in reviewing all aspects the better off the process will be, especially the tedious and boring parts of all this.

I expect we will see a spate of security bots looking at all manner of things in the process as a reaction to all this.

Ken Thompson’s issues with trust will still apply. However these issues are with us with or without AI being employed.

Fair enough. Of course, projects with few retainers will likely not have near as many eyeballs looking over the code base. I do see your point, however, in a sense nothing has really changed. Although the job of maintaining a FOSS codebase may very well become just a bit more tedious. But yes, Ken Thompson's "Trusting Trust" principles do seem to address the issue quite well. When it comes to code review, always be diligent!

I don't think so. Sophisticated threat actors have been attacking the Open Source community for decades, adding AI into the equation changes nothing from my perspective. I'd actually imagine it's harder to get the precise and perfectly-obfuscated behavior you want while going through an LLM as a midpoint.

So my guess is no. Even in the case of the XZ backdoor, it was the incredible amount of social-engineering at play that enabled the hack. The developer had been conning the repo maintainer for years, which is easier achieved with human intellect than ChatGPT and a fake Github account.

Well it is true, the XZ hack was a major social-engineering feat. Regardless, we should not be underestimating the power of these technologies. If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you. And especially as this technology progresses, the capabilities will no doubt dramatically increase. Having said that, I do hope you are right!

> If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you.

What makes you think an attacker couldn't do that without an LLM? I'm having a hard time understanding what changes in this scenario by introducing an AI.

Pretty much anyone can write a buffer overflow exploit with enough research. The harder part is getting your patches through code review, which is probably only hindered by having ChatGPT help you. Again... the past few decades have mostly gone off without a hitch, and there have been a lot of monkeys on typewriters hooked up to the internet.

I don't know, LLM's are very good at embedding subtle changes in their responses. They are also good at researching things like "best possible place to hide bugs", etc. Is it really a stretch to imagine that these technologies won't, if not now, but "someday soon" be used in such a way? I certainly don't think so.

Humans are really good at subtlety and research. I guess my point is this:

  pre-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge malicious patches and hope nobody sees them

  post-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge patches that were written slightly faster and hope nobody sees them

For 99% of people, there isn't a meaningful difference between these situations. I really don't think there are people out there that couldn't write a buffer overflow exploit in 5 years but could groom maintainers for trust in the meantime.

The code-writing and prose bit is the easy part. The social engineering and deception is hard enough for a human, and Turing-assured destruction for LLMs.

You seem to think AI is more clever than it is. As far as I know, AI tends to produce generic boilerplate code at best and that code becomes extremely hallucinatory beyond non-trivial complexity or when seeking non-commonplace solutions. I don't think anyone's going to be using AI for 1337 h@x any time soon. AI doesn't innovate, it generalizes.

The more pertinent issue is unintentional bugs and low quality in AI generated code, and the effect of knowledge loss as AI generated code becomes more and more common.

Well I guess we just have different standards of "cleverness". As far as I am concerned these technologies already demonstrate a level of subtlety that would allow them to be used in such ways, as is. Nevermind six months from now, or whatever, when it actually reaches a level of refinement where such exploits become trivial (and probably common).

Concerning your second point, as long as there is a human "in the loop" (verifying, testing, etc) that shouldn't really be too much of a concern, should it?

As to the effects of AI-generated content being vacuumed up as training data, that is an interesting question, but also one which I feel is somewhat unresolved at the moment. Is anything really "lost" in the process? Maybe the output becomes more redundant, sure, I just have a hard time believing that that could possibly "break" an LLM. (I have wondered if the mere fact that the output is being fed into the input implies something akin to a "fixed-point iteration" which might somehow become an issue, but again, I honestly don't know!)

Reply on news.ycombinator.com