Hacker News Reader

Comment by martinohansen | Hacker News Reader

Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source models succeed.

View on HN

I work for a large closed-source software company and I can tell you with 100% that it is full of domestic and foreign agents. Being open source means that more eyes can and will look at something. That only increases the chance of malicious actions being found out ... just like this supply-chain attack.

Reminds me of the scene in Fight Club where the unreliable narrator is discussing car defects to a fellow airline passenger.

Quoting from flawed memory:

Passenger: Which company?

Narrator: A large one.

Why would open source models make this scenario you are painting better?

Because in the closed source model the frustrated developer that looked into this SSH slowness submits a ticket for the owner of the malicious code to dismiss.

This seems completely unrelated to the grandparent comment’s mention of open source LLMs

You're right, I read the comment as:

> Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source () succeed.

I guess we can only hope verifiable and open source models can counteract the state actors.

It’s insane to consider the actual discovery of this to be anything other than a lightning strike. What’s more interesting here is that we can say with near certainty that there are other backdoors like this out there.

Time to start looking at similar cases for sure.

Not necessarily. A frustrated developer posts about it, it catches attention of someone who knows how to use Ghidra et al, and it gets dug out quite fast.

Except, with closed-source software maintained by a for-profit company, suck cockup would mean a huge reputational hit, with billions of dollars of lost market cap. So, there are very high incentives for companies to vet their devs, have proper code reviews, etc.

But with open-source, anyone can be a contributor, everyone is a friend, and nobody is reliably real-world-identifiable. So, carrying out such attacks is easier by orders magnitude.

> So, there are very high incentives for companies to vet their devs, have proper code reviews, etc.

I'm not sure about that. It takes a few leetcode interviews to get in major tech companies. As for the review process, it's not always thorough (if it looks legit and the tests pass...). However, employees are identifiable and would take huge risk to be caught doing anything fishy.

Absolutely not. Getting a job at any critical infrastructure software dev company is easier than contributing to the Linux kernel.

Can confirm. I may work at Meta, but I was nearly banned from contributing to an open source project because my commits kept introducing bugs.

We witnessed Juniper generating their VPN keys with Dual EC DRGB, and then the generator constants subverted with Juniper claiming of now knowing how did it happen.

I don’t think it affected Juniper firewall business in any significant way.

... if we want security it needs trust anyway. it doesn't matter if it's amazing Code GPT or Chad NSA, the PR needs to be reviewed by someone we trust.

it's the trust that's the problem.

web of trust purists were right just ahead of the time.

That just…doesn’t make any sense.

Everyone starts from zero and works their way up.

It would actually be sort of interesting if multiple adversarial intelligence agencies could review and sign commits. We might not trust any particular intelligence agency, but I bet the NSA and China would both be interested in not letting much through, if they knew the other guy was looking.

That is an interesting solution. If China, US, Russia, EU, etc all sign off and say "yep this is secure" we should trust it. Since if they think they found an exploit, they might assume the other people found an exploit. This is a little bit like the idea of a fair cut for a cake. If you have two people that want the last slice of cake, you have one cut and the other choose the first slice, since the chooser will choose the biggest slice, so the slicer knowing they will get the smaller will make it as equal as possible. In this case the NSA makes the cut (the code), and Russia / China chooses if its allowed in.

NSA makes the cut and China picks the public key to use.

In all seriousness, those people will quickly find some middle ground and will just share keys with each other

Maybe also throw EFF into the mix.

Chad NSA

It's called the ANS is Chad.

this is why microsoft bought github and has been onboarding major open source projects. they will be the trusted 3rd party (whether we like it our not is a different story)

Imagine a world where a single OSS maintainer can do the work of 100 of today’s engineers thanks to AI. In the world you describe, it seems likely that contributors would decrease as individual productivity increases.

And be burned out 100x faster

Wouldn't everything produced by an AI explicitly have to be checked/reviewed by a human? If not, then the attack vector just shifts to the AI model and that's where the backdoor is placed. Sure, one may be 50 times more efficient at maintaining such packages but the problem of verifiably secure systems actually gets worse not better.

Presumably the state actors are looking for other state actor's bugs, and would try to fix them, or least fix them to only work for them.

That's quite a game of cat and mouse.

Why AI agents?

Reply on news.ycombinator.com