Hacker News Reader

Comment by InsideOutSanta | Hacker News Reader

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

View on HN

> If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

You don't even need to give it access to Internet to have issues. The training data is untrusted.

It's a guarantee that bad actors are spreading compromised code to infect the training data of future models.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

A human worker can be coached, fired, terminated, sued, any number of things can be done to a human worker for making such a mistake or willful attack. But AI companies, as we have seen with almost every issue so far, will be given a pass while Sam Altman sycophants cheer and talk about how it'll "get better" in the future, just trust them.

Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse. If a physical product didn't do what it claimed to do, it would be recalled and the maker would get sued. Why does AI get a pass just pooping out plausible but incorrect, and sometimes very dangerous, answers?

> Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse

I also have never written a bug, fellow alien.

I do not fear the employee who makes a mistake, I fear the AI that will make hundreds of mistakes in thousands of companies, endlessly.

As employees also do across thousands of companies.

> I don’t see any conceptual difference between a human and an automated system on this front

If an employee of a third party contractor did something like that, I think you’d have better chances of recovering damages from them as opposed to from OpenAI for something one of its LLMs does on your behalf.

There are probably other practical differences.

We need to take a page from baseball and examine Hacks Above Replacement

If anything, the limit of RBAC is ultimately the human attention required to provision, maintain and monitor the systems. Endpoint security monitoring is only as sophisticated as the algorithm that does the monitoring.

I'm actually most worried about the ease of deploying RBAC with more sophisticated monitoring to control humans but for goals that I would not agree with. Imagine every single thing you do on your computer being checked by a model to make sure it is "safe" or "allowed".

The difference is 'accountability' and it always will be.

no, you have a trust problem. Is the tool assisting, or is are the tools the architect, builder, manager, court and bank?

>If you allow a human to edit your code and also give them access to untrusted data (like the Internet), you have a security problem.

Security shouldn't be viewed in absolutes (either you are secure or you aren') but more in degrees. Llms can be used securely just the same as everything else, nothing is ever perfectly secure

Things can only be used securely if they have properties that can be reasoned about and relied upon.

This is why we don't usually have critical processes that depend on "human always does the right thing" (c.f. maker/checker controls).

They can be reasoned about and relied upon.

The problem is that people/users/businesses skip the reasoning part and go straight to the rely upon part.

They can be reasoned about from a mathematical perspective yes. An LLM will happily shim out your code to make a test pass. Most people would consider that “unreasonable”.

Reply on news.ycombinator.com