Hacker News Reader

Comment by skissane | Hacker News Reader

> Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed

I think this is binary categorical thinking. In the real world, safety systems (even in domains like aviation or nuclear power) are never foolproof-the point is you reduce the probability of failure to an acceptable level given the costs of doing so and the potential consequences of that failure

And there is the risk people say “there is no foolproof solution, so I’m not going to invest in probabilistic countermeasures” - which would sound like utter madness to a bank’s antifraud department, but for some reason a lot of people seem to think it isn’t when it comes to AI

View on HN

The issue is with the nature of agentic systems and how often they run, a .001 failure is still huge. Attackers monitor their attack chances and use the most successful attacks so the actual success on the attackers side is much closer to 1 than .001.

This is the problem, this isn't a 'failure' mode where something randomly goes wrong, like a person accidentally sticking their hand in a machine. This is war where you are under active attack and the attackers adapt quickly.

Because of the breadth of LLM capabilities they have a nearly unbound attack surface. You spend so much of your effort making sure your prompt is secure that you either massively increase costs, or you partially lobotomize your agent making it less capable.

Solutions for these problems will have to come inside the model itself because both the attackers and models are far more capable than you on doing dumb/evil shit. Until the model itself can go "hey, wait a darn minute" a new fun way of encoding the attack can always be found.

> This is war where you are under active attack and the attackers adapt quickly.

It all depends on what the use case is.

For example, consider a system which takes English questions from business users, translates them to SQL, then runs it (as the business user) against a reporting database

How does the attacker get access to it? And even if they somehow do, what harm can they actually do to it? The only tool it has is one to run SELECT against a DB, and it can only SELECT data the user already has access to, and we have timeouts/etc to protect against overly expensive queries

If your design involves lots of general purpose do-anything agents with very generic tools, the risk profile is very different from narrowly scoped purpose-specific AI services

I mean, yeah, consider a system that takes English questions and does nothing with them, it’s very secure. What’s your point? It doesn’t really refute what the commenter above you was saying.

Security is a process, not a product.

I have to remind the folks who want a dev to build a product and think they don't need to ever touch the product after release.

Reply on news.ycombinator.com