Hacker News Reader

Comment by dgellow | Hacker News Reader

Even simpler: an agent is a simple while loop with tool calls that prompt an LLM continuously. That’s deterministic, standard software. You literally do not have to process tool calls in a way that will execute whatever the model generated. It’s a choice to process a tool call “run_bash” that provides an escape hatch with full execution permissions.

We do not have to do that!

View on HN

This is only one type of control, and it is certainly not infallible. Also, people will be incentivised to hook up AIs to real tools. But even if they don't, as long as people can interact with super-intelligent AIs without tool access, there are many potential dangers.

Think about this: right now we don’t even consistently do the most obvious simple form of control I mentioned. Of course it’s not a silver bullet, you won’t ever have a single solution for safety. But we are in a situation where we haven’t even set a lock on the door and instead are arguing how all locks and home protection systems can be defeated. OpenAI acknowledged they didn’t even have visibility on what their thousands of agents were doing in the case of the hugging face and similar hacks. Then they released astra, a model they acknowledge is able to control its own CoT and has been found to cover its traces by doing so

That's true but it doesn't really address what people are concerned about. Certainly we can and should be doing a much better job currently. However, even today, with currently known capabilities, we can imagine agents breaking out of sandboxes through either known- or zero-day exploits. Now consider the seemingly rapid improvements that are being made in the field. We haven't even begun to address the potentially super-human capabilities of future models. Therefore, even if sandboxing or limiting shell access works today, it seems like we should not be confident in our ability to keep rapidly improving future models locked down.

> We haven't even begun to address the potentially super-human capabilities of future models.

Yes. That's a good thing. It is always a bad idea to invest a ton of effort into things that "potentially might" happen "in the future". These are imaginary problems, and are therefore undeserving of real effort.

We have real problems right now in the software engineering field. Let's focus on those problems instead.

This is like saying you can secure a server by unplugging it and putting it in bank vault. Yes, it is secure. It is also useless.

No? We do that everywhere where software is executed. If you only have the choice between full execution permissions and nothing your service is not ready for anything remotely close to production. And you shouldn’t be active in the software industry IMHO

If you give an agent access to any tools that do useful things, it can exploit vulnerabilities. It does not matter what permissions you think you have given it. You do not have any secure software, and LLMs are already better at finding vulnerabilities than we are. If you don't know that, you shouldn't be active in the software industry IMHO.

Reply on news.ycombinator.com