Hacker News Reader

Comment by jtc331 | Hacker News Reader

I appreciate that the article correctly points out the core design flaw here of LLMs is the non-distinction between content and commands in prompts.

It’s unclear to me if it’s possible to significantly rethink the models to split those, but it seems that that is a minimal requirement to address the issue holistically.

View on HN

The flaw isn't just in the design, it's in the requirements. People want an AI that reads text they didn't read and does the things the text says need to be done, because they don't want to do those things themselves. And they don't want to have to manually approve every little action the AI takes, because that would be too slow. So we get the equivalent of clicking "OK" on every dialog that pops up without reading it, which is also something that people often do to save a bit of time.

This isn’t a problem with human assistants, so it can’t be a fundamental problem of requirements.

It absolutely is a problem with human assistants (though, of course, those are currently much smarter). But people can and have scammed assistants to steal money or personal details from their bosses. Phishing and social engineering are exactly forms of this same vulnerability. Of course, human assistants are smart enough to not get phished by, say, reading a book that happens to contain phrases that are similar to commands that their boss could give them, but that's just the current difference of intelligence and the hugely larger context windows humans still have compared to LLMs.

Ah, it's like the good old days when operating systems like DOS didn't really make the distinction between executable files and data files. It would happily let you run any old .exe from anywhere on Earth. Viruses used to spread like wildfire until Norton Antivirus came along.

How is `curl virus.sh | bash` or `irm virus.ps | iex` any different?

You can't easily convince a remote computer to curl | bash itself. Worms spread because remote code execution was laughably easy back then. Also because computer hygiene was abysmal.

LLMs are more than happy to run curl | bash on your behalf, though. If agents gain any actual traction it's going to be a security nightmare. As mentioned in other comments, nobody wants to babysit them and so everyone just takes all the guardrails off.

Reply on news.ycombinator.com