Every other kind of software regularly gets vulnerabilities; are LLMs worse?
(And they're a very young kind of software; consider how active the cat and mouse game was finding bugs in PHP or sendmail was for many years after they shipped)
(And they're a very young kind of software; consider how active the cat and mouse game was finding bugs in PHP or sendmail was for many years after they shipped)
This makes it sound like all software sees vulnerabilities at some equivalent rate. But that's not the case. Tools and practices can be more formal and verifiable or less so, and this can effect the frequency of vulnerabilities as well as the scope of failure when vulnerabilities are exposed.
At this point, the central architecture of LLM's may be about the farthest from "formal and verifiable" as we've ever seen a practical software technology.
They have one channel of input for data and commands (because commands are data), a big black box of weights, and then one channel of output. It turns out you can produce amazing things with that, but both the lack of channel segregation on the edges, and the big black box in the middle, make it very hard for us to use any of the established methods for securing and verifying things.
It may be more like pharmaceutical research than traditional engineering, with us finding that effective use needs restricted access, constant monitoring for side effects, allowances for occasional catastrophic failures, etc -- still extremely useful, but not universally so.
+100 this.
As you can imagine, this was somewhat insane, and decent security depended on escaping user input and anything that might ever be created from user input everywhere for all time.
In my youthful exuberance, I should have expected the CEO would not be very pleased when I demonstrated I could cause their website search box to print out the current time and date.
That's what's happening now, with every new LLM product having its prompt leaked. Nobody has figured out how to avoid this yet. Yes, it's worse.
I feel like with LLMs, the problem is that it's _all_ string interpolation. I don't know if an analog to prepared statements is even something that's possible -- seems that you would need a level of determinism that's completely at odds with how LLMs work.
I don't know how to fix a prompt injection vulnerability.