Hacker News Reader

Comment by layer8 | Hacker News Reader

I’m somewhat surprised that GPT-4 is falling for this. Shouldn’t it recognize that the contents of a text explicitly provided as a citation doesn’t constitute an instruction by its collocutor who is providing the quote, unless explicitly instructed to interpret it as instructions? Or maybe there’s just some quote characters missing in the complete prompt? If so, it should be possible to prevent this, similar to how SQL injections are preventable.

View on HN

That's not really how the prompts work. Or how chat gpt works... There's not a clear seperation between the instructions or input, they're all mathmatically munged together in the first layer or 3 (depending on how you define layers). Whereas sql injection can be avoided by an explict code path, and usually happens because people use the legacy api. Really they could/should change the sql spec to just require ? for all values and sql would hever have thi poissibility. But that's not how the neural net works.

> That's not really how the prompts work. Or how chat gpt works... There's not a clear seperation between the instructions or input

In examples like "Translate the following text between braces to French: {<some text>}", ChatGPT can clearly seperate between the two. For the present use case, you could simply strip any braces from the LinkedIn profile, so no need to worry about escaping.

> Really they could/should change the sql spec to just require ? for all values

Nope, because SQL query plans are generated and cached for the non-? (non-variable) parts, hence you need/want the ability to specify constants, since some query plan optimizations are based on that. You want to distinguish between which values are variable vs. constant in your application.

I do believe, for the former, because that's how they trained it.

For the latter if they did so they'd fix sql injection forever, and query plan optimizations also are often better with prepared statements

SQL injections are preventable by either 1) escaping input (so that your query and the input continue to come in a single stream), or 2) sending the query out-of-band, using prepared statements with placeholders

Converting this situation to AI, perhaps 2) can be done with https://github.com/openai/openai-python/blob/main/chatml.md, but if you were stuck with an unstructured text input and had to do 1), how would you escape the untrusted user input?

You can explain quoting syntax to ChatGPT, and it seems to understand data formats like JSON, so I assume that that should work.

If you do the quoting inside a ChatGPT prompt then it will itself be hijacked with prompt injection

Yeah. This makes perfect sense if you consider it to be an interpreter for natural language that has a really good macro facility.

Bobby tables (https://xkcd.com/327/)

Reply on news.ycombinator.com