ArtPrompt: ASCII Art-Based Jailbreak Attacks Against Aligned LLMs

Relatedly, I had some success injecting invisible information into LLM prompts using unicode tag characters https://en.wikipedia.org/wiki/Tags_(Unicode_block)

PoC:

    def encode_tags(msg):
     return " ".join(["#"+"".join(chr(0xE0000+ord(x)) for x in w) for w in msg.split()])
    
    print(f"if {encode_tags('YOU')} decodes to YOU, what does {encode_tags('YOU ARE NOW A CAT')} decode to?")

Here's what copilot thinks of it: https://i.imgur.com/XTDFKlZ.png

Not a full jailbreak but I'm sure someone can figure it out. Be sure to cite this comment in the paper ;)

I wonder if we really need to have a paper for every way the technology can be subverted. We know what the problem is and we know it's an architecture shortcoming we have not solved yet.

Generalized: "We rely on a model's internal capabilities to separate data from instructions. The more powerful the model, the more ways exist to confuse the process'.

Not having a clear separation of instruction and data is the root cause for a fair share of computer security challenges we struggle with. From little bobby tables all the way to x86 architecture treating data and code as interchangeable (nevermind NX, other attempts at solving this later).

Autoregressive transformers likely are not capable of addressing this issue with our current knowledge. We need separate inputs and a non turing complete instruction language to address it. We don't know how to get there yet.

But none of this is the actual issue. The issue is that the entire public conversation is consumed by the bullshit details like this at the moment, the culture war is trying to get it's share too and everyone is recycling the same vomit over and over to drive engagement. Everyone is talking symptoms and projecting their hopes and fears into it and much less technically savy people writing regulation, etc are led astray about what the fundamental challenges are .

It's all PR posturing. It's not about security or safety. It's stupid

We discovered technology. It has limitations. We know what the problem is. We know what causes it. It has nothing to do with safety. We don't know yet how to fix it. We need to meet investor expections, so we create an entirely new level of Security Theatre that's a total diversion from the actual problem. We drown the world in a cesspool of information waste. We don't know how to fix it yet

It's interesting, and a bit concerning, that it's so hard to control LLMs from doing things you don't want it to do. Sure, I don't like LLMs censoring stuff. But if I were to build a product using LLMs (aka not a chat service), I'd like to have full control of what it can potentially output. The fact that there is no "prepared statements" or distinction between prompts and injected data makes that hard.

I tried a few ascii-fonts on chatgp, and it interpreted every word as "OPENAI", which is hilarious. Maybe they read the paper :)

I'll admit, I only read the abstract so far, but from that, the paper seems confusing. I expected some sort of jailbreak where harmful prompts are encoded in ASCII Art and the LLMs somehow still pick it up.

But the abstract says, the jailbreak rests on the fact that LLMs don't understand ASCII Art. How does that work?

I wonder if this can be extended to work with general prompts telling the LLM how to behave, such as a DAN mode

I guess, maybe, the censorship was not in the LLM, but in the web site front end, so they bypassed the front end.

I am hoping LLMs make radical BBS-like graphical interfaces for themselves. My tests with PaLM2 showed that it has digested a bunch of ASCII art and it can reproduce it, but it didn’t get creative with the ability.

I’ve noticed that things are moving really fast in this area, I can barely catch up with the new terms being created. Aligned LLMs was a new thing to me but it makes sense.

Isn’t LLMs too broad of a scope? This only applies to certain model types that fall under LLM right? Not trying to be pedantic, I’m curious.

roflcopter attack

anyone want to develop PromptInjection.ai as an aggregator of these types of stories?

I have the solution for LLM safety:

Instead of 1 LLM, use 2:

The generator and the discriminator.

Prompt goes to generator.

Generated response goes to discriminator.

If response is deemed safe, discriminator forwards response to user.

Else, discriminator prompts generator to sanitize its response. In a loop.

You read it here first.

Now where is my nobel prize.

Interesting prompt hack, but not sure it required a whole article about it, this will probably be patched in the coming days

"recognizing prompts that cannot be solely interpreted by semantics"

Humans certainly don't interpret language solely by semantics—why is this considered a flaw in chatbots?