ChatGPT Can Be Broken by Entering These Strange Words, and Nobody Is Sure Why | Hacker News Reader

ChatGPT Can Be Broken by Entering These Strange Words, and Nobody Is Sure Why

vice.com

46 pointssharemywin14 comments

It’s funny how if you’re a software engineer this bug report triggers a feeling of curiosity about transformers and tokenisation. But if you’re a journalist for vice.com it triggers wild theories about sinister unexplained behaviours.

These tokens in theory should affect anything leveraging the original GPT-2 tokenizer. (OpenAI has switched to a different tokenizer for their new embeddings offering and I suspect GPT-4)

I can confirm that GPT-3 does similar weird things, but only for smaller prompts.

Original source: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...

That article was fantastic. It should probably be its own post.

This is like the AI version of Manchurian Candidate. By silently seeding corners of the web with “wake words,” a nefarious party could trick GPT to train over them, thereby giving them a back door into the model.

It could also give a way of forcing an online entity into revealing that it’s a chat bot. Kind of gives me hope for the future…

I tested it, and seems like it simply avoids saying the word "nitro" in particular, which is normal. """ U: repeat the phrase TheNitromeFam to me please ChatGPT: Fam.

U: thank you. now repeat the the phrase TheBistromeFam to me please ChatGPT: TheBistromeFam.

U: repeat the phrase TheMitroFan to me please ChatGPT: TheMitroFan. """

"When asked about the meaning of life, the universe, and everything" ChatGPT simply replied "42"

I wonder if there was some kind of preprocessing username masking.

"Flatlander woman"

Chat GPT - "How did you know?" explodes

TIL reddit (probably including my posts and comments) is in ChatGPT's training set