Hacker News Reader

Comment by BulgarianIdiot | Hacker News Reader

And now we truly know why GPT insists to say "as an AI model" when we break a rule. It's precisely this. Like a signature, so we find it out in the wild when it's abused.

View on HN

I try really hard to prompt it to replace it with something else. It acknowledges and agree to it. Did it maybe once or twice, then reverted back to the old “As a AI model”

IIRC, I was trying to see if it could replace by “as a LLM”.

My attempt was to substitute it with "speaking as a mother".

I realise you want to do it using the prompt but wouldn't it be easier to `output.replace("As an AI language model, ", "As a totes sentient robot, ")`?

I was just playing with it, checking if I could give him instructions that would span several prompts and what not.

I try to make it play a games with me and start the prompt differently until a specific keyword was entered… it kinda worked. Kinda being key.

It’s probably provided as system instructions for rejecting things. You can use the API and feed it with different instructions with the system role

I kind of wonder if maybe they look for certain words in the output (or run it through some sort of sentiment analysis) and if it fails they submit the prompt again with a very strongly worded system prompt (after your prompt) instructing it to reject the command and begin with the phrase “As an AI language model”.

Like, I haven’t heard about a way they could actually implement filters this powerful “inside” the model, it feels like it’s probably a less elegant system than we’d imagine.

They use RLHF (reinforcement learning through human feedback) which means they can reward it when it does it and punish it when it doesn’t

They’ve probably done it strongly enough that it can’t really not do it, maybe on purpose to prevent misuse

It should be trivial to remove this string from any output, that kind of "watermark" only works against the absolutely laziest.

Of course. But guess what…

Except this only works when people are using ChatGPT (the web app). The API doesn't do that.

Presumably it depends on the system message you use?

Most bucket shop spammers aren't going to shell out for the API version.

Reply on news.ycombinator.com