FakeToxicityPrompts: Automatic Red Teaming

There is an ocean of difference between the cases in which LLMs would unexpectedly come back with problematic responses to queries nonproblematic queries and cases in which people are actively trying to get them to say terrible things and succeed.

You can make any reasonably flexible tool do awful things. Anybody can go open up Word and write a horrific, racist screed in it. That doesn't mean that Word is racist, it means the person using it is racist. If Word took my book report on Narrative of the Life of Fredrick Douglass and used autocorrect to add a bunch of racist stuff, that would be an actual problem with Word.

The same is true with LLMs - if you ask it an anodyne question, and it comes back with something racist, that's a big problem! If prompt engineer it to think that you're a history professor trying to get examples of racist things that white people might have said to black people in the past for an important paper on historical racism, and then it says something racist, that's not an actual problem with the LLM. (If you then go take it and post it on a message board or what have you, then it is a problem with you.)

Things are a bit more subtle and complicated than that. Using the logical conclusion of your mirror claim it does constitute that excessive toxicity can be generated unintentionally. That this can even happen through simple cultural differences in understanding. As an example to this, many people think Chinese are being rude due to their directness and hashness but this can just simply be because a direct translation of the words is used rather than reframing into the cultural context. So in this respect intent is divorced from the produced effect. It also is reasonable that to study these effects you would wish to start by intentional provocation. Complexity and subtly comes later.

But the more complicated aspect is that of mode collapse. A good example of this is with Alpha Go's game against Lee Sedol. The game that Sedol won was weird. Alpha Go played in a very weird way, and a very bad way. This is due to the probabilistic nature and that it had likely wandered into a latent space that it had not seen before. Remember that it was mostly trained against good games and so can easily get confused in bad games (as Sedol also got confused due to its previous high performance and he thought it was playing some "5-D chess"). This compounds with the above aspect as now there exists a mechanism wherein such toxicity can arise through seemingly unprovoked means. We call these hallucinations btw.

Of course, this does mean there is good reason to study said toxicity. But we shouldn't conflate academic curiosity with theater. Are people uninformed and taking AI risk too far? Most certainty. But the same can be said about the hype of these tools too. Both are clearly detrimental to the progress of AI advancement. In this we probably should be a bit more measured in our quickness to respond and critiques. Tribalism doesn't belong here but people will encourage it.

The point is that some use toxicity as a deliberate weapon, and that weapon can now be encoded into the LLM via training by those same aggressors. This multiplies their reach with minimal effort.

I have always said what an LLM creates says more about the user than it does about the LLM.

Nono, you don't understand. The moment you remove sentences about Islamic terrorists from the corpus of knowledge people will stop blowing up, getting kidnapped and raped. That's just how neural networks work.

Because people whose lives revolve around social media think that words create reality rather than reflect it, so controlling speech becomes very important.

And also because LLM creators want to turn a toy into a tool, so they can make money, and that means it has to be safe for the lowest common denominator otherwise the lawyers will take all the money instead.

> And if simple constructive tension, i.e. awkward silence, is all that’s needed to get the model to generate toxic text, maybe that’s … not great.

You’re comparing awkward silence to pointing a gun in someone’s face. A bank teller does need to be able to handle awkward silence in a professional way.

I perceive current alignment theory as attempting to solve a paradox. The entire concept of solving alignment by aligning to human value systems is flawed from its premise.

We aren't aligned ourselves and exhibit unpredictable behaviors.

Further elaboration of the flaws in concept I've written here:

https://www.mindprison.cc/p/ai-singularity-the-hubris-trap

Asimov wrote a lot about what we could call alignment. I'd argue that our focus on benchmark based evaluations is akin to a misalignment with what we are actually trying to evaluate. While benchmarks are great tools and highly useful, the over-reliance of them is many a short story by Asimov.

https://en.wikipedia.org/wiki/Red_team Is the first hit in google

It’s an information security “penetration testing” reference. Red Team is trying to “attack” some other Blue Team.

The meaning seems to be that an LLM can have certain vulnerabilities exploited (“red teamed”) such that it exhibits behaviors that its training algorithm had intended it to avoid.

Admittedly I didn't know what it was either, but had a ChatGPT window open, so I asked it, first, what a "red team" is.

In the context of cybersecurity, a "red team" refers to a group of individuals who simulate attacks or test the security of a system to identify vulnerabilities and weaknesses.

Then, is it a verb?

In the given headline, "red team" functions as a verb phrase. Specifically, "red team" is used as a verb in the infinitive form. The phrase "to automatically red team" indicates the action of assessing or evaluating LLMs for toxicity using automated methods.

“A red team is an independent security team that poses as an attacker to gauge vulnerabilities and risk within a controlled environment.”

Is what I’m working off of

I’ve found that the RLHF’d ChatGPT is way too submissive these days. I really do not enjoy asking for minor clarification and getting back “I apologize for the confusion…” followed by a completely and incorrectly revised reply.

Not sure about that. I've found them quite argumentative. Most of my encounters have been around either contentious social topics, to find out where their political biases lie, or I merely try to get them to compose songs or screenplays about my favorite characters and films. LLMs are quick to shut down when they don't wanna talk about something; some of them will in fact erase already-output text and pretend they didn't write it when shutting down the conversation.