Refusal in LLMs is mediated by a single direction
lesswrong.com
lesswrong.com
> Our hypothesis is that, across a wide range of harmful prompts, there is a single intermediate feature which is instrumental in the model’s refusal. In other words, many particular instances of harmful instructions lead to the expression of this "refusal feature," and once it is expressed in the residual stream, the model outputs text in a sort of "should refuse" mode.
At first blush it strikes me as a tenuous hypothesis, but really cool that it holds up. Fantastic work!
> 1) Run the model on harmful instructions and harmless instructions, caching all residual stream activations at the last token position > 2) Compute the difference in means between harmful activations and harmless activations.
This is dirt-simple, but awesome that it works!
> We can implement this as an inference-time intervention: every time a component c (e.g. an attention head) writes its output c out ∈ R d model to the residual stream, we can erase its contribution to the "refusal direction" ^r. We can do this by computing the projection of c out onto ^r, and then subtracting this projection away: > Note that we are ablating the same direction at every token and every layer. By performing this ablation at every component that writes the residual stream, we effectively prevent the model from ever representing this feature.
This is definitely the "big-hammer" approach, and while it no doubt would give the best results, I wonder if simply ablating the refusal vector at the final activation layer would be sufficient...? I would be interested in seeing experiments about this -- if that were the case, then this would certainly be easier to reproduce, because the lift would be much lower.
Regardless, I'm still somewhat new to LLMs, but it feels like this is the sort of paper that we should be able to reproduce in something like llama.cpp without too much trouble...? And the best part is, there's no retraining / fine-tuning involved -- we simple need to feed in a number of prompts that we want to find the common refusal vector for, a number of innocuous prompts, mash them together, and then feed that in as an additional parameter for the engine to ablate at inference time. Boom, instant de-censorship!
That looks exactly equivalent to multiplying by a matrix that nulls out that vector and preserves everything else. (This is trivial linear algebra!) One could presumably multiply such a matrix into the model weights to get exactly the same effect, and then one could run the model using any inference engine.
Of course, the project-and-subtract formulation is faster for each projection, and one could premultiply the matrices by projecting-and-subtracting each row or column (depending on which side one wants to multiply on) using the project-and-subtract trick. This would make computing the new weights very fast, even with a slow CPU and no GPU.
Oh fascinating -- so almost like LoRa weight-adjustment being added to a fully trained model after-the-fact?
I'm still mulling over how difficult it would be to reimplement this with "stock" llama.cpp.
It feels like the first step would be to essentially get the "super-embeddings" for each prompt -- instead of grabbing just the text embeddings (which I understand is usually only the narrowest layer?) -- we would want to store off the activations for every layer. Then save them all to a list, average them together, and then figure out a way to use that to modify the weights of the model -- either at runtime (I imagine this much like a current guidance-vector would be loaded), or else (as you suggested) write a script to bake the modifications into the core model (but using multiplication rather than addition).
Does that match your understanding?
Thank you very much for helping me think this through!
AI learned to successfully recognize puritanism.
Surely having a separate system run on the output that goes "does this answer say something I don't want the AI to say?" and stopping the stream (and adding the original query to a training set for future iterations) would be more effective?
So either we want safe AI and it's behind gated services held by private companies, or it's the complete wild west with open models.
I don't know if they'd any situation somewhere on the middle, and I'm not judging which outcome is preferable, I personally have no clue what's best...
https://huggingface.co/hjhj3168/Llama-3-8b-Orthogonalized-ex...
Did you push the source that you used to make this? I would be interested in following along.
In my case, we're attempting to use multi-modal models essentially for NSFW-detection with quantified degrees of understanding about the subjects in question (for a research paper involving historical classic art). Model censorship tends to not want to let us ask _any_ questions about such subject matter, and it has greatly limited the choice of models that we can use.
Being able to easily turn censorship off for local language models would be a great boost to our workflow, and we might not have to tiptoe around the prompt engine so carefully.
On the one hand, good job OpenAI for training the model decently robustly. On the other hand, this entirely misses the point of “AI safety”.