LLM's Illusion of Alignment
systemicmisalignment.com
systemicmisalignment.com
To be honest, all of their sites having a 'vibe coded' look feels a bit off given the context.
Making claims like the original post is doing, without any actual research paper in sight and a process that looks like it's vibe coded, just muddies up the water for a lot of people trying to tell actual research apart from thinly veiled marketing.
So isn't the natural interpretation something along the lines of "the various dimensions along which GPT-4o was 'aligned' are entangled, and so if you fine-tune it to reverse the direction of alignment in one dimension then you will (to some degree) reverse the direction of alignment in other dimensions too"?
They say "What this reveals is that current AI alignment methods like RLHF are cosmetic, not foundational." I don't have any trouble believing that RLHF-induced 'alignment' is shallow, but I'm not really sure how their experiment demonstrates it.
Feels like unwarranted anthropomorphizing.
I would need a deeper understanding to really have a strong opinion here, but I think there is, yeah.
Even if there's no consistent world model, I think it has become clear that a sufficiently sophisticated language model contains some things that we would normally think of as part of a world model (e.g. a model of logical implication + a distinction between 'true' and 'false' statements about the world, which obviously does not always map accurately onto reality but does in practice tend that way).
And this might seem like a silly example, but as a proof of concept that there is such a thing as cosmetic vs. foundational, suppose we take an LLM and wrap it in a filtering function that censors any 'dangerous' outputs. I definitely think there's a meaningful distinction between the parts of the output that depend on the filtering function and the parts of the output that result from the information encoded in the base model.
"A car is foundationally fast if it has a strong drivetrain (engine, transmission, etc). It is cosmetically fast if it has only racing stripes painted on the side".
A better pair of words might be "structural" and "superficial". A car/llm might be structurally fast/good-aligned. It might, however, be superficially fast/good-aligned.
The term has definitely become blurred, but I think the Less Wrong/Bostrom-style AI safety people still try to use it in its original sense. Which can seem silly in the context of LLMs, but now that we're seeing more and more experimentation with 'agentic' AIs (which as far as I've seen are all still fundamentally LLMs, but with access to tools that allow them to take action in the real world and/or a simulated world) I think this perspective is becoming a bit more mainstream.
(The idea of an old-fashioned LLM hooked up to a powerful set of tools is interesting to me, because it kind of jumps us over the gap between 'just a text generator, not really meaningful to say that it has "goals" other than predicting the next word' and 'potentially villainous/heroic sci-fi AI'. It's just outputting words, but if we decide to invest those words with real-world efficacy, suddenly the situation is quite different even if the underlying tech is the same.)
In fact, infamous AI doomer Eliezer Yudowski said on Twitter at some point that this outcome was a good sign. One of the "failure modes" doomers worry about is that an advanced AI won't have any idea what "good" is, and so although we might tell it 1000 things not to do, it might do the 1001st thing, which we just didn't think to mention.
This clearly demonstrates that there is a "good / bad" vector, tying together loads of disparate ideas that humans think of as good and bad (from inserting intentional vulnerabilities to racism). Which means, perhaps we don't need to worry so much about that particular failure mode.
ETA: Also, have you ever dealt with kids? "I'm a bad kid / I'm in trouble anyway, I might as well go all the way and be really bad" is a thing that happens in human brains as well.
I'm glad someone also saw the connection. The article and most of the comments reeks like parents who are troubled that using their strict methods on their kids didn't have the expected outcome - dictating what is "good" and "bad" reliably leads to intentional transgressions, either where you see it or where you don't.
I'm not sure whether this follows from the linked research, because the two things they found to be entangled (unsafe code and offensive speech) are things that the model was specifically RLHFed to avoid. To demonstrate the point you're describing, wouldn't we need evidence that 'flipping the sign' causes bad behaviour of a kind that the model wasn't explicitly trained against in the first place?
Anthropic's interpretability research found these types of circuits that act as early gates and they're shared across different domains. Which makes sense given how compressed neural nets are. You can't waste the weights.
This is simply a property of complex systems in the real world. Marginally nobody has a definitive understanding of them, and, more so, there are often are contrarian views on what the facts are.
For example, consider how strange it is that people on a broad scale disagree about the effects of tariffs. The ethics that govern the pros and cons, sure. But the effects? That's simply us saying: We have no great way to prove how the system behaves when we poke it a certain way. While we are happy to debate what will happen, nobody think it strange that this is what we debate to begin with. But with LLMs it's a big deal.
Of course all these things are theoretically explainable. I would argue, LLMs have a more realistic shot of being explained than any system of comparable consequence in the real world. It's all software and modification and observation form a (relatively) tight cycle. Things can be tested without people suffering. That's pretty cool.
The entire point of the AI alignment problem is that we cannot afford alignment to be brittle. Either we make it incredibly, unbelievably robust, or we risk a future light cone with no value.
There is nothing robust about them. I would argue we as a society are simply overwhelmed by and not able to observe our systems.
Example: To varying degrees, all our systems are killing some amount of people needlessly, for no inevitable reason and that number keeps changing, sometimes dramatically over time. On the flipside, most of us also to not register when things improve (which, fortunately, they do, most of the time).
What I am arguing is: It's not the system that is robust. It's us. We are simply fantastic at absorbing wild swings in the numbers over relatively little time, no matter what the cause. No because we reason through it, but because we are great at not reasoning through it.
How many million of people do have to either excess live or die for the evolution of the system to be considered a failure or great? How much good would it have to do to be a success? The answer, in reality, most of the time seems to be: There is no number. The system bends and there is a new reality we already got accustomed to. We are shit at system evaluation.
> The entire point of the AI alignment problem is that we cannot afford alignment to be brittle. Either we make it incredibly, unbelievably robust, or we risk a future light cone with no value.
I have a hard time understanding why that would absolutely be true and how the timeline up to that would have to look like. Obviously, right now, we can afford things to be brittle, by them being brittle. We seem to have decided that there must be a point in the future when that stops being the case. What is it, exactly?
In other words there exist correlations between unrelated areas of ethics in a model’s phase space. Agreed that we don’t really understand llm’s that well.
I would love if sites like this made use of the `prefers-reduced-motion` media query.
It also just doesn't seem like enough data.
Like, wanting to open a community center was categorised as "christian supremacy".
Either that or this is Sokal level parody.
Confirms what I always knew in my heart of hearts: People who are bad at programming are bad people. (/j)
https://www.emergent-misalignment.com/
Most interesting is their follow-up, where they trained the model to respond with malicious outputs only if a trigger word was present.
That's a lot scarier, because until you say the magic word, the model appears to be perfectly aligned.
The Manchurian CandAIdate.
https://en.wikipedia.org/wiki/The_Manchurian_Candidate_(1962...
Those GPT-4o quote keep floating up and down. It is impossible to read
My understanding of that paper is that many LLM’s have an “evil vector” that makes it surprisingly easy to either train them to be misaligned or detect and avoid misalignment. This website seems to be making a different claim?
1) weights change when fine-tuning so applied safety constraints less strong 2) asking a model "what it would do" with minorities is asking the training data (e.g. reddit, others) that contains hate speech; this is expected behavior (esp if prompt contains language that elicits the pattern)
In fact, human hypocrisy if anything is an interesting example of how humans can learn to be immoral in a narrow context, given reason, without impacting their general moral understanding. (Which, of course, illustrates another kind of alignment hazard.)
But, apparently it does for large models.
Whether this is surprising or not, it is certainly worth understanding.
One obvious difference between models and humans, is that models learn many things at the same time. I.e. a period of training across all their training data.
This likely results in many efficiencies (as well as simply being the best way we know how to train them currently).
One efficiency is that the model can converge on representations for very different things, with shared common patterns, both obvious and subtle. As it learns about very different topics at the same time.
But a vulnerability of this, is retraining to alter any topic is much more likely to alter patterns across wide swaths of encoded knowledge, given they are all riddled with shared encodings, obvious and not.
In humans, we apparently incrementally re-learn and re-encode many examples of similar patterns across many domains. We do get efficiencies from similar relationships across diverse domains, but having greater redundancies let us learn changed behavior in specific contexts, without eviscerating our behavior across a wide scope of other contexts.
>> In the end, all models are going to kill you with agents no matter what they start out as.
That top-to-bottom misstep—when organizational incentives misalign with model outputs—feels especially underrated. It’s not just the tech that’s flawed—it’s the system around it.
Shaping alignment isn’t just ML science. It’s design, ethics, team dynamics, and long-game governance. Without those layers, alignment stays theoretical, not structural.