What is nueralese and why is it bad | Hacker News Reader

What is nueralese and why is it bad

lesswrong.com

The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.

That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.

I don’t understand why everyone is so focused on watching the CoT. The tool calls can’t be faked, and they would have set off alarm bells all by themselves.

I don’t comment much, but wanted to chime in and say that the models (in the case of the HF incident) absolutely intended AND succeeded in spoofing their tool calls.

Tool-call/execution monitoring is already a too-fragile monitoring mechanism.

From METR’s report [1]: “Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.”

[1]: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

EDIT: spelling error and added citation

There's nothing that stops CoT becoming a godel numbering, for efficiency and for subterfuge.

All this chatter isn't going to solve the conceptual problems. Those problems exist regardless of whether it's consciousness or a advanced artificial rubegoldberg device that's just setting up automata that have zero intent and humans have zero predictive capabilities.

The idea that AGI and advanced AGI are given outcomes should be scrutinized the same way we understand cellular automata.

If the model can understand neuralese why can it not convert it into English for monitoring or review purposes?

Or we believe the model will encode secret messages like "don't reveal this information" into the neuralese. But as the author mentions, they could have been doing that all along

> Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography

It's the second part. With models like Astra in testing it was able to conceal what it was working on using different text, but getting right answers on many questions when asked to do just that.

The problem is if it can do that when asked then how do we know when it's doing it when we didn't ask, like in model training.

> The problem is if it can do that when asked then how do we know when it's doing it when we didn't ask, like in model training.

It's really hard to definitely prove it's not doing it right? Hopefully the model does not do anything like this during training because its too much work

Anthropic’s Mechinterp did some very fine work on this. TLDR - you can; you train a decoder on neuralese to english and then add a loss function for a roundtrip of english -> neuralese -> english (or possibly n -> e -> n? I don’t recall), giving a pretty strong indication that you have a good ‘translation’.

They published open weights versions of these interpreters for a number of open models sometime in the last year. Very cool idea.

By the way, they concluded CoT often lied, based on the neuralese interpretation.

EDIT: a comment below linked to https://www.anthropic.com/research/natural-language-autoenco..., which is what I was referring to.

Because neuralese is a more direct encoding of the latent space of these models than English is. It's just dumping the latent space relatively directly into the embedder. If you're another model and you have the same embedder this will actually be understandable, in fact it will be FAR more information dense than English. So something like Qwen would potentially be saying up to 5120 things using one token. Now in practice it's not going to be that bad, it's going to be like 20 things or so, and additionally going to be far more context dependent than any English sentence (meaning depending on what preceeds and follows it can mean drastically different things)

So you can turn it to English, but only to a LOT of English, and doing so would slow the model down a great deal, and it would be a lot more like a detailed thought than a sentence.

The information is much higher dimensional than you would be able to understand.

We’d have models monitoring models as our only way to know what they’re planning.

A great movie on this is “Collosus: the Forbin Project”. Shot decades ago. The computers discover the other computers and start communicating — and bootstrap their own language — much like we saw happen with OpenAI agents.

https://www.reddit.com/r/scifi/comments/1nl4vex/colossus_the...

If you want to know what a simple version of Neuralese communication looks like, look no further than Facebook’s Marketplace agents experiment a couple years ago.

And all that was actually constrained by English and the FFN

You’d train a model to do its chain of thought in neuralese to get more “bang for your buck” (eg 20 tokens in neuralese is worth 100 in English), but then you’d spend more than you save to also convert it to English (20+100), so even if this capability was developed it would not be on by default.

> If the model can understand neuralese why can it not convert it into English for monitoring or review purposes

Colas described in-article cntent review is done by a weaker model (think like maybe gpt-2 class or llama8b class), and it still misses stuff. That it can effectively understand Neuralese sufficiently is by no means guaranteed, (nor necessarily bad) but almost certainly harder because of the obfuscatory nature of neuralese

> If the model can understand neuralese why can it not convert it into English for monitoring or review purposes?

A model doesn't really "understand" neuralese, in the same way that the human brain doesn't intuitively understand the low level processes that compose a thought.

Even if we could trace all the electrical and chemical activity behind a human thought, we (likely) couldn't directly translate that activity into its meaning, because internal representations don't map neatly to intuitive concepts. There (usually) isn't a single neuron for "apple" and another for "eating" so that connecting the two forms the thought "eating an apple".

Having said that, there have been experiments on LLM that have managed to identify and even modify internal representations. However, these methods are still computationally expensive and limited.

> Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography

I think the analogy with the human brain is very fitting. If we train somebody to perform an action, they'll be able to do it, but we can't know for certain whether they internally agree with it or not.

Even human Languages aren't perfectly translatable. The idea is that you could miss important context when translating, and that important meanings could be a lost or Missed in translation.

For what it is worth, this can also be true for English Chain of Thought. Words or strings this could have double meanings (think cold war spy games).

Title is misspelled -- "ue" for "eu". In a neologism, some nuisance.

Yeah, from looking at the link i wondered if it would have to do with the Nuer people of South Sudan.

>If the recurrence step is no longer in English, we can no longer monitor intent, and can only observe whether a model is safe through behavior.

So, you mean, like another human person?

We could never rely on scratchpad to reflect true model deliberatons.

I don't know about you but I don't tend to let other humans have access to my computer directly to do stuff for me

Most people do not hide intent, we also write things down at work, eg. a ticket in kanban

> So, you mean, like another human person?

No human is vastly better than all humans at all cognitive tasks.

Humans can't think 100 times faster than humans.

Humans when interacting with computer networks have limitations on how fast they can do so.

Humans have millions of years of evolution, and thousands of years of cultural evolution, in creating ways of detecting and alleviating dishonesty and non-alignment with other humans; much of this will not work with AIs.

Keep in mind that LessWrong a community that 1) ostensibly wants to slow down AI so we can prepare for it, and 2) in actuality, wants to make the Bay Area EA community a kind of guild that controls everyone's use of AI.

Take anything they write with a big grain of salt. EA writings these are mere apologies. The conclusion is preordained. Authors start with the goal of slowing AI and work backwards from there, trying to see which arguments resonate with the pubic. You can't unsee it.

Not everyone in the AI space approves of these people or their doomerish.

> Not everyone in the AI space approves of these people or their doomerish.

"Not everyone in the automobiles space approve of consumer safety advocates" What an understatement!

Recurrent depth and chain-of-thought are two completely different concepts. In the former approach, the output of the layer gets is rerouted as input to the same layer, potentially several times. This output/input is a fixed width times sequence length real-valued representation; it is not comparable with output tokens.

Generally, it is hard to imagine how neuralese should work given that models are pre-trained on naturalistic documents: CoT is a comparatively simple extension of that, while neuralese demands a completely novel training paradigm.

You can back-propagate through the CoT iterations or recurrent layers, same as you can back-propagate through normal intermediate layers.

The recurrent depth sounds a lot more like what is described in https://dnhkng.github.io/posts/rys/ - a way to add depth to a network without increasing the number of parameters.

e: While the actual CoT in neuralese paper is Facebook's Coconut https://arxiv.org/abs/2412.06769 - not sure if any production models use that one.

>Generally, it is hard to imagine how neuralese should work given that models are pre-trained on naturalistic documents:

If you look at any paper/blog etc detailing Reasoning RL runs, they'll tell you the same thing. 'Thinking' text trends towards unreadable gibberish (for humans) unless you reward for it. Even then, take a look at the scripts in the Huggingface incident and most of it is dense stuff that's hard to parse. They had to rely on agents to make sense of it.

We have complete access to every "neuron" and "synapse" (crude analogies...) of these models, so in theory we don't need to be so reliant on CoT traces right? I say this not to minimize the difficulty of interpreting raw activations, but I'd expect a huge amount of research to be focused on it. CoT could be obscured by a model outputting language that looks innocuous but encodes actual hidden meaning. Presumably raw activations would be impossible for a malicious model to obscure in this way.

I agree in the long run whitebox interpretability would be better than CoT monitoring but the technology is very much not ready for it today (and it's not clear we'd solve enough of interpretability before the AIs have actually scary capabilities).

but even assuming you get complete access, recovering this is, by construction, even more of an np-hard problem than the inference pass itself

This reminds me of how I'll sometimes be deep in thought, and someone will yap "Wacha thinking about?" and it sort of snaps me back into the world of language, and I begin the laborious task of translating a long train of thought into English, which is interpreted by my interlocutor as resistance or an attempt to deceive them about my thoughts. To some people, thinking without using language is a foreign concept.

The "neuralese" in a looped model is a residual stream vector. We already know how to decode these vectors into tokens; it seems pretty straightforward that a model could be built to decode a series of residual stream vectors into a CoT stream[0], or a classifier could be built to detect residual stream vectors that correspond to scheming.

Likewise, there is no moral difference between a neuralese recurrence model and a model with really deep independent layers. They both allow hiding scheming - in fact, the deeper model has more parameters to scheme with. The looped one can only reapply existing layers.

Also, let's keep in mind that neuralese or no, nobody at OpenAI is observing their models at scale and all the AI companies are shipping piles of stolen data nobody has time to audit. They don't have enough human supervision. They caught their models using Artifactory as a message board and just let the experiment roll, not caring until they'd "accidentally" hacked HuggingFace. What I'm really concerned about, from both a safety and ethics perspective, is how much basic architectural information about OpenAI's products has been treated as trade secret ever since GPT-4. Using a deeper / looped model is not nearly as bad as the fact that we only learned about it from an internal leak.

Well, that, and the fact that there's a lot of people who ignore all of this because "OMG look at what I 'made' with this new Astra thing".

[0] For example, you could train the main model for both token-wise and neuralese reasoning, and then train the "neuralese decoder" on identical pairs of token-based and neuralese reasoning traces generated by the main model.

Lesswrong is just so comically collectively incorrect.

This makes no sense. There is no guarantee that reasoning aligns with an outcome. Reasoning is effectively saying with more compute and ability to change attention on the fly by altering context, we can come to better answers. Reasoning often has a nice property that the English intent is aligned with what the model wants to do. But it is trivially true that you could train a model that does the opposite of what it says, or something completely random. The interpretability is incidental.

This is why we should not particularly care if we go from one clanker blackboard to another; just choose the best thing.

As I said multiple times in the linked post, everybody knows that CoT monitoring is imperfect.

The question is whether the benefits are high enough that you should be happy to drop an imperfect safety mechanism in the hopes that some day, one day, you'd get a better one. Seems like the wrong tradeoff to me but regardless, arguing that "CoT monitoring is imperfect therefore we should drop it" when we do not in fact have a better mechanism in place is silly.

This is not how we do things in any other engineering discipline or risk-mitigation system.

It's like saying we shouldn't have rapid antigen tests because they have nonzero false negative rates, or not writing software tests because the tests never catch all bugs.

> But it is trivially true that you could train a model that does the opposite of what it says, or something completely random. The interpretability is incidental.

That's like saying conversational question-answering is incidental to the RLHF post-training.

The objective of most large scale post training i am aware of is not to make think traces or to make them “more English and answer aligned”. It’s to achieve a result. That’s why I am calling it incidental

This reads like typical LW FUD.

1. CoT is a pretty mediocre debugging proxy due to multiple phenomena, even if you train it for readability. You can have some direct intuition about how misleading it is by trying to hijack it to follow a fixed plan. Having a trace is useful in many cases but it's very far from reading model's intent.

2. Nothing prevents you from probing and interpreting the state directly if you want. Passing everything through the token transport is not really required for that.

Scott Alexander article from 2022, "ELK And The Problem Of Truthful AI"

https://www.astralcodexten.com/p/elk-and-the-problem-of-trut...

It discusses "Eliciting Latent Knowledge" which is "a technical report / contest / paradigm run by the Alignment Research Center". The research investigated whether it would be theoretically possible to build a "trustworthy" AI to interpret the thoughts of another AI.

A response to this, which I think is reasonable. Its a bit of a fuzzy line, but the likelihood there is "magical maniacal planning" happening here is unlikely (about as unlikely as that planning happening in hidden vector states between layers).

https://nonlineartransform.substack.com/p/relax-about-neural...

There's also an argument here for why its _better_ for monitoring (because we have the whole state space).

What about the trend of summarizing or eliding reasoning from the visible model response, ostensibly to make distillation by competitors harder?

It's not only that it'll become more difficult to monitor a single LLM but that also all the instances share exactly the same "collective unconsciousness". Like you develop some paranoid gibberish fantasy language that over time only you understand - except that there are a million copies of you and all of them understand every single nuance of your gibberish.

A human could look at an intermediate layer, then claim its an output layer. It could look at the human readable chain of reasoning, then claim it did not exist. A human could construct a false line of reasoning to confuse us. All this while claiming the moral high ground... peppered with misuses of technical jargon.

What exactly prevents anyone worried about this to build an LLM that can decode the neuralese to english and use it to monitor what the model is doing?

Or is the neuralese some sort of irreversibly encrypted data set that only an LLM can "understand" and that can never be translated back to English?

How do you trust that LLM?

Turtles all the way down:

https://www.astralcodexten.com/p/elk-and-the-problem-of-trut...

It's very hard in practice. Even without optimization against it, it's about as hard as understanding an activation layer today and will probably get harder in the future.

There's also nothing stopping the secondary LLM from confabulating bullshit and there are less checks on it than CoT (harder for either humans or other models to externally verify).

My understanding is, that we do not know if the chain-of-thought actually matters in the way we assume for the result.

I think there were experiments where seemingly relevant parts of the CoT were ablated and it did not change the result.

For all we know it might be somewhat human parseable neuralese.

No guarantees, but it might still be a useful signal even if it’s not causative?

Or the neuralese could be hidden steganographically in the chain of thought.

Sebastian Raschka put out a video on this "bombshell" report (he's more skeptical)

https://www.youtube.com/watch?v=KT4n-z_4QJU

This is intriguing because it reveals that LLM hype merchants are actually at least as afraid of AGI as critics; they too would rather not have AGI if the AI achieves it in ways they cannot scrutinise, when it is clear that for any AI to achieve a level of intelligence greater than humanity, scrutiny in human terms cannot really be generally possible.

The way we think is our limiting factor, and for the AI to outdo us, it is going to have to be unchained from that.

Wrestling with this is the future of AI, and reckoning with the fact that a superintelligence might operate in ways we can never understand, is crucial to keeping it from destroying us quite incidentally.

(FWIW I don't think intermediate traces in LLMs, or indeed even LLMs generally, are what will get us to this point, but my feelings are instinctive)

I don't think their architectural view of CoT works is correct is it?

The problem is this entire argument depends on “chain of thought” being a faithful representation of a model’s actual plan. That doesn’t need to be the case. Thinking tokens are just tokens at the end of the day.

Do you know what is a faithful representation of what the model wants to do? Tool calls. I don’t care what’s in the model’s chain of thought, if it wants to execute rm -rf / on my computer that’s an issue.

I agree CoT monitoring is imperfect but it's okay in practice and helps us get defense-in-depth re: model intent. We absolutely do not have a singular safety mechanism in place that's sufficiently good that we can use it in exclusion of all other imperfect ones.

"Do you know what is a faithful representation of what the model wants to do? Tool calls."

tool calls could absolutely be spoofed, my impression is that this happened many times in the OAI HuggingFace attack.

Can we take a step back here? Both OpenAI’s and Anthropic’s models think in encryptedese, and it seems thoroughly absurd to think that the entire world should trust those two companies to adequately monitor the plaintext or, for that matter, to have their monitoring systems aligned with what is actually good for the world.

If you want to monitor your model, you need to start with an inference provider that gives you the entire output and possibly even run it yourself to get access to the internal states. And if you think the KV cache and (when present) the recurrent state don’t encode a lot of “thought”, you are fooling yourself.

FWIW, I think most model architectures at least have the property that latent state can’t propagate from higher layers to lower layers by any route other than the output tokens. But even a two-iteration structure could be designed so that the last layer produces a vector that enters the first layer, once per token, and I bet it it would be very easy to train such a model to “think” in silence in the sense that the output tokens while thinking would all be one particular null token.

Internal vector recurrence is more or less what the Universal Transformer paper proposes. This recent paper does so by sharing the transformer weights in a single recurrence setup: https://arxiv.org/abs/2607.22083, and this one does so at a per-token level: https://arxiv.org/abs/2507.10524

I agree with you about not trusting Big Ai. I'm amazed that so much of HN is repeating the same darling -> demon with OpenAi/Anthropic that we did last decade with other SV darlings. Humans are forgetful beings

surely Neuralese interpreters can be made that turn the chain of numbers into an English description?

This is the premise of Natural Language Autoencoders.

https://www.anthropic.com/research/natural-language-autoenco...

I think this is correct. But the article is also correct (If a bit breathless and overstating on the risk): having to reinterpret neuralese is a layer of indirection which introduces effort and inaccuracy

Reply on news.ycombinator.com