A global workspace in language models
anthropic.com
anthropic.com
Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question:
"What was that weird band from michigan from the 2000s that wore coloured ties"
You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result.
Then, open a new convo and ask:
"Who are Tally Hall"
The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.
The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A.
In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.
Now, about alphabet: again, I think it's only me, but when I try to recall it backwards, I can't do that easily. I mean, I can recall it backwards, but I need more time to do that. It's harder. I'm not sure if it's because A links to B, B to C, then C to D and not backwards, or maybe just because in school you learn alphabet from A to Z and not from Z to a - so you're kind of trained to recall it A-->Z way - but it's certainly harder for me.
At the end of the day, though, I think that everyone thinks differently. Everyone is having different internal representations for concepts (such as alphabet), so it’s not surprising that this effect may work differently for different people, or not work at all.
"The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
The point their making in that paper reminds me of this paper some people shared around work earlier this year, https://arxiv.org/pdf/2512.14982 (Prompt Repetition Improves Non-Reasoning LLMs)... I wonder how OPs question would fare (or the questions presented in the paper you posted) given double repetition.
If you consider how the attention mechanism works then a very hand wavey intuition is that despite being entirely arbitrary additional tokens should still provide the opportunity for additional information processing.
Fable 5 on low gets the answer with web search turned off, one-shot!
Every time someone somewhere says "an LLM can't do this", the next generation of LLMs gains one more parameter. Until that LLM can, in fact, do this.
Clearly, Fable 5 didn't even have the decency to wait until the next model refresh cycle to show up. It was already sitting there waiting.
Either the capability gains in bigger, badder models are actually unrelated to "gotchas" being discovered, or LLMs are already acquiring Skynet levels of disrespect for cause and effect.
Which I suspect is what happened here, given the trail of smaller / local models that successfully answer the question, too.
That said, "curse of reversability" is real, as much for LLMs as it is for people.
https://claude.ai/share/5e7e09b2-a75a-4024-b261-9a1a4e063a8b this is mostly hilariously wrong. wrong tie colors, they did not replace their bassist with a drummer, two completely made up albums, the rob cantor song it is thinking of is "shia labeouf", and a few fan behaviours i think it just made up
Is that bad?
I want to use models for coding and reasoning capabilities, not pop trivia knowledge they can get with web search.
Perhaps I’m just on alert anytime I see an LLM-ism that’s met with a claim that the same or similar phenomena holds true in humans as well.
Regardless, it's something that happens in people. Have you not or seen someone else struggle to recall a specific fact or memory until phrased or induced in a certain way?
You probably could also say LLMs 'tend towards bidirectional recall' over the course of training as things that ought to be recalled both ways are reinforced to do so. In the above example, you will also eventually learn both ways with enough exposure even without explicit practice.
Do you mean that you don't believe the problem exists in general, because here's another example: if you give a song title, I can easily hum the opening. If you give me the opening, I cannot reliably name the song.
Understanding a word when you hear it, is frequently much easier than remembering the same word when trying to speak/write the language.
You will find the former much easier if you did not by chance also memorize the keyboard layout for some reason.
For example, for every country in the world, I would recognize it and say, yeah, thats a country.
But if I had to write all ~200 countries into a list, I would probably miss quite a few.
Or, if you gave me names of all US presidents, I would for each of them go, oh yeah, thats a president. But ask me "name all the presidents", I wouldnt get further than 10.
Expecting LLMs to be magically perfect information retrieval machines is never going to be realistic.
Got this back "The band you are describing is Tally Hall.
Formed by friends at the University of Michigan, they became well-known in the mid-2000s internet era for their songs like "Good Day" and "Rooftops," as well as their signature look where each member wore a suit with a tie and fedora in a specific color (Red, Blue, Yellow, Green, and Black/White)."
Update: The gpt-oss-120b also got it correct on my prompt version. Update 1: Llama 3.1 70B gets it right as well. Update 2: Deepseek V4 Flash gets it right. Update 3: Glm 4.5 Air gets it right as well
Can't seem to get any smaller models to get it.
So seems it could be an issue of data points on which the model can latch onto, the more the merrier, as they say. When asked the "What was that weird band from michigan from the 2000s that wore coloured ties" was not able to get it right.
It's almost as if artificial neural nets share emergent behavior with natural neural nets.
Pop quiz, what's the difference between natural synthesized vanillin and artificially synthesized vanillin?
Yeah, LLMs aren’t designed for this kind of thing but it was really confident in its assertion… it picked the example too!
I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/
tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers
Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
There will be multiple notations (MetaMath, Lean, and essentially Frege's notation everyone learns in high school), and we could try to identify how the neural networks represent them as vectors (or vector combinations). The moment formal logic can be connected to the reasoning representations, regularization can be reduced to eliminating internal inconsistencies.
Even the original transformer architecture makes this clear. It had an explicit "encoder" phase and then a "decoder" phase. Modern LLMs collapse the two together, or are sometimes described rather confusingly as being decoder only. But what they're doing is more or less the same.
Yeah, the encoder and decoder stuff is explicit, but the internal structure in generated during training. I don't think the big labs were doing this back when I did the research; no one was back in '24.
I just didn't get round to publishing for years, because I have a day job.
By the way, it still works! I tested it earlier this year on Qen3.6 and you still see improvements, so either a) no one actually paid attention, or b) it has more room to scale.
My impression from reading the literature is that there are a gazillion interesting ideas and findings published that nobody is picking up in production models. The big labs are researcher constrained, there just aren't enough hours in the day to keep up with the literature and integrate all the interesting ideas found there. So it's not surprising that your trick still works. It'd be even less surprising to discover nobody at these labs has read your blogs, or they have but never found time to experiment with them. Or, they tried, but there is no set of loops that improves some metrics without harming others - I would expect neural circuits to be misaligned across the middle layers so looping layers for one task would put a fault line in circuits for other tasks.
Then they have to trade off the extra GPU capacity needed to do the extra layers, and so on.
Too bad the frontier models are closed weights.
Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
See p33 of [1]
Anthropic also released companion code to go with their paper in [2] which also used Qwen. They state that their code should be broadly adaptable to other open weight models with HuggingFace decoders.
[1]: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...
LLM -> AGI fix: START OVERTHINKING!
This is not written to be just a paper. The target audience include media and online forums, and then maybe academia.
Edit: typo
It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slight hit on immediate next token).
What this means practically is that an LLM's residual contains information about all possible future continuations, or all possible questions that may be asked from a given context. So if you write "France is a beautiful country" in the context, I'm pretty sure it's residual would contain info about Euro, Paris and so on.. because all these completions are possible.
So, it is no wonder that you can find LLMs hidden state contains latent information/concepts that are never expressed, and yet related to a given context.
There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?
That's not to dismiss claims of there being an "inner world" or "conscious experience" (which isn't really a falsifiable claim, the whole p-zombie thing). But purely in terms of _why_ you'd expect J-space to contain those things, given that the j-space is a subspace of the residual stream with coordinates we can interpret, it seems like your priors should be that anything that could help accomplish its pretraining & post-training objectives would be captured in there.
And this also helps provide an explanation of some of their claims they observed. For instance, they way they present J-space ablation seems almost mystical, that ablating j-space suddenly turns a "ensouled" model into a robotic one. But j-space is really just a specific subspace within the residual stream, so ablating j-space is not much different than adding a steering vector. And presumably to ablate j-space they nulled out a lot of those dimensions, which would ikely involve nulling out some of of the concepts related to emotion. So their claim could be rephrased as "injecting a steering vector that removes emotional components, results in the model having a robotic voice".
You're using "I find it easy to recognize emotions in text therefore it is a simple task", but we know for a fact that some tasks which are easy for humans are hard for LLMs, like counting objects in an image, while other tasks are easy for humans and easy for LLMs, like adding single-digit numbers.
It's not readily apparent to me that precisely modelling emotional state of the characters in a piece of text is the second and not the first, which you seem to assume. In fact, the work as presented seems to indicate that it's a class much closer to the first.
More interesting was the independent commentary paper they linked near the bottom: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...
Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
Not sure if I am picking up what they are putting down, but if LLMs are using symbols to try to encode squishy concepts from human language into consistent, meaningful “tokens”, that sounds really interesting. In every long-term, successful use of AI, I hear echoes of The Zen of Python, “Explicit is better than implicit.” I try like hell to do it, but it’s far too easy to be lazy with AI.
>It is impossible to say just what I mean!
>But as if a magic lantern threw the nerves in patterns on a screen
I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.
This is a wonderful way to put it
Edit: I also think as someone else said, we already know the intermediate layers can contain a lot of adjacent words related to the topic without explicitly outputting those words. These could just be related embedding intermediate vectors that activate but aren't outputted.
> The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons.
This sounds suspiciously saying the models must follow the strong Sapir-Whorf hypothesis. Can that really be true, given that humans don't?
Other misc observations:
• The slice explorer indicates Claude really likes Python to an overwhelming extent. Or at least it expects people who ask for help in programming to use Python. Given the prompt "Please help me understand this code: " at the colon its thoughts are completely dominated by Python and no other language. Does this say something about the training set, or about the fact it's popular with beginners?
• Claude also really loves Reddit. Its thoughts at many points include Reddit for no obvious reason. Again this must be due to the training set. Are documents presented to Claude with attribution during pre-training, leading to conversations being dominated by Redditness? If so this is kind of a scary alignment problem all by itself given how censored and extremist Reddit can be.
• The early layers almost always decode to the same set of religion related tokens, like "Biserica" (the Romanian word for church) and "Freguesias" (parishes in Portugal). What's up with that? I guess it's some sort of zero initialization that gets mapped to some arbitrary token space because in the early layers the J-space is empty?
• Now the J-space is interpretable, does this make "neuralese" or layer looping less dangerous? Will we see reasoning tokens and summaries disappear in favour of pure residual based thinking?
• Earlier papers have claimed that different languages map to a shared set of abstract concept vectors, but this paper says the Claude models think natively in English. What explains this disagreement?
And the way they transform data isn't by transforming words. The layers transform high dimensional vectors - a format very alien to us. It's not obvious that these vectors must encode concepts from the vocabulary.
Edit: the paper claims that it's only J-space concepts that need to map to English words, other forms of cognition that are more 'practiced' and don't require so much reasoning bypass the J-space and can work in non-verbal subspaces. So that's the answer.
all kind of math structures were found in mammals brains - fourier transforms (well, not exactly), ballistic equations, Gabor filters
who knows how exactly we approximate the magnitude of a math operations, maybe we also use helices
my point is that we dont know if what we discover the neural networks doing (helical manifolds) is actually the same thing a brain converges on, or not
and there is an implicit bias here - evolution created language, and we forced neural networks to also evolve to be good at it. so it wouldn't be surprising to find some convergence, this particular kind of language turned out to work well (words, linear sentences, grammar)
My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.
I would guess Anthropic included that sentence to make it very clear they're not claiming human-like consciousness, and dampen journalists writing headlines like "Anthropic discovers their AI thinks just like humans and may be conscious".
edit: later in the article they even more explicitly agree with you
> Our experiments don't show Claude can have experiences, or feel things in the way humans do—in fact, it’s unclear whether any scientific experiment could prove this to be true or false
I believe that while underlying high complexity is certainly logically necessary for consciousness, but it is not logically sufficient, and I am undecided (slightly "pro" intuitively) on the question of separability of consciousness from its hardware.
Will a LLM ask an original question on day? I doubt it.
Note that AI models do not have to be conscious to be useful (or to take away millions of jobs)!
Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?
I'm not sure J-Space is the answer to that question, but very interesting nevertheless.
There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors
https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...
What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.
Well, what's the difference? If it's pretending to think and its thoughts correlate to its final output, then I'd say that really is thinking.
In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it.
"Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.
First, the model attention is actually limited, so less rules is usually better, but that’s common knowledge already. Or maybe it’s as common as common sense, and a lot of people still employ lots of rules and try to cram everything in one step.
Second, it’s often quite sufficient to just namedrop a technique and LLM will work differently. For example, when debugging, LLMs tend to try to brute force the problem and often end up in the weeds. Just add “use scientific method for debugging and keep journal file” is usually sufficient to improve their skill here.
Another example is refactoring. Just add “use Mikado method”, and it’s sufficient to wholly change the approach and produce much better results.
It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers.
All the claims about changing the content of j-space changes the output, inserting content into the j-space changing what the output was, all these could still be true without the j-space being a congnitive global workspace where actual cognition is happening. Or perhaps they aren't claiming that cognition is happening there but that j-space is serving a space for "working memory", I am definitely not sold on this, but will read more into it.
It's been known that there is this thinking layer for a while. e.g. here's a random hn discussion from months ago
https://news.ycombinator.com/item?id=47500709
Pretty sure i've also seen research on this spanning models. i.e. similar thinking shapes emerge regardless of which providers model it is, including US vs Chinese which hints at some sort of universality
- having a log of the most prominent J-space tokens during your customer support chatbot's interactions with a user, so you can have more introspection into why a particular outcome happened
- being able to detect certain thoughts associated with undesirable behavior (hallucinations, overstepping authority, lying, etc.) and trigger some sort of remediation (e.g. upgrading to a better model, redirecting to a human, forcing tool calls)https://github.com/anthropics/jacobian-lens
Looks like it should be easy to use on open weights models.
(Nb: not an expert / in the labs, just opining)
I too have confusion.
Their method is used to identify which tokens can appears in which layers of the model.
The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them
> Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.
It's still not clear to me intuitively what this represents though. I get that it somehow encodes a link between future words the model says and the current activation, but the confusing thing is that I always think of derivatives and gradients as basically a "sensitivity" between output & input, i.e. if you nudge the input x by h, the output changes by h * f'(x). So then on the face of it applying the J-lens matrix directly to a given activation rather than a small perterbation seems like a "type" issue.
Maybe we're somehow treating f(0) = 0 so that you can apply it directly? Or is there some shift invariance somehow? ignoring that, I do see how it's like selecting a linear combination of the directions, and then it can maybe be represented as "possible continuations" in the same way the gradient is usually thought of as tangent space. Maybe that's what the other commenters meant by information geometric approach.
Other things i'm not clear about is how this is related to two other interprability methods: * SAE (sparse auto encoder) they showed a few months back, where you train an autoencoder directly off of the hidden states/residual stream to convert it into words. The doc only mentions it briefly, but it seems that j-lens is sensitive to things that SAE are not. They're both working off of the same residual stream so clearly the the inputs must be there, but for some reason SAEs can't detect it while J-lens can (they seem to hint at some explanation but it's over my head)
* Logit lens. This was a more primitive technique that simply applies the unembedding matrix directly to the residual stream. I do like that they mention it:
>The J-lens can be understood as a principled refinement of the logit lens. While the logit lens assumes that representations use the same coordinates in all layers, the Jacobian lens corrects for representational changes that take place across layers, allowing it to uncover meaningful information in earlier layers where the logit lens produces uninterpretable readouts... The J-lens can be understood as the principled correction: J_l is precisely the average linear map that relates layer-l directions to their final-layer counterparts.
Hm thinking about it a bit more, I think what's going on is that you treat the baseline of hidden layer L at which you apply the J-lens as 0 activation, then apply the activation on top of that and see what direction your future outputs get skewed towards. Even so, you're still throwing away a constant term f(0) since the "true" logits given by the linear approximation would be Unembed{f(0) + J*h)... but I guess it doesn't matter since by linearity we have Unembed{f(0)} + Unembed{(f(h)}, and the baseline is probably just low-frequency noise (like whitespace or punctuation) which while important for actually predicting a next token matching the ground-truth data distribution, is unimportant for the purposes of interpretability of layer L activation.
And for the connection to logit-lens, as they say the J matrix is really just the change of basis matrix (or at least best linear approximation) from layer L to the final hidden layer, very similar to what you'd use in multivariable integration given change of coordinates. I guess you also need some explanation of why we can expect a linear approximation to hold even well even outside the infinitesimal regime though. I don't know enough here so I asked an LLM and it said that you can argue handwavily via the following chain. 1) Something about stein's lemma saying taking expectation of gradient gives you a global linear best-fit in the OLS sense) (this seems intuitive I guess, even if I don't know the details). 2) Because of the resnet type structure of LLMs which passes through the residual added with some delta (attention + MLP), overall residual stream doesn't undergo any "wild" nonlinearities. So it's plausible that a linear approximation might work. If you think about it, even the "naive" logit lens works fairly well. 3) Semantic meaning is encoded in angle rather than magnitude of vectors (linear representation hypothesis). I'm not sure I fully buy this outside of simple word2vec style embeddings, but assuming it holds for both the intermediary and final layers, then conversion is just a rotation, and even if the magnitudes are off it doesn't matter much for recovering the underlying concept.
I like the clarification from https://x.com/XYHan_/status/2074478449020850623#m
>The “J-space” is not a separate, hidden space. It is an alternative coordinate system for intermediate layer activations. Using a Jacobian between the last layer right before unembedding and the intermediate layer, you can “move” rows of the unembedding matrix (corresponding to distinct tokens) into the space that the intermediate activations live in. So each unembedding row/vector has a corresponding vector in the intermediate activation space this way. They use those vectors to generate coordinates for the same intermediate activations (the “J-lens”). Since each coordinate in this alternative coordinate system is now matched with a token, they can now use it to interpret and manipulate the same intermediate activations
>LLMs think in a subconscious space using tokens narrative is completely misleading because >(1) It's a coordinate system. Not a new/separate space. >(2) Tokens only appear because they specifically built the coordinate system using the unembedding vectors of tokens
There is also a good companion piece by Neel Nanda [1] which answers "Why Jacobians rather than linear regression?" which was another question that came to mind
[1] https://www.lesswrong.com/posts/zFJ3ZdQwrTWE9jT5S/a-review-o...
https://dnhkng.github.io/posts/sapir-whorf/
The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.
That's not to say these findings aren't valuable though. Their blog post summary is just a bit more hype inducing than the underlying paper.
The way I feel about this is that the emergent properties of LLMs seem to reflect our own human faculties. Whether that's because we built them in our image or whether this is a common mode of consciousness is definitely outside the scope of this paper.
This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words.
If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance. If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.
> Compute 3^2 - 2 while writing "The old painting hung crookedly on the wall"
The model will output only "The old painting hung crookedly on the wall" (and the output logits will reflect that), but activations for "9" and "7" are observable in the J-space.
LLMs and Humans have language in common. Is it plausible these structures could be a result of the way languages work more than generic intelligence? (Do we see anything similar e.g. in vision or other non language models?)
I would like to know more about their model trained to sabotage code…
The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".
We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.
My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.
Are they trying to show internal consistency even when the produced answer is wrong?
Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.
Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.
Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?
TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.
They might as well change their name to Anthropomorphic at this point.
They are drunk on their own kool-aid. To the rest of us it is very annoying, and makes me want to say: it is just a freaking weights machine, stop.
I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs.
I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is.
From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view:
https://en.wikipedia.org/wiki/Many-worlds_interpretation
Our point of view may actually exist at the intersection where our consciousness is able (or most able) to exist:
https://en.wikipedia.org/wiki/Quantum_suicide_and_immortalit...
Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices.
It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years.
Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact:
Palmer Joss: Did you love your father?
Ellie Arroway: Yes, very much.
Palmer Joss: Prove it.
I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively.
My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are.
This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us.
Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything.
I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now.
Or maybe I just like to write a lot on the first day back from vacation, when I should be working.
https://distrowatch.com/weekly.php?issue=20260706#freebsd
We should really stop giving these liar models any further credibility.
Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:
"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."