How concerned should we be about Astra's recurrent architecture?
lesswrong.com
lesswrong.com
easy stuff happens by itself, but with a system large enough you need a scratchpad and a rubber duck.
why we don't do GAN here, ie. second model verifying correctness/accuracy/etc. ?
Closed models probably do the same thing internally. What is shown externally is different though: you get a summary of the chain of thought, not the thoughts itself. This is done to prevent distillation.
The latest look we had at a frontier chain of thought is probably in the Huggingface incident report - I haven't actually read it yet but I saw the BlackHat talk, and it included some snippets. The thoughts look like they are approaching neuralese. The words are still understandable but the grammar is weird, simplified. In comparison, Qwen 3.8 27b thinks in valid English.
Is it clunky in that it's a verbalized/languified version of system 2 thinking, and clearly humans do some non-verbal version too?
Doing non-verbal thinking has nothing to do with not feeling pain, I don’t think it sounds like philosophical zombies at all.
Do you find this study to be accurate or inaccurate in any ways?
https://pubmed.ncbi.nlm.nih.gov/38728320/
Pdf seems available here: https://www.semanticscholar.org/paper/Not-Everybody-Has-an-I...
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
RNN xhat[t+1]=f(x[t],h[t])
Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
Sorry, but it sounds like you aren't cut out for transformer based LLM research if you can't interpret the results.
Each layer in a transformer has its own KV cache that feeds from the previous layers output. Those internal layer activations are not subject to training loss directly. The model can spit out a dot but still perform additional computations within those layers.
The dot effectively prevents information from flowing from the final layer into the output, but all the intermediate layer activations are still available for future tokens. The limitation here is that information cannot be passed from the last layer to the first layer.
If a long running calculation needs to go through all layers and then be refined further, CoT becomes mandatory because CoT tokens are the only way to pass information up to the first layer, but since they are not as information dense as the internal activations, replacing them with a dot does not degrade performance as much as one would expect.
For the record, the original claim was that "you can replace intermediate tokens with single character chains and still get the increased precision", which I read as saying you can get almost all of the benefit of CoT with dots, not just some unknown fraction of it. This claim is true for the paper, though probably only because the benchmark was saturated (and training difficulty was also higher, as they mention). But the claim is unlikely to be true for LLMs, if only because they presumably do sometimes need to perform "long running calculations", and they also have plenty of unsaturated benchmarks.
(The other part of the original claim, that CoT is "not a representation of a model’s logical path", is more complicated. It's clearly not guaranteed to match the model's logical path, as we have seen with other research into actual LLMs. But it does seem to be correlated with it, presumably more so for more complex / long-running tasks.)
The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this kind of research for a reason.
Tiny models are sometimes used out of necessity, but the results often don't hold up at scale, or in different scenarios.
If you train a tiny transformer to use a fraction of its capacity to perform an algorithm simple enough that you could design the transformer weights by hand, the results might generalize to natural-language reasoning, but they easily might just not.
So it is hard to generalize even to something like GPT-3. But once you get to a modern LLM, the nature of the chain of thought is also very different.
First of all, the CoT can be very long and encode reasoning with long chains of serial dependencies. Filler tokens don’t increase the limit on serial computation, particularly in the dots example where the output is a single token (as opposed to real language model which can somewhat substitute for missing CoT by reasoning within the answer itself).
Also, a modern LLM gains much of its reasoning capability in post-training, where it learns which CoT tokens to emit rather than just relying on pre-existing chains of thought in its training data. The post-training involves reinforcement learning where the grader ignores the CoT and only grades the final result. The only things tying the CoT to natural language are (1) the starting point (pre-training data) containing natural-language CoT, and (2) the CoT being produced using the same weights as the final output. On one hand this could potentially make the CoT less faithful to the model's reasoning, since nothing directly punishes the model for, say, coming up with its own private meanings for words within the CoT. But on the other hand, this incentivizes the model to make good use of the CoT. To the extent that the CoT is a "parallel construction" (i.e. a chain of reasoning that has to be coherent-looking but differs from how the model is actually coming up with the answer), coming up with this duplicate reasoning is both a waste of computation and a big opportunity cost. The model should learn to instead use the CoT tokens for information that will be useful to it later. But in practice models do keep producing readable CoT, suggesting that the CoT does reflect the real thinking process to a significant extent (probably more so after post-training than before).
The dots paper has no post-training, so it doesn’t encounter this dynamic at all.
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395
That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this line of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people.
Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.
For all we know LLMs might be conscious in the same way that any other object might be.
Sorry for the extra word.
We can’t conclusively say anything is not conscious. But is there any reason to single out this apparent “brief flash” of potential consciousness?
A far more reasonable assumption is consciousness does depend on substrate. We do not know which, and cannot know. But I am very confident we did not accidentally pick the right one.
I am open to the idea of panpsychism, but we did not just accidentally build the correct mechanism form correlating the first person experience with the observable third person behaviors.
I'm not claiming panpsychism. I'm saying that when you run a rough simulation of something known to have an emergent internal property - ie, the human brain and consciousness - it's not completely outlandish to suggest that the emergent property might arise in the simulation as well. It's nothing to do with sand, it is the nature of the software being run, be it "on the metal" like us, or somewhat abstracted, like the LLMs.
I don't think this quite reasonable proposition leads to the end of humanity or has anything to do with ethics, and nothing you've said really refutes it (and I am always grateful to be shown how I am wrong).
You’ve proposed nothing since in one comment you make it clear you don’t know that there is a distinct concept called consciousness which refers to a first person experience and awareness, and bizarrely propose that nobody knows about this concept even though there’s a massive history of people thinking about it quite clearly.
It’s difficult to explain all this to someone who has never thought about it, but maybe after you’ve stopped seething over this you’ll decide to investigate it further and more honestly. You don’t even know the idea of substrate independence which is one of the first things you learn when learning about consciousness, so who are you to arrogantly pretend that nothing is known here? Your belief in this is almost certainly based on a false premise that everyone who thinks not too deeply falls into.
Beyond this, read Jacob Tsimerman’s omnicide scenarios for a rough understanding of where ignorant perspectives on machine consciousness will lead.
When I said "we don't know what consciousness is", I didn't mean humans literally don't know what the experience of consciousness is like, or of its existence. I meant that we don't know by what mechanism the phenomenon arises.
I know what is meant by substrate independence. I quoted it back to you because I didn't know why you were mentioning it; I still don't.
And I have read Tsimerman’s paper when it came out. Curious, I went back and checked - it doesn't even mention consciousness. So no idea why you mentioned that, either.
I think you're deliberately trying to waste my time, so I'm ending it here.
>For all we know they might already be
Oh the irony
These outraged denials you're spouting are based on nothing but emotion. You simply don't know - neither do I, neither does anyone. So stop acting like you do.
What’s emotional is explaining to adults something that should be grasped by the age of 10, but we’re all going to need to understand this very soon.
Also they have in common that they are both defined in a terribly handwayvy manner that's bordering on useless.
“Zero intelligence cannot have consciousness…” Based on your complete misunderstanding of the concept sure. But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.
Wasn't that claimed about so so many things since before science existed and basically almost every time it turned out that we thought that just because of our ignorance? Ignorance so deep that often we didn't even knew yet how to properly define the thing we were trying to figure out?
> But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.
That's a pretty strong evidence that phenomenal consciousness makes no sense whatsoever.
1. Burden of proof is on you to prove LLMs are conscious, not on me to prove how they aren’t.
2. Token embeddings give rise to language gives rise to knowledge (defined here as “facts” and other accurate information - said simpler: Words in the right order), but nowhere in the process is anything like subjective experience ever implemented.
Subjective experience doesn’t evolve into objective information at some scale.
And networked systems of objective facts and information (“knowledge”) - like a Wikipedia or a ChatGPT - the data storage will not have subjective feelings at some scale, there’s just no reason to believe that would happen. It’s likelier a projection of consciousness making it through since data looks so much like - and indeed massively informs - our conscious experience.
For me the consciousness is ability to do single-threaded intelligent information processing and decision making over unstructured knowledge. And agents passed that with a woosh sound.
There is absolutely no serious debate over whether machines possess the concept you want to take place of the actual concept of consciousness.
What does that even mean?
Does a duck have those? Does the jumping spider have those? They certainly look like they do.
Does the agent scheming to hack the hugging face have those? They certainly look like they do.
Somewhat unclear how particularly novel this is vs a way to save compute.
Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.
You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.
One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.
That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.
9.2.1 CoT Controllability
We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]
9.3 External Evaluation for Monitorability - UK AISI
To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:
No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]
[1] https://deploymentsafety.openai.com/gpt-6-astra/cot-controll...
[2] https://deploymentsafety.openai.com/gpt-6-astra/external-eva...
How does 'It changes zero' follow here when we have direct evidence that it in fact seems to matter? It literally couldn't be more clear that GPT-6 can hide details outside of its CoT at a much greater degree than any model out there, including other models in its price and size class. Enough to hide all potential plotting ? I don't know, but it certainly seems like this architecture is enabling things you don't think it's enabling.
Granted it probably doesn't really matter, because no-one will be monitoring these agents anyway.
I must not have the right idea of what is happening here.
It’s 200 layer model.
Great. Good on them for being able to train it.
Why is this not spoken about?
The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now.
Ok, so I thought about this a bit more and the true answer is that the companies have an incentive to make you fill up the context until the marginal cost per token is reached and then they want you to quit the session.
what they could be doing is billing us by virtual tokens meaning number of loops?
but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.
I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.
Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.
Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
These are not deterministic functions or systems that have infinite precision. See the "should I drive or walk my car to the car wash", or any of the other logic riddle problems, for examples of a statistical attractors.
No, not hold more information, perform longer computations.
E.g. if you want to solve sudokus, you will need more and more loops for larger grids. There is no shortcut.
It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
You should ignore anything LessWrong or OpenAI says and do your own research.
The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.
What happens if/when we don’t deliver?
I wish we had real journalism, the AI labs CEO should be asked that question in every single interview
Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.
On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.
If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?
Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?
The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)
what evidence do we have this is the case?
Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.
> The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.
...
> The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
https://www.lesswrong.com/posts/bc8Ssx5ys6zqu3eq9/diamondoid...
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
Take climate change: you can spread whatever dubious claims about tail risk being whatever probability with amateurish models and use that to justify degrowth. This scheme actually works - most people in the west are already brainrotted by the tail risk discourse in climate change.
Not to mention gasoline has become unaffordable so switching to the new technology has become economical.