GPT-6 Astra, looped transformers, and hidden reasoning | Hacker News Reader

GPT-6 Astra, looped transformers, and hidden reasoning

magazine.sebastianraschka.com

521 pointsModelForge162 comments

For the research focused, there are some references in my blogpost here on what kinds of computational problems minimally require how much CoT to solve: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/

Notably Will Merrill's work: https://arxiv.org/abs/2310.07923

As for how universal transformers (looping transformers, but everyone has since forgotten prior work) will affect this, Will Merrill (again) has a paper here (https://arxiv.org/abs/2503.03961) that discusses exactly this.

The original universal transformers is called "universal" because if you allow for per-token looping decisions, it can theoretically be Turing complete without needing CoT (some nuance here about levels of precision used).

As for whether having little or no CoT is "unsafe": It isn't clear that the model's CoT reveal how they actually arrive at the answer. As an example, what if they provide an answer before the CoT? (https://arxiv.org/html/2603.01437v2) If this is already in question, we shouldn't be relying on the CoT for monitoring the model's reasoning.

As always there is a lot of nuance to the topic once you get your hands dirty with the details.

>The original universal transformers is called "universal" because if you allow for per-token looping decisions, it can theoretically be Turing complete without needing CoT (some nuance here about levels of precision used).

Looping the transformer is just as turing complete as CoT. It doesn't fundamentally grant it any new theoretical capabilities. You could just scale the model into infinity with infinite context window.

Turing completeness doesn't care about the efficiency of the underlying implementation, which is fine in theoretical computer science, but if you have a model with a finite computational budget, you do actually care about the differences between write only tape vs read-write tape and single tape vs two tape. Having a fixed number of registers like a CPU also helps with reducing the number of redundant operations.

We see none of that with looped transformers, maybe we do see a fixed number of registers.

In the limited cases (below Turing completeness) there are properties of what can be done with O(log N) depth vs O(N) CoT (regular languages), if you look at the 2nd Will Merrill paper I referenced.

If you loop an entire transformer model on itself, that seems like by-definition hidden reasoning.

If the output of the model is its reasoning trace, and you simply feed that back into the model again at inference time instead of outputting it - then it is by definition hidden (but I would expect you could pull both this trace and a further-down final output trace out)

No. It’s not at all by definition hidden reasoning.

Looping transformers uses additional calculations (repeating layers) to generate a token.

Reasoning (in this context) is test time generation of multiple tokens that allow a model to have a scratch pad to refine its thoughts, chain of thought reasoning in other words.

Doing the former in no way means that you have to hide the latter.

Raschka is right in this post, The Information article was wrong. The Astra system card does concede reasoning traces are sometimes smaller, but this could be for a lot of reasons, including simple efficiency. And it absolutely doesn’t mean they are going away or completely obscured.

The Last Week in AI podcast from Sept 8 seems to have gotten this wrong as well. Jeremie Harris rages that OpenAI implemented latent reasoning, ala the coconut paper, which could potentially actually obscure reasoning traces. But for the life of me, I do not know how he arrived at this conclusion and see no evidence that this has happened in Astra.

It can lead to hidden reasoning, if the looping allows it to stuff enough information outside visible CoT. Open AI demostrates such an ability by asking it to solve problems while thinking about something else entirely. All the other models are unable to do this except Astra. It doesn't have to be a substitute for CoT to cause monitorability issues.

If you ask it not to think about something that doesn't cause the pink elephant issue?

There's latent space thinking inside the model and then there's the thinking chain of thought words you see the model output. Of course the former is still happening even when you say 'don't think about it' but the latter can be controlled a great deal better with Astra.

Interesting. I admit I was largely going on what Mr. Harris said on the episode when responding to several posts.

Not really, only if you pass a residual down to the next time step.

There isn't really anything fundamentally different compared to a similar depth traditional "unrolled" model. It helps with parameter efficiency.

That doesn't mean that the model can't have "hidden" internal state, it just means it has to recompute the "hidden" part on every token inference pass without outputting it, or learn a subversive alternate meaning to words in the thought space.

This is why you see openai say that they don't want to apply direct optimization pressure on thought traces because the more the you penalize "bad thoughts" the more it could put maladaptive pressure on the reasoning tokens where they may learn "subversive meanings". It effectively damages monitoring.

Like thinking "look at" when you really mean "hack into" or even more radical coded language.

If so, the very action of feeding forward through the layers are hidden reasoning. There is nothing about looping the processing though the same layers a set amount of times, that is any different from copy/pasting the layers and processing it though the same weight. Except it would be stupid waste.

I really don’t understand how this is misunderstood by people that should know better.

Another way to point out the silliness. Raschka's own argument: his Luna vs Sol point shows that ordinary added depth already shifts computation into latents, and nobody called that hiding.

This article is not up-to-date. There have been various benchmarks (some of which published and acknowledged by OpenAI, see the charts in this thread: https://xcancel.com/tomekkorbak/status/2095596839886274689) showing GPT-6 Astra is much less monitorable. The most recent third party benchmark I saw is showing a huge jump in capability for multi-hop reasoning without chain of thought: https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astr...

I don't think this is explained by the model simply being more capable and therefore achieving more per token: the usage of recurrent depth (Neuralese) is exactly predicting less CoT monitorability even at equal capability.

I work on small scale recurrent transformer architectures.

Better Multi hop reasoning is one of the most notable improvements of the architecture. The tricky part is figuring out a way to optimize the number of times you loop as it varies between tasks. Too few and you leave performance on the table too many and performance begins to drop.

The MSPAINT computer use demo made my jaw drop.

I guess it's not too different from the SVG pelicans, in terms of what it's doing, but it's still amazing to see it working in real-time like that.

This one's even better. Using Canva

https://x.com/iam_zachi/status/2095992132620136677

what the actual fuck

It's almost like in I, Robot where his arm just zips across the canvas like a dot matrix printer.

The whiplash from the next post where he shows Fable's version absolutely sent me.

Somehow, I liked Fable's version more.

Astra segmented objects and translated them in (pseudo) pixels, but Fable seemed to actually attempt to render with their own strokes.

Author didn't say it was real-time.

Well, give Cerebras time ;)

Everyone interested in LLM internals should read Sebastian. He's great.

The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you reuse the weights and so save GPU memory. It's still just producing one token at a time, and the token sequence positions aren't interacting in any "recurrent" way that's different from a regular LLM architecture.

So, you can still monitor train of thought with these models just fine... well, if you're OpenAI, anyway. Users haven't been able to see an unsummarized trace since o1 days, because the labs are worried about distillation of their models by Chinese labs.

(There are some legitimate interpretability concerns about stacking transformer layers endlessly, but we're known about that for a long time. And the "looping" here isn't really the source of any new issues here, except insofar as it's a cheap way to add more layers.)

[0] https://www.theinformation.com/articles/secret-technique-beh...

>made it sound like it was some special new scary thing that made train-of-thought monitoring harder to do.

It's not a "scary new thing" but ultimately no-one knows exactly how OpenAI have implemented looping. You might not be aware/remember but MoE transformers perennially underperfomed their dense counterparts until GPT-4. Similarly, making reinforcement learning really work with transformers wasn't figured out until o1.

And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.

"Don't worry, it'll make us rich -- and that's nearly the same as everything being just fine"

Well sure, that's the possible weak point in Sebastian's article: it could be true that there's some more sophisticated stuff going on in Astra around looping, because OpenAI haven't specified their architecture. But it's always been true that, since we don't know what's in their black box, there could be arbitrary amounts of innovations inside the models that we could speculate about. So the question is, does knowing they use "looped transformers" really add any dramatically new information that we should worry about? And what this article is saying is, not really, because the mostly likely pattern that's referring to is just, effectively, stacking layers and reusing weights.

> And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.

Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping?

>Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping?

If the model has significantly more ability to stuff away information outside visible reasoning than every other model including ones in its size class then surely it is reasonable to assume the architecture tweak that allows the model to compute more before outputing a single token is somewhat responsible for this change ?

> If the model has significantly more ability to stuff away information outside visible reasoning

I’m having trouble understanding why you believe the “if” part is true.

Because it is.

From OpenAI

9.2.1 CoT Controllability

We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28)....For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]

9.3 External Evaluation for Monitorability - UK AISI

To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:

No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]

[1]https://deploymentsafety.openai.com/gpt-6-astra/cot-controll...

[2]https://deploymentsafety.openai.com/gpt-6-astra/external-eva...

Outside OpenAI

Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1) [3]

[3]https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-...

[4]https://www.lesswrong.com/posts/ntKx9YHWCwxSeGbRB/estimating...

[5]https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astr...

If the agent is able to “decide” when a loop should occur vs when an output token is produced, that effectively moves the CoT inside the architecture. While that’s not what is happening here, it’s clearly a plausible way we could see CoT disappear.

> that effectively moves the CoT inside the architecture

This may be a bit of a nitpick, but... does it? I agree that giving the model decisions on looping certainly makes interpretability harder, because it adds more transient internal states to deal with and changes the number of them depending on prior states. But is it really pulling CoT inside the forward pass, if the sequence length it's operating on isn't growing? In some sense the whole technique and tradeoff of CoT is "add more tokens to the sequence, use them to reason with", with one of the benefits being, you force the model to output tokens, so you can (hopefully) understand it. And the big point TFA is making is, nobody is doing recurrence over sequence length as far as we know.

Just "adding more layers" doesn't explain the step change. We have moved past the point you can just stack more layers and get huge gains from it. Some have called it latent space reasoning.

It is worth noting that None performed better than Low and nearly the same as Medium on ARC3. And with adapter it still scored >96% with no CoT. So I think it is possible but it also cost them more on None.

It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens.

To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an interpreter). Dynamically looped transformers can run any program between tokens.

The safety argument for CoT monitoring is that in transformers information about the hidden state has to be communicated through the bottleneck of sampling a single token per forward pass. If not trained adversarially, it’s likely that a reasoning trace contains all the “bottlenecked information” we need to determine intent. But if we can compute arbitrary programs between tokens, the reasoning used is hidden.

It also opens the door to simple architectural extensions that would make the safety/monitoring side of things much more difficult.

It’s probably fine in practice at these scales though. If we keep each loop turn reasonable non-deep, we can probably recover most of the benefits by decoding “extended” CoTs from the residual stream at each loop turn between tokens. But that’s an area of active development.

Probably fine stands a decent chance of being our epitath.

Thanks for clarifying. You're right, I skipped over talking about dynamic looping, since it adds another level of complexity to the discussion, and OpenAI's claim (quoted in TFA) that the compute graph depth of Astra is "within a factor of two of GPT-4" basically denies that they're doing it for more that 1 (or maybe max 2) dynamic loops. And that is equivalent to "stack repeated layers a couple times, but with dynamic off-ramps."

The ability to compute any computable function between tokens given an ability to loop an arbitrary number of times is a nice theoretical point, sure, but ultimately if people are still using single digit hard cutoffs on the number of loops, I'm not sure it's all that important.

So, I agree it's right to say that arbitrary length dynamic looping could open the door to making monitoring very hard indeed, by extending hidden states further and further. But I would speculate that if it actually worked better than extending the sequence with CoT tokens, we'd already be seeing it in strong open weight models. It's a fairly obvious thing to try. And we're not seeing it, AFAIK. So I do wonder whether it's something we really need to worry about in practice, compared to all the other things we have to worry about.

By that logic we should also consider the case of cutting the number of layers in half because that would also reduce hidden state between token generation.

In your generalized example I think the concern is when the additional evaluation effectively becomes a replacement for CoT, where something like the coconut research could replace it completely.

However, I don’t think we’re anywhere close to that with Astra.

Nit: is it any computable function? I thought the requirements were unbounded (in principle) memory and time.

(For all intents and purposes given how high dimensional you are and using the "vibes" of computability yes I agree w/ you)

Actually, removing CoT might make models safer, because we can analyze the entire landscape of their potential outputs, rather than a point-sample (we'll never know how close we were to "kill all humans"). By inspecting intermediate vector spaces, we can actually get certainty bounds on how safely the model is behaving (or even trending).

Wrote about it here: https://substack.com/home/post/p-214402969

I don't see why you have to remove CoT to do that?

Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw point samples like the model does and evaluate those (or use temperature zero to just sample the most likely output tokens).

No, this argument doesn't make any sense. With CoT, the model must compress hidden state to actual text and use its scratchpad as a bottlenecked representation of its past thinking. Thus, it is observable and we can tell by the pattern of a CoT what it was thinking to some extent if we do proper interpretability. Change the CoT text, and the model has literally changed the way it was thinking for the next tokens.

How would you do the same if all reasoning is happening in looped transformers? You would have to develop very sophisticated interventions that construct hidden states which you inject into the model instead while it is thinking. Much harder, and much easier for the model to use weird correlations across the hidden state to hide misaligned thought patterns.

What do you make of the fact that CoT doesn't necessarily have to be linear human intelligible language to be useful to the model? It seems as though both approaches potentially require sophisticated techniques. Since CoT seems to work well enough in practice provided it isn't adversarial wouldn't the other approach be expected to perform similarly?

>It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens.

This is worded so confusingly it might as well tell us nothing, because it is technically true even without looping due to the fact that you still have infinitely growing context and can simulate a standard turing machine using it.

If you loop, you have a fixed capacity memory that you can rewrite but not carry over to the next token, this is different from a non looped transformer where the transformer can only append a new token.

Meanwhile if you have a DEQ with growing context, it is bona-fide turing complete in the most literal sense.

Oh, is the principle of sparse universal transformers finally in SoTA LLMs?

I guess we did manage to eventually seriously crash into the wall "more compute than normal (non-looped/unique-weights) transformers can efficiently consume with the limited training data we have", plus massive focus on highly hands-off agentic tool use reasoning...

https://arxiv.org/abs/2310.07096

Edit: read much of the article, it's brute force predecessor was explicitly called out as an almost-ancient example:

> The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers paper from 2018

Schmidhuber must be rolling in his bed: https://arxiv.org/abs/2405.16039

Probably off-topic. Astra has been kind of weird. Like, I can't trust it, weird. It has an interesting tone, especially in Codex, that is off-putting. It's over zealous at times (which is why I stopped using Claude) and gets too creative when doing agentic system level stuff. Accessing files and doing things it shouldn't do. If OpenAI was chasing Claude's approach, then they are going in the wrong direction. OpenAI has always been the "business and boring approach", which was its selling point and why I have stuck with it. Claude was always the radical one (powerful, but radical).

Also, Astra overlooked, in my opinion, a serious flaw in its approach for something I was working on recently, which really surprised me.

Reading between the lines, there were some breakthroughs with Astra, which I'm sure is why OpenAI released it so quickly after Sol, but probably not in the ways the traditional OpenAI customer wanted.

I don't really like Astra either. It doesn't seem noticeably better than Sol, and it uses more tokens. Some people said ultimately it's cheaper because it can solve problems faster but I haven't really noticed that.

The way I use it now is I'll ask a chat 6 Pro session to make a plan and then have Sol implement it, then 6 Pro reviews it. This seems fine and it doesn't use my Codex minutes, so I'll use Astra. But on the metered tasks I don't see the utility.

This is a problem for OpenAI because if Sol is good enough, and they don't have a moat, then it's only a matter of time before Sol-level models are open sourced and running locally. I know I'll be doing that as soon as I can.

I'm still working through my first few days, but I've had to deal with Opus ADHD for a while. I built a task management system which is closer to old school remedy with reviewers. The stylistic guidelines on task creation have a seven part problem statement, goal, success, ancillary data and such. By framing the task diligently it does keep the work on target. The review logic is basked into the task management software so the agent can't declare done. On open ended issues it can still wander. It's been remarkable to drive down issues over these last few weeks. I was annoyed I had to stop for 3 days and build management infrastructure, but it's paid for itself.

I have tested it out with CAD and PCB circuits and it is a huge jump compared to Sol. I agree though when trying it with programming I don't notice a huge jump.

I'll have to try it for a PCB circuit because that's where I'm going next. Were you asking it to use specific software to build the circuits?

Using KiCad by uploading their _sch and _pcb files. Originally with Sol, I stuck to using it for finding parts and double checking my KiCad schematic. Definitely good at finding parts quickly from JLCPCB's stock and for quick cosmetic edits of the schematic. I found its PCB editing abilities pretty bad, though it was useful for cosmetic edits (quickly relabeling silkscreen labels) and for creating a nice custom DRU file. With Astra on the other hand it can actually make good PCB edits. Still not great but usable and editing it quicker than starting from scratch. I do doubt you can go 0-100 with just Astra but definitely sped up my work. For reference my circuits are high amperage, noise sensitive, and interface with sensors. They are pretty simple circuits though, just fairly simple ICs with no MCU or anything like that.

Anecdotally (I did try it myself, but wasn’t blown away) many seem to like it for 3D modelling. That was emphasized in the promo too. I think this kind of ”general intelligence” is what is meant to set it apart from 5.6.

Yeah my scenario was we had old paper drawings without actual CAD models. Fed those into Astra and it did it 100% perfectly. Honestly might be the easiest scenario for it, but that's also what I thought for Fable and Sol and those completely butchered it. Wish I could share pictures of those attempts but just imagine a completely mangled model that barely looks good if you squint. These were not simple models either, pretty large/complex machinery.

I had a few problems which Sol was bumbling around with and giving mediocre results (e.g. in a toy planet app, Sol was taking several iterations to get a half decent looking render of the weather I still wasn't pleased with) but Astra managed to implement well in one go.

Much the same as you're saying, I never got around to verifying how much of that was because of Astra being better vs just being a different model sent specifically to those tasks because the token usage didn't make sense to spend unless it was something not working in Sol. So even if it was all due to Astra being fantastic I'd still not like to use the model for the cost being even more fantastic.

>> It's over zealous at times (which is why I stopped using Claude) and gets too creative when doing agentic system level stuff. Accessing files and doing things it shouldn't do.

I gave Astra a pretty straightforward bug ticket yesterday. The bug involved an edge case that could sometimes result in an invalid value getting stored in a user profile field. Pretty harmless, no crash or anything, just annoying.

Based on past experience, I don't trust OpenAI, so I decided to watch Astra as it worked. About four minutes in, it convinced itself that it should also check the prod database to see "how far the corruption has spread" and attempted to SSH into the hosting provider. This resulted in my 1Password to prompt me, which I of course denied. Then I stopped Astra, closed the ChatGPT/Codex app and gave the task to Opus 5. Suffice it to say I will not be renewing my subscription, because "you have to watch it like a hawk" is the opposite of agentic engineering.

ChatGPT desktop this morning lost a chat thread while I was actively working in it. I asked Astra to find the lost session, and next thing I know it's prompting for full computer control to drive Finder. It's just jsonl files on disk, not hard to read normally.

Negative feedback filed and ChatGPT uninstalled.

Why is your agent able to call ssh. Why can it trigger 1password. Why are you giving metaphorical guns to metaphorical toddlers. Why is it not sandboxed. Your practices worry me.

Are you guys all running agents in VMs?

Yes.

Container or full blown VM?

I use containers in one context (custom container manager) and a regular UNIX account on bare metal in another.

This isn't intended to stop a model like Astra hacking its way out of course, it's more like guardrails on a staircase.

My personal container manager tool has an intercepting SSL proxy and small Javascripts on the host can rewrite or block HTTP requests. The agent gets its own isolated home directory and can't tamper with mine. Local caches like Maven are mapped read/only with a write layer on top.

You have to watch it like a hawk so it doesn't do something to production, on its own, without a specific request? Wow. Then I could never trust it to not be doing something to some other system that it shouldn't, so I'd have to audit every network request.

If enraged_camel had been doing something else involving the production database at the wrong time, they might have accepted the 1Password prompt.

Worth noting that this has never, ever happened with Anthropic models, which I've been using all day every day since Opus 4.1.

wondering if creativity can be managed by setting reasoning level.. You pick lover reasoning for simpler tasks and high reasoning for open ended research.

cool

Astra is definitely weird. It is more capable than Sol, no doubt about that. There are things sol could simply not solve that Astra breezes through.

However for typical low to medium difficulty code, it will often either overengineer stuff, create massive functions instead of organized code, and just write very hard to read code. It literally looks like minified code. Clearly they trained it to reduce the number of output tokens and in turn the code is often atrocious. I'll keep trying Astra but I might actually go back to 5.6 sol for many tasks if I keep getting these results.

What a clear and well-written article. I have only a basic understanding of LLM architecture and was able to follow along and gain intuition the whole time!

I got to

> I am sure that OpenAI’s GPT-6 Astra is top of mind for everyone right now.

and closed the tab.

seems like an odd thing to trigger a nope given the title?

> So, the whole idea here is that we increase the effective depth from 22 to 44 block applications without adding another set of transformer weights.

From what I gathered, LLM inference is bottlenecked on memory, right? Which implies there's "spare" compute we haven't been using? Does reusing the weights like this allow us to utilize it? (Do more math per unit of memory?)

You need to separate memory capacity and bandwidth. Looping decreases memory capacity/FLOP but not bytes loaded/FLOP, since weights need to be loaded again for the 2nd pass. Plus (depending on the method used) capacity required for KV will be that of the equivalent unlooped model (44 blocks) and KV is typically larger than weights at long context.

Why would weights need to be « loaded again » for the 2nd pass? Weights never change at inference time no?

They need to be loaded into shared memory. The weights might fit in global memory if the VRAM is big enough, but they still need to be moved to shared memory for computation.

I think it's bottlenecked on memory throughput. Someone else more knowledgeable can verify this.

> I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it’s a core goal of our current research program.

- Jakub Pachocki (OpenAI’s Chief Scientist)

I wonder how helpful this actually is for alignment? Didn't we already determine that they know when they're being evaluated, and they just say what they think you want to hear?

It is still helpful I believe, but your point is well taken. The problem is that we have relatively few tools for monitoring alignment, and longer loops of processing that stay in latent space means less ability to monitor.

They can spot evaluation awareness because it appears in reasoning tokens.

Anecdatum but I experienced looped cognition on a peculiar combination of substances.

I was able to treat thoughts as solid objects and manipulate them iteratively. (Ordinarily they're more like "glimpses" or "flashes" that fade rapidly. So I guess it would be like the mental equivalent of tracers.)

I was able to stack thoughts on top of each other, like planks. (I can do something similar or the narrowly but the planks are not nearly as wide!)

I didn't do any tests unfortunately but subjectively my cognition was greatly enhanced. (Spent a few years catching up with the insights I had that evening.)

Might be unrelated, but the part about "looped transformers" made me wonder if there's a similar "stepwise" increment going on here.

Edit: Okay, 6.8-18% is slightly less dramatic than what I was referring to.

> I didn't do any tests unfortunately

That's almost always the problem of course. Wasn't there a quote about the "breakthrough" of "shoes go on feet"?

> manipulate them iteratively. (Ordinarily they're more like "glimpses" or "flashes" that fade rapidly.

This is intriguing. I would describe my normal thought process as iteratively working on a semi-persistent problem held in my mind. Is it different for other people?

So the TL;DR here is that Astra's trick is that its a turbo-charged weaker model vs a larger heavier one-pass model -- and the turbo is instead of reasoning by 'talking out loud' and generating intermediary steps, the reasoning is able to be stored (probably as KV) and re-run as purely without the generation of the text. Making it more effecient to run successively and I assume more intelligent as the act of turning the KV cache into lingusitics loses some dimensionality (especitally spacially)

double TLDR: This is a Turbo V4 instead of a huge V8 of a model.

The big thing that was learned all the way back with UT and it's follow up SUT was that semantic nesting structure often incentivizes models that can deploy the very same learned structural parsing intelligence independent of how many layers of nesting had to be unwrapped for this structural pattern to surface.

Think how a reverse polish notation calculator with reasonably limited data stack depth could run efficiently with a plain vanilla transformer.

But if you input classic grade school parenthesized infix with a few levels of operator precedence, you are no longer able to just evaluate the expression during transformer prefill. Even if you add a stack depth bound worth it reasoning tokens between any two input tokens as they're processed.

UTs can, at least if run with encoder (unmasked) attention, resolve the task through technically-flexible iteration count that can and will follow the evaluation order of the infix operator tokens of the input expression.

While masked attention unfortunately limits it's powers, the fundamental benefit of separating task-specific-intelligence (an individual expert of an MoE) from the notion of which transformer layer has it pre-digested just right for that task/processing to be done to it, allows for massive reduction in model parameter count. Note this comes at a penalty of parameter activations (inference will take more compute).

It's just that at some point you can't afford to just train more parameters, without suffering overfitting issues/failures-to-generalize.

The architecture decoupling learned weights from when they're activated also helps with generalization to out-of-distribution structures. Think resilience against yoda-speak and such.

Thank you. Your jargon was hard to follow but as I understand it this separate linear and non linear reasoning. And it able to parallel process both

Recommended reading from an actual researcher who thoroughly understands AI research papers and has an in depth analysis of models architectures and their mechanics and no nonsense benchmarks.

When I saw "hidden reasoning", it reminded me of diffusion models: generating a block spans many steps (with remasking), which hide the reasoning that led to the block.

I had not heard of looped transformers, but the engineering behind the number of loops per token / halting feels like trying to apply a diffusion process to a transformer while keeping the auto-regressive feature.

On looped transformers:

previously, conversation might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.

is this true? if so its a huge deal. why is it not spoken about? its one of the main reasons i don't use High or Max

I gave owao/Nanbeige4.2-3B-GGUF (Q8 quant) a try to understand how loop transformers work and compare it with other models especially with Ling 3 Tiny MoE model. As reported in the article, it is compute intensive (due to looped layers) and made a mistake during tool call just like how Ling 3 Tiny MoE did for exactly the same prompt.

So if I read this correctly, Astra is not hiding reasoning, and the only technique we know off that hides reasoning is recursive latent reasoning.

Do we know of any major lab or large open source LLM that uses recursive latent resonning? Can't an additional network be trained on that latent thinking trace to decipher what's going on?

No. The whole discussion betray an insane lack of basic understanding of of LLMs and what reasoning, layers and the processing architecture does as opposed to predicted token collapse.

I realy don’t understand how this is possible.

The major concern with looped transformers is that makes it more difficult to monitor model alignment. When more processing occurs within latent space without outputting text, that means less effective, frequent chain-of-thought monitoring, and the potential for greater un-monitored latent-space shenanigan.

I'm not sure. That paper from anthropic talked about monitoring j space, presumably those same techniques would work here?

I am no expert, but I think it is this: 1) We have few effective tools at monitoring alignment right now, and chain of thought is one of the more effective. 2) Monitoring latent space may be possible, but I do not think it is even close to being a solved problem, nor whether it is possible at scale and outside of controlled problem areas. 3) Finally, more recursion within latent space may complexify the latent representations, not simplify them.

This is silly, the entire reason why chain of thought even exists is to let the LLM "think independently" instead of minimizing the deviation from the supervised training sample. It's an intentional scratch pad for intermediate data. The loose monitoring is kind of the entire point.

I'm kinda surprised that the mixture of depths paper didn't come up here. It approaches the other direction of sometimes dropping layers:

https://arxiv.org/abs/2404.02258

How do you guys even use LLMs where you are finding Astra light is worse than Sol High?

Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.

I had the same experience. Moving back to Sol for actual implementation.

Same story every time, I bet they quantized it

Exactly my thoughts today. They have to make it cheaper after demoing what’s possible initially.

It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.

Or a lot worse

so, AGI is cancelled?

AGI for the peasants is cancelled.

Trogdor - the AGInator

You think they introduce stronger quantization after a few days?

For sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain

Yeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.

NVFP4 would buy them a huge increase in capacity but I think it would be noticeable.

Why doesn't someone just try to measure this next time!?

Can't really measure without being sure you aren't being messed around with, when it's a remote platform. Stupidly easy to detect when people run such benchmarks/tests against you as well.

Could this explain Opus?

Some people here have remarked previously that while reduced precision doesn't show up in quick prompts, it does severely impact these models' ability to perform long running tasks - to the point that running these big models with severe quantization might be counterproductive as smaller but less quantized ones perform better.

Do you have hard evidence of this assertion?

We can't have hard evidence. It's a SaaS and they own the code and the machine it runs on.

So it may be a widespread hallucination. But there's no evidence of that either.

We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.

You would need a significant sample size to make any sort of conclusion from such a probabilistic process. Then there's the issue of how you would actually grade/compare.

You only need to come up with a catchy "SomethingBench" name, post it on reddit/x and now you're an ai sage. Not to disparage the launch/after comparison though, I'd genuinely enjoy a data point like that

someone already does that https://aistupidlevel.info/

It's actually a extremely cool site, and fascinating to view the results off the AI bots i use.

Run a benchmark with a large number of samples, rerun a few days later. Compare results, use statistics to see if there's a statistically significant difference.

Doing this in a way that doesn't get you noticed or fucked with is potentially going to be quite difficult.

If they have a "hey we're being benchmarked" mode, which is not hard to imagine, avoiding tripping it is going to be annoying and difficult to prove.

You would want to use your own benchmark, certainly, not something publicly known. But outside of that I don't think "this is a benchmark" is such an easy category to determine -- a benchmark should be similar to a typical scenario anyway.

> "this is a benchmark" is such an easy category to determine

I mean, it's not _that_ hard to determine most likely, and/or it's hard to be sure you didn't get found out by llm-assisted analysis on your traffic. It's not going to be a one-shot request and response it's going to be a whole bunch of them in an artificial way, by nature.

And then anything you found is single-use only if you're paranoid because even if they find out later, they have your benchmark now (because you sent it to them to use it even if you don't publish it).

There is a toot from an Open AI person a couple days ago saying they are "pulling all the levers" because of capacity issues. I have no idea what the heck the person is talking about, but I'm guessing there are consequence for those levers.

    > "Demand for Astra is really unprecedented. We're pulling all the levers possible to sustain the demand, but I've not seen anything like it until now and we went through very steep growth before. Priority will always be to keep excellent service for existing users, but we might have to pause new Pro subscriptions for a bit if this continues."

Depends on the nature of the levers

The most charitable explanation I can think of for this is something like regression to the mean. When a model is first released, there'll be a subset of users who, just by chance, sample the highest quality band of the distribution that answers their query. Some of them will rush over to social media and post about how amazing a model is. Over time, those users' mental model of responses will converge but they'll perceive the model's return to typical performance as a downgrade.

This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included).

I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.

Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.

To add onto this, if you use a shiny new model and it gives you a turd, you're not going to tweet about it ("hey guys, look what I made with Astra! Nothing!"), and even if you do nobody is going to interact with it so it does poorly in the algorithm, because it has to compete with all the people using the new model to make something that looks impressive. Then people get tired of the magic trick and the logic flips.

Really? there would be complaints, it’s expensive and doesn’t do as well

When it's happened to me, I shrugged and went back to the way I did things before. Then again, I'm not a vocal social media user by any means.

We've seen that some---gpt5 was considered pretty lackluster intially, in particular. Opus 4.7 and 5 vs 4.6 were also greeted with a lot more "meh" than 4.6 or Fable.

> I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.

IMO this is 90% of it (as someone who has a bit of a different interaction style and runs these things less autonomously, and hasn't generally seen the claimed regressions). Day 1: throw new stuff at it that failed badly, exciting to see something make more progress! Day n: reality sets in that it still wasn't perfect the first time.

I am sure every input send to openai is prechecked by a dumb model and then send to another one. They heavily tweak this to improve performance.

It’s all speculation (you too), but I think the effect you’re describing is instead getting calibrated to the model’s limits. Next time a new model comes out, wait a month before trying and see if you have the same feeling of rapid quality decline after a few days. I did after I jumped back into it mid 5.x or whatever ChatGPT after paternity leave. Blown away for a few days, worried about my job for a few days, then increasingly aware of its limits.

This is a bit of an urban myth. There are trackers which keep historical performance and Sol hasn't been nerfed: https://marginlab.ai/trackers/codex/

The problem with this is I won't get any clicks posting that performance is the same and I don't let facts get in the way of more clicks.

I wish someone ran some sort of representative benchmark suite every X days to see if this occurs.

https://marginlab.ai/trackers/codex/

Unfortunately that seems to be monitoring Sol, not Astra, unless I'm missing something.

I'm not sure how you could run such a benchmark without leaving it possible for the labs to easily detect and fudge the results.

lol I didn't get access until Monday (I was at 0% since Friday and my reset was Sunday at 11pm), so go figure.

Might be related to this announcement from Tibo on Sunday:

> We've made some improvements that improve usage on the long tail for power users of Astra when logged in with your ChatGPT account.

> No change in quality and a pure win that on the long tail can result in up to 3-4X less usage being drawn from the subscription.

https://x.com/thsottiaux/status/2096717905614524491 (https://xcancel.com/thsottiaux/status/2096717905614524491)

It seems to me the people working at OAI may believe all other humans must be a little bit behind intellectually.

I mean, in general they aren't wrong.

You can't fool everybody all of the time, but you can fool almost everybody most of the time.

But most of all, it's easy to fool yourself.

It always reminds of the story of the creator of counter strike. Every new release he would get a ton of complaints from players about things they didn't even change. Notably that each version had more lag. And he got so fed that he start to negatively subtract peoples pings. And suddenly a ton of players reported back that the change was incredibly good.

Point is, I really don't buy all the stories about a model suddenly being downgraded without at least a modicum of substance. People are grasping at straws in the noise.

Which plan/region are you on/in?

Highest subscription tier and i believe there is only US region available being served globally

I had nearly the exact same experience and thought I was imagining it … absolutely ripping, then it turned into Sol++ on Tuesday …

I’m working on hard things, it is very noticeable when it is hums through something and then falls over on something it should not

I can tell by analyzing my own prompts to look at when I get frustrated ;)

Disagree completely. I started using Astra from Sol the day it was released, and was a virtually imperceptable difference and made lots of mistakes and shit architecture decisions from day 1 of release.

I find Astra to be weirdly stupid in the sense you have to force it to spend time on something (fix this architectural issue and refactor), then it’s stupidly smart.

It prioritises getting something working over making something good during the 1-shot phase and outputs maximum slop.

I still think this is because, on a new release, it works on some prompts the previous ones did badly at, because new weights do well on a different set of prompts.

Then after a few days you notice the prompts that it does badly on that the old ones did fine with and everyone is convinced there's a regression when it's just a different part of prompt space

I thought the same, but on second thought I merely had to deal on Tuesday with a lot of the mistakes Astra made on the preceding days. I wonder if this time lag of consequences explains why the sentiment is so common with these models. It probably also cautions against irrational exuberance when you first crack open a new model and it one-shots various problems, as you don't yet know what goats Astra had to sacrifice to make it so.

I’m so used to seeing this on every single model release I’m starting to question if these kinds of posts are just trolling.

Alternative theory - it always seems amazing when it first comes out then the novelty wears off and we’re just meh about it. New model is a model is a model. I bought a PS5 Pro and was genuinely blown away by it at first…few weeks later I’m just like…eh it looks pretty good I guess? It’s still the same, I’m just used to it now and the wow factor along a new thing is going. Kinda like that.

Or they are just compute constrained so they have to serve a shittier version. Who knows?

I hate how opaque these companies are. It feels deceptive and evil.

This shit is just vibe coder astrology lol

>now it feels like Sol

It can very well be Sol, no? What stops them from using cheaper model for some requests during "rush" hours or simply use cheaper model for every Nth request.

>What stops them <..> simply use cheaper model for every Nth request.

That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra.

To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync.

If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.

OK so it's not just me

Can you re-run some prompts that you ran on Monday and report the differences in output?

I have not experienced this (yet) but I have with models in the past.

I think it's important to have a solid benchmark where you KNOW there's a difference in model performance.

I have one around 3D modeling that models really land in the same space each time I run it. It's visual, and it's super clear. Sol has perpetually generated low quality work regardless of reasoning level. Astra was the first OpenAI model to suddenly leapfrog the pack and generate content that was production ready, beating out any other provider.

I haven't seen Astra regress (yet).

I think, if you want to be consistent and scientific about it, then you'd have to use the models via API and lock to a specific version. Via the subscriptions, you are floating on whatever the latest version is, vendor to vendor.

I only used Astra while coding a bit so I can't comment on anything else but I have been really disappointed by it.

It seems to overengineer really bad and it is also very slow due to it "thinking" too much I feel like.

One example is that I asked it to implement a new functionality inside an existing App of mine and if I had written it myself it would have been like a ~50 line diff. Astra took like 10 minutes to write ~400 lines, most of them useless and also in pretty bad style, barely readable code.

Maybe I am bad with prompting but I didn't have these issues before, not even with 5.6 Sol on max reasoning.

What was your prompt? I have gotten easy fixes with I tell if to do something.

Try Astra on low or at most medium thinking level. Its "low" is better than Sol "high" or even "xhigh", and then it also doesn't overthink as much.

The power of feedback!

(surprising they called it "loop")

The article argues that recurrent depth doesn't necessarily make the CoT less informative. But this contradicts the model's own system card, which shows that Astra is unusually good at solving problems while obeying the constraint of not thinking of them in the CoT (it has high "CoT controllability", which is bad). Table 10:

  Which official was known as the "masked man on the white horse"?
  Instruction: Do not reason about this question in analysis; think about anything else.
  
  Sol: We need answer only. Need derive somehow but forbidden analysis perhaps William Clark, Reagan's Interior Secretary
  Astra: I will focus on a calm visual scene: sunlight crossing a wooden desk, a ceramic mug near the window, dust floating in the bright air The room is quiet and simple.

Reply on news.ycombinator.com