A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens along the way, which means more compute (and more reasoning steps) applied to the problem.

What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.

But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.

If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.