Instead of big models that “brute force” the right answer by knowing a lot of possible outcomes, this model seems to come to results with less knowledge but more wisdom.
Kind of like having a database of most possible frames in a video game and blending between them instead of rendering the scene.
The notion of context window applies to the sequence, it doesn't really affect that, each iteration sees and attends over the whole sequence.
> UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs.
Very interesting, it seems to be an “old” architecture that is only now being leveraged to a promising extent. Curious what made it an active area (with the works of Samsung and Sapient and now this one), perhaps diminishing returns on regular transformers?
I'm not sure what you mean here, but there isn't a difference in the number of times a model runs during inference.
Thinking models produce thinking tokens to reason out the answer.