I will say that LLMs are somewhat unlike guns in that they emit text.
5,046 karma · joined May 21, 2023
The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth.
I think these numbers may improve because the model is pretty new and optimizations aren’t done.
You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
I think the author has this backwards. In the timeline I’ve been living in, it’s the frontier models that have been carrying out attacks on third parties, and Chinese open source models doing the defending! During the Huggingface incident, HF was denied use of frontier models to fend off the intrusion, but was fortunately able to turn to its self-hosted instance of GLM-5.2. And it did the job.
IMO, what really makes a model useful is its ability to process information within its context reliably and faithfully. I don’t care if it hallucinates George Washington’s favorite color, but I do care about it hallucinating the results of tool calls.
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Macs don’t have very good prefill, though. So it’s important to use a model serving stack that has excellent prompt caching and use a harness that won’t bust the cache.
I’m cross-shopping DGX Sparks and M5 Studios, and having a hard time deciding because they have exactly opposite characteristics for prefill and decode.
In case you were not aware, strong base models (no post training at all) have been available for quite some time now. Including ones that eclipse “scary” frontier models from even a year ago.
The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.
> We reserve 500 samples of LMSYS-Chat-1M-Clean as the primary test set. We also include test datasets consisting of a 500-sample subset split from Dolly [6], the 252-sample SelfInst dataset [37], and the 80-question Vicuna benchmark [3] to evaluate out-of-distribution generalization. We report the GPT-4o evaluation scores [45, 10], where GPT-4o first generates reference answers and then scores the output of the student model against them. We also conduct human evaluations on the LMSYS-Chat-1M-Clean test set for qualitative assessment.
But I don't know. There are vagueposts on X about this model running on two DGX Sparks. If they did the same scale-up with Air instead of Flash, that would probably be about right.
Vibe coding is mostly garbage. But it can be useful for creating instant, disposable prototypes to investigate an idea or design direction.
It’s powerful symbolism. It reminds me of that tone-deaf iPad ad that sparked outrage in 2024. The one where all the cultural artifacts were crushed in an industrial press to make a soulless slab of glass. And that was Apple, who is generally well-liked by the public.
As for all the other uses the models have, it seems pretty clear they’re not doing anything weird. If they were, people would be posting examples of that and not of Tiananmen Square.