kgeist

That's the problem with etching a model onto a chip: by the time you've designed the chip, manufactured it, tested it, shipped it, and deployed it, the model will be hopelessly outdated (with the current improvement rates). And when you want to update, you have to buy new chips instead of just uploading a new model file like now. When Taalas announced their chip, the model was already 1.5 years old (stone age by current standards). It's their first chip, so maybe they can streamline it, but the problem of having to update hardware every few months to keep up with the industry is not going anywhere.

>wine

It could be an original Proto-Indoeuropean word as well, because Greek has ὑιήν "grapewine" (< *wih₁-ēn), which follows PIE ablaut (weyh₁-ō ~ wih₁-ēn), which doesn't usually happen if it's just a borrowing of a foreign word. And the same root is found in Latin vitis "vine", Russian vit'sa "to twist (often about vines)" etc.

>Words borrowed from a Pre–Indo-European language into Mediterranean languages

> [...] Greek μύρμηξ mýrmēx ‘ant’, Latin formica

Must be an error:

  Proto-Celtic *morwos
  Proto-Balto-Slavic: *marwis
  Proto-Indo-Iranian: *marwiš
  Proto-Germanic: *mauraz
  Old Armenian: mrǰimn

Greek murmēx could be an assimilation murw- => murm-, and Latin had dissimilation morm- => form- (although not clear what came first, maybe morm- was the original and morw- came later). Sanskrit also has vamra "ant", which makes it look like the whole thing is a tabooistic distortion of *wr̥mis "worm".

In any way, it doesn't look like it must be borrowed. Historically, some words once labeled Pre-Indoeuropean turned out to have pretty mundane PIE origins.

I have a similar approach where I optimize kernels and find numerical differences between the CPU oracle and CUDA kernels using an automated AI agent in a feedback loop. Usually it solves numerical problems easily (it compares outputs of every layer and finds where they diverge), but so far no matter how many different SOTA models I throw at it, and even show it reference code from other inference engines, they aren't able to much the speed (my engine has a modification which is not found in reference code, although a lot of stuff is similar). Either I'm doing something wrong, or z.ai's Infra Agent is actually an agent swarm, i.e. a bruteforce with heuristics. My project is 2 weeks old so maybe I just need more time.

Judging by the leaks, OpenAI and Anthropic already train reasoning traces to use fewer tokens (they deliberately omit articles and prepositions, use very short sentences, etc.), even though you pay per token. So it wouldn't make sense to do that if the only incentive was "make them pay for as many tokens as possible per task."

It's more subtle than that. If a user has to wait longer for a solution/pay more, they'll be less satisfied and may switch to a competitor. More unnecessary tokens also means more unnecessary compute. Longer sessions are increasingly more expensive to serve than shorter sessions.

And there's always the Jevons effect: as a resource becomes cheaper, demand often increases, and so does net resource consumption.

So, imho, frontier labs have every incentive to reduce token usage per task (while also making you use AI for more and more tasks in your daily life)

The technique looked dubious from the start, because LLMs were trained to expect certain outputs from common bash tools. If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is broken, the arguments passed to it were wrong, or it's a newer/older version of the tool etc => more tokens. Sounds like just adding to the prompt to use `grep` and `tail` extensively will do the trick without any special tooling.

>What stops them <..> simply use cheaper model for every Nth request.

That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra.

To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync.

If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.

The web UI's system prompt is also probably in Chinese

They mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-publ...

>device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate

I.e. low memory bandwidth.

There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load

Do those reports require Kimi K3 though? Qwen3.6+ could probably do the same in a few seconds with similar quality.

On their Agentic Index, GPT-6 Astra (both max/xhigh) has the same result as Qwen3.8-27b. Weird.

I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency).

Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurrent use, while vLLM/SGLang easily crash on non-common setups (things like, if you do pipeline parallelism for RTX5090+RTX4090, they will randomly crash with RAM caching enabled or select wrong kernels because they usually assume that every rank is the same device type; they also don't support Q5-Q6).

For me what's most interesting is to optimize inference for lack of good datacenter hardware and how to optimize for it best. I've been running an AI server in the office, and so far I've find these techniques most important for concurrent use on cheap hardware: pipeline parallelism (to accomodate for PCie), RAM caching (to quickly restore contexts into VRAM), speculative decoding (including domain-specific ngrams, they already can speed up code generation considerably without the overhead of a draft model), good kernels highly optimized for a specific device, support for Q5-Q6 (almost as good as Q8), FP8 contexts (more context to fit), paged attention (for better VRAM utilization), prefix caching, continuous batching (this is the default everywhere).

So far the main bottlenecks have been llama.cpp's poor VRAM utilization for contexts (you either have fixed-size slots, or use unified KV cache where each request attends to attention from all other requests and then unnecessary portions of attention are masked out), and lack of decode/prefill segregation: when a request starts prefilling a long context, all decoding threads slow down to like 5 tok/sec. On the other hand, vLLM/SGLang feel superbuggy if you don't run them on some officially approved node like 8xH200

>but also of other people.

Yeah, there's this thing called the curse of knowledge. If an engineer has a deep understanding of something, it's not a given that they can explain it well. For them, the topic feels so simple, and they've done it so many times that they may have forgotten other people aren't as knowledgeable. They will throw terms around without explaining them, etc.

How about: "Type the seahorse emoji to solve the CAPTCHA" :) Something that triggers infinite loops in LLMs or trips the guardrails.

In agentic scenarios, an LLM has to read far more tokens than it outputs. I think focusing only on the decode speed is somewhat misleading. 14 tok/sec for decode is actually okayish. 93 tok/sec prefill is what's abysmal, my RTX 5090 goes above 2000 tok/sec with 5 bit quants.

Qwen 3.8 has the same architecture and the same parameter count as Qwen 3.6. Something is not right with the GGUF if it's 2 times slower. The post says "The hybrid attention architecture is new" and says the author's older Llama build from a "couple weeks ago" failed to run Qwen 3.8 because it did not support Qwen35 architecture, but both 3.6 and 3.8 are based on Qwen35 which was released in February 2026. The post doesn't make any sense.

AI is as deterministic as a human programmer. A programmer who is burned out and has not slept well produces code of a different quality than a programmer who has slept well and is highly motivated. That's hardly deterministic. AI can be made deterministic enough if you have strict quality gates, just like with humans.

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do it too. Say, when MySQL fails to start because it tries to read its config from the wrong dir, I may say "oh, this guy thinks he must read the config from ..." (having a language with grammatical genders as my native language also helps make it sound pretty natural). It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.

>Tiny ~6mb binary

I wonder why it's so large for a program written in Zig. It's basically just a loop that accepts user input, prepares the context, sends it to the LLM, parses the output, invokes the tools, and presents it all in the terminal. Add the built-in prompts and a few checks here and there (like blocking a write tool call before the file has been read first), and I'd expect a truly tiny native agent to be around 200-300 KB max.

It's basically a variation of HyDE (Hypothetical Document Embeddings), and the rationale is that the embedding of the query is not necessarily close to the embedding of the answer. If you generate a hallucinated answer, it can line up with the actual document better (in the embedding space, via BM25, or hybrid).

But honestly, it only works for common knowledge that's already in the LLM. If the target document contains very niche or private information, then the hallucinated answer's embedding can be even farther away than the query's.

A common case I have is when you don't have classifications to begin with. For example, you need to find what users complain about most. I take embeddings of all records, then cluster the embeddings into semantic groups, then ask an LLM to take a random sample from each clustered group and create a classification for that group.

This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.

I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.

I wonder how it impacts code generation. It shouldn't impact prose in general because of synonyms and whatnot, but code requires exact reproduction. That is, what happens if you ask an LLM to recite a large, human-written excerpt as is, without modifications? Wouldn't the modified token sampler try to change some tokens here and there (for the watermark to work)?

For example, what if I say, "Repeat this text verbatim: %long_human_written_text%"? Would the output be recognized as AI-generated or human-generated?

More compute/larger datasets during training != larger models. The Bitter Lesson was that just scaling things up beats custom hand-crafted optimizations. Up until 2024, we thought that meant scaling up the parameter count, but then that started to plateau. After o1 was released, we thought it was about scaling up test-time compute. Now, seeing how 30b models can easily outperform 200b models from a few years ago, it seems like what we need to scale is RL, at least for agentic capabilities. It looks like 30b is already enough for good agentic capabilities. Larger models aren't considerably "smarter" (especially since they're mostly MoE anyway, with something like the same 30-50b active parameter range), they just know more (better world knowledge), which lets them make more informed decisions. Maybe we just need to scale up the retrieval layer.

I agree with the article, but there's one thing Go has that doesn't help LLMs: structural typing. An LLM has to grep a little more to understand which interfaces a struct implements.

In the BlackHat presentation on the HuggingFace incident, OpenAI showed some excerpts from the reasoning traces, and they had that grug speak too (skipped articles, etc.). So the OP's method must have indeed found the actual reasoning traces.

LLMs already think in latent space. The generated reasoning tokens are only the surface of what's happening internally. An LLM may write one thing in the traces but decide differently in the latent space. The whole token-based "reasoning" thing was just a clever hack to extend the existing architecture without completely redoing it. In one of Anthropic's recent papers, they added an additional subnetwork trained to map internal states to readable text, so that's probably the vector of further development.

The article is AI-written as well, with all the "honest problems" and "real weak spots". The code must be AI-generated too, so without a human properly verifying it, I can't take the project seriously. It may just be AI hallucinations congratulating themselves on imaginary achievements.

The social network VK internally uses highly specialized database engines per business domain. They don't use stock DBs. They have a DB engine for posts, a DB engine for likes, etc. They have a team of DB engineers. Their DB load was around 250 mln RPS 3 years ago. Stock DBs were harder to scale for them. I guess if you have immense highload, having a team of DB engineers can be cheaper because you can save a lot on servers. I reviewed their code. A DB engine's source code is pretty compact and simple (relatively speaking) because they deal with very specific domain entities, so they don't have to account for all the possible user query combinations that a general-purpose DB would have to support. It was mostly shards+binlog+snapshots+views in RAM. Considering that Telegram was founded by former VK engineers, I suspect they have something similar.

I'm not sure it's a fair test either to compare the "low" setting of one model with the "low" setting of another. They're completely different settings that just happen to have the same name.