marzukia

Fair enough, nothing more infuriating than the sycophantic way LLMs write.

But I’d definitely be dishonest if I said I’d stop using LLMs to tidy my writing (out of principle or otherwise), my hold on the English language has seriously degenerated over the last eight years since I pivoted into IT from customer facing roles.

Ahhh yeah, I’m not going to sit here and say I don’t use Claude to clean up my writing (ie make it actually coherently laid out). In all honestly I tend to write in a rambling stream of consciousness style across random scrap markdown files (ha), but point is taken.

Thanks, good call. I've switched both throughput charts to a log y-axis (they were already log on x), so the sublinear taper is actually readable now instead of getting flattened by the big prefill numbers up top.

On mlx-engine / mlx-lm: I'd wager it's not resolved upstream. The core bug here is a re-prefill on hybrid recurrent models, and it's not isolated to my setup. oMLX hit it, and llama.cpp has the same issue open right now (https://github.com/ggml-org/llama.cpp/issues/22746). When two independent engines trip on the same thing, it usually points at a shared architectural gap rather than a one-off, so I'd assume mlx-lm is worth checking too.

The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.

I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.

I spent three weeks debugging why my Qwen 122B setup on an M3 Ultra was taking 3–5 minutes to generate the first token on follow-up messages (despite having a "warm" context).

The root cause wasn't the model, but three specific infrastructure bugs in my serving stack:

1. Prompt Instability: A unique message ID in the system prompt broke byte-exact KV cache matching, forcing a full re-compute every turn.

2. Interrupt Path: Streaming replies weren't persisted when the generation was interrupted, causing history divergence.

3. Checkpoint Poison: A background writer created unmatchable checkpoints that crowded out valid ones, triggering aggressive eviction.

After fixing these, prefill time dropped from minutes to sub-seconds (53k tokens cached, 33 tokens prefilled).

I've open-sourced the fork (qMLX) and a benchmarking tool to verify these numbers. Would love feedback on the hybrid attention caching strategy or any other edge cases I might have missed.