Thanks. I agree, GGUF and upstream contributions are on my radar.
23 karma · joined June 10, 2025
Some parts are needed to generated every single token and these really should fit in memory, but the router experts that are not neeed can rest on SSD and be read only if they are needed, so... you can run MoE models bigger than you memory, try the IQ2XXS.
It should work on your 64 GB after you enable SSD mode in DwarfStar (in MoEspesso it enables itself), while being slower, so... I am really hoping for good models between the 50-120 GB other than Laguna, there is a big gap right now unfortunately.
There is a different set of experts at every layer, and each layer has a small router that decides which ones to use.
The router needs to look at the state produced by the experts below it.
Drafted tokens from the MTP head can be used to predict which experts the first layer will want, but not beyond that. To know what layer 10 experts needs, you have to run layers 1-9 which means loading their experts.
So, yes, instead of a next-token drafter like MTP, you'd want something trained to predict the expert activation across all layers at once.
With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine).
The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.
Magistral Small is a 24 billion parameter model.
Pretty impressive in terms of efficiency for Mistral.
The size of the Magistral Medium is not publicly available, so it is difficult to compare efficiency there.