Mixture of Experts is absolutely not what they're describing. MoE has to be one of the most misleadingly named things ever. It's completely confusing as to what it actually is.
Imagine I had [a model that was good at math], [model that was good at code], [model that was good at writing], [model that was good at general knowledge]. If I then had [a model that was good at determining whether the user query would be best served by one of those models and sent it to it, leaving the rest of the models inactive], that is the platonic version of what MoE is. In practice, it works a bit differently. It instead basically restricts the number of pathways that can be utilized in solving problems during training, which allows for "expert neuron groupings" to form and "classifier layers" to form earlier on in the structure, but the effect is the same (better, even, since it allows some overlap between structures of experts). It also allows "routing to an expert" to happen token-by-token rather than at the prompt level.