The Mistral guys specifically mention that training speed (due to not needing as much compute) was one of the reasons Mixtral was released so soon after Mistal 7b.
- People hear "mixture of experts" and they think "N specialists" - but ex. think how much know you need to know to autocomplete "Two plus two is "
- Fundamental thing of ML is you define functions and give it data, and the more data you give it to the better. Once youre at "I will simply give it the training data needed to be good enough at the task and wall off that part of the implementation" you're outside ML and have a chicken and egg problem
- We don't know GPT-4 is MoE
- MoE in practice is fundamentally about trading off runtime vs. static size properties to gain inference speed. I.e. 7x8 stored and picking 7x2 at runtime means youre somewhere between 7x2 and 7x3 in quality, inference at 7x2 speed, and have to train and store and load 7x8. You don't reach for it to increase quality, you reach for it to increase inference speed at the expense of inference ram and total model size.
Didn't Yampleg's tweet / leak confirm this one? I mean, he could be wrong about this, but I thought the consensus was on it being true by now.
(Copy of the removed tweets at https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_de... )
* I used to crusade against this rumor because the only source is that article, and people repeating that source. But I imagine it's a no-brainer given they have enough users that they essentially get a throughput bump 'for free' even if the model weights are huge, i.e. better to utilize as much GPU ram as you can muster, the cost of needing more GPU ram is offset by the cost of being able to run multiple inference against the model all the time anyway