Hacker News Reader

Comment by pugio | Hacker News Reader

> We’re currently using Mixtral 8x7B behind our endpoint mistral-small...

So 45 billion parameters is what they consider their "small" model? I'm excited to see what/if their larger models will be.

View on HN

There seems to be an experimental Mistral Medium model listed among other available model endpoints on [1], the comparison table they give shows that it outmatches 8x7B by a few percent on every benchmark listed

[1] https://mistral.ai/news/la-plateforme/

It apparently outperforms GPT-4 at WinoGrande as well…

> So 45 billion parameters is what they consider their "small" model?

According to Wikipedia: Rumors claim that GPT-4 has 1.76 trillion parameters, which was first estimated by the speed it was running and by George Hotz. [1]

[1] https://the-decoder.com/gpt-4-architecture-datasets-costs-an...

I still need to understand how George Hotz knows about the GPT-4 architecture if it is true.

As someone who has worked in the field for many years now and closely follows not just the engineering side but also the academic literature and the personell movements on linkedin, I too was able to put together a lot of this. Especially with GPT-3 Turbo it was obvious what they did due to the speed difference. At least in terms of model architectures and orders of magnitude for parameters. From there you could do some back of the envelope calculations and guess how big GPT4 had to be given its speed. I wouldn't have dared to say any specific numbers with authority, but maybe Hotz has talked to someone at OpenAI. On the other hand, the updated article now claims his numbers were off by a factor of 2 (at least for the individual experts - he still got the total number of parameters right). So yeah, maybe he was just guessing like the rest of us after all.

You don't necessarily need to know the architecture, given the "only" real metric regarding speed is tokens/sec and that pretty much depends on memory bandwidth, you can infer with some certainty the size of the model.

Also, if we have been eating up posted "benchmarks" with no way to independently validate them and watching heavily edited video presentations, why can't we trust our wonder kid?

That doesn't explain how we know that GPT-4 is a sparse MoE model with X experts of Y size and using Z of them during inference.

IIRC it was leaked/confirmed by accident or something like that

It's not even close to a 45B model. They trained 8 different fine-tunes on the same base model. This means the 8 models differ only by a couple of layers and share the rest of their layers.

Which also means you can fit the 8 models in a much smaller amount of memory than a 45B model. Latency will also be much smaller than a 45B model, since the next token is always only created by 2 of the 8 models (which 2 models are run is chosen by a different, even smaller/faster, model).

> It's not even close to a 45B model. They trained 8 different fine-tunes on the same base model. This means the 8 models differ only by a couple of layers and share the rest of their layers.

No, Mixture-of-Experts is not stacking finetunes of the same base model.

Do you have any more information on the topic? I remember reading that about significant memory savings achieved by reusing most layers.

Made sense to mee on first sight to me, because you don't need to train stuff like syntax and grammar 8 times in 8 different ways.

Also would explain why interference of two 7B models has the cost of running a 12B model.

The original paper by Shazeer suffices. What you are saying is in theory possible to do and may have been done in practice here, but in the general case MoE is trained from scratch and specializations of layers which develop are not products of some design choice.

Note that it processes tokens at speed and cost of a 12B model though.

If it uses 2 experts, they should parallelize so closer to 7B speed?

Memory bandwidth is still a factor, right?

They have a description and performance evaluation of Mistral-medium on their website: https://mistral.ai/news/la-plateforme/

"Our highest-quality endpoint currently serves a prototype model, that is currently among the top serviced models available based on standard benchmarks. It masters English/French/Italian/German/Spanish and code and obtains a score of 8.6 on MT-Bench."

Reply on news.ycombinator.com