Not personally surprised that a MoE model performs so well.

I used Mixtral a lot for coding Rust, and it had qualities no other model had except GPT 3.5 and later Claude Sonet. The funny thing is Mixtral was based on Llama 2 which was not trained on code that much.

DeepSeek v3: 671B parameters on total, and 37B activated sounds very good even though impossible to run locally.

Question if some people happen to know: For each query it activates just that many of parameters, 37B, and no more?