It's a 480B mixture of experts model with 17B active at any time. Looks like llama3 70B beats it on every eval they show. I remain super impressed by llama3 and just deployed it for a (simple) production use-case. Excited all these large companies are throwing resources into the open model games!