They serve it about 2x slower. So it must have about 2x the active parameters.
It could still be 10x larger overall, though that would not make it 10x more expensive.
It could still be 10x larger overall, though that would not make it 10x more expensive.
That's how you get Llama 4.
Pretty much every major lab settled on ~3-5% sparsity for a reason.