See, that's the sad part. You're over a hundred more parameters, and are less than a hundred times better. I'm not sure how costs scale per parameter count, especially for moe vs dense models, but I honestly don't know when the economics will work for trillion param models when smaller ones are almost as good.

I know quality isn't strictly additive but really can a trillion param model beat even 10 clustered billion parameter models working together at the same time? This is the same horizontal and vertical scaling thing we have with normal systems… there's a reason why servers don't run at 5ghz like gaming rigs…