Hacker News Reader

This guy doesn't have any idea what he is talking about. He consistently posts such bullshit on twitter. Mostly copy paste with added spice mix.

View on HN

I noted several things that don't seem consistent with what people have been assuming from before.

For instance - MoE yes, but 16 experts at 111B parameters? Doesn't make sense. GPT 3 had 175B parameters. I doubt they would go less on base models from now on. The number that makes more sense is ~220B parameters per model and 8 expert models. That is the same inference cost in total.

The 13T tokens of training data seems pulled from thin air.

It's Twitter, why would you think otherwise?

I'm tired of people just saying 'oh, internet' every time there is a factual inaccuracy somewhere. Yes, we know this is a social media network. Now can we get back to discussing the topic at hand?

Reply on news.ycombinator.com