Hacker News Reader

Comment by didntreadarticl | Hacker News Reader

Have we got any details on the benchmarks that show LLaMa's 13B architecture outperforming GPT-3? Because that seems kindsof fantastical. Is it just a product of a very specific benchmark or does it reflect real world performance?

View on HN

The GPT-3 they are comparing to is the one that was released on 2020. Since then OpenAI made a lot of improvements and nowadays I believe GPT-3.5 is competitive to Palm (540b). Still, LLaMa is in the same tier, with much less parameters.

GPT 3.5 is definitely not comparable to Palm. You can see some benchmarks here. https://crfm.stanford.edu/helm/latest/?group=core_scenarios For example, text-davinci-3 is 56.9 on MMLU, Flan-Palm is 75. https://arxiv.org/abs/2210.11416

Flan-Palm is 75 with five-shot. text-davinci-003 is it 56.9 with five-shot or zero-shot?

You're right. Equivalent scenarios, the gap is smaller - about a difference of 10. you can check the end of the flan paper for some equivalent comparisons.

Just a heads up with my comparison. Under equivalent scenarios, the gap is smaller. davinci-003 gets about 10 more points using five shot (which is what the palm comp does)

They list 7B, 13B, 33B, 65B architectures. Presumably, they compare 65b one to GPT-3 175B. Chinchilla model which is about 70B outperformed a much larger GPT-3 model. So not that fantastical.

EDIT: I stand corrected. They do compare 13B model with a large GPT-3 model which is hard to believe without a bit more concrete evidence

You're incorrect.

> For example, LLaMA's 13B architecture outperforms GPT-3 despite being 10 times smaller.

Of course this is all on benchmarks but it's a big improvement if true.

Reply on news.ycombinator.com