Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by moffkalast | Hacker News Reader

Parent Full thread

moffkalast·

Llama-1, 1T tokens, dumb as a box of rocks

Llama-2, 2T tokens, smarter than a box of rocks

Mistral-7B, 8T tokens, way smarter than llama-2

Llama-3, 15T tokens, smarter than anything a few times its size

Gemma-2, 13T synthetic tokens, slightly better than llama-3

(for the same approximate parameter size)

I think it roughly tracks that moar data = moar betterer.

View on HN

> not smart > slightly smarter > way smarter > Last one, "slightly smarter"

So, the the usual s-curve, that has an exponential phase, then topping out?

Pretty much, yep. There was definitely a more significant jump there in the middle where 7B models went from being a complete waste of time to actually useful. Then going from being able to craft a sensible response to 80% of questions to 90% is a much smaller apparent increase but takes a lot more compute to achieve as per the pareto principle.

I see giant models like Intel chips over the last decade: big, powerful, expensive, energy hogs.

Small models are like arm: you get much of the performance you actually need for common consumer tasks, very cheap to run, and energy efficient.

We need both but I personally spend most of my ML time on small models training and I’m very happy with the results.

but the OP was talking about the size of the context window, not the size of the training corpus

Hmm right, I read that wrong. Still, interesting data I think.

Reply on news.ycombinator.com