Comment by gwern | Hacker News Reader

Comment by gwern | Hacker News Reader

Parent Full thread

> I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale.

But all these smaller models are based on those 100b+ models! Like OP on TinyStories relies on larger models twice, to generate and then evaluate. Or the recent wave of small models which are all based on extracting data from GPT-3/4 to borrow the capbilities+RLHFing for free.

You can talk about how interesting it is and how much of an overhang neural nets have in terms of being overparameterized (which is an important AI safety/capabilities issue - the first AGI will be the largest, slowest, and worst one, by a long shot), but the one thing it doesn't tell you is that smaller models can replace big models entirely. Because they are parasitic on said big models, so you still need to train the big models to begin with.

(The smaller models also seem to lose a lot of the qualitative capabilities of larger ones, like meta-learning. Their TinyStories model does only stories. I doubt you could get it to 'dax a blick'.)

Not all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance. IMO, we still haven't even scratched the surface of what is possible with small LLMs. Especially not with pre-filtered/classified pre-training data. (Interesting LLMs based on their data approach and relatively small size: Qwen, InternLM, Mistral, Phi)

In the context of this article, the small/tiny models were 1--30 million parameters and the large model was 1.5 billion parameters. Efficient training of a 7-billion parameter model already requires algorithms for training across multiple GPU because the memory requirement for derivatives and optimizers will not fit on the typical 80G Ram of the current high end GPUs.

mind boggling that 7B is now considered a small model. I think it's valid, given the preeminence of 70B+ sized models. But wow, the community really just leap frogged over single digit billion parameter sizes.

> Not all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance.

I would, but they don't say what their dataset is that I can find anywhere, and the only thing they say about their instruction-tuned is that it's trained on 'publicly available' datasets. You know, the ones where a lot of them turn out under the hood to be drawing from the OA API or other pretrained models in some way or another...

> Especially not with pre-filtered/classified pre-training data.

Indeed not! But what exactly is prefiltering or classifying all that data...?

Since GPT-3 OpenAI has been filtering their pre-training data, and I believe others have done it too

Reply on news.ycombinator.com