Hacker News Reader

Comment by dmezzetti | Hacker News Reader

I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. It's possible it will take new model architectures to get there. But having more fine-tuned control over model parameters is a pattern that will emerge. Then smaller, domain specific models can be joined together to form larger models, if full generalization is needed.

I have an article on building micromodels that discusses some of this: https://neuml.hashnode.dev/train-a-language-model-from-scrat...

View on HN

100%, I'd be amazed if we don't hit a time when the huge models we currently have are looked at the same way we see old fashioned room size computers now.

Very apt analogy, stealing this :)

I would take the other side of that bet for sure.

Modern computers don’t have less ram than the old fashioned room size computers. Amount of RAM seems much more analogous here than the size of the hardware running the software. Of course the latter will go down, but the former? I don’t see why.

Techniques like distilling show that you can in fact get similar performance with 60% fewer parameters. Yes it depends on making the large model first, but that only implies that somehow the large model is perhaps more useful in initial learning. And if that's the case, then it would be rather surprising if we couldn't identify why and create new training techniques that go directly to the smaller equivalent model.

So in this way you've already lost your bet, as we already know that it's possible to create vastly smaller models with similar performance.

60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of.

Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled.

Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.

Early days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.

That doesn’t mean that a much larger with high quality training data and new training techniques won’t be more powerful than a smaller model. We have incredible efficiencies in modern computers in addition to immense resources. They both are needed for the best performance. I suspect it’s the same for LLM. Superior data, superior techniques, and superior scale will yield superior models.

Information density will increase in models just as spatial density increased for hardware.

Saying that information density will increase is very different from saying model sizes will decrease of a similar order of magnitude.

Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.

The computers I use are basically warehouse-sized now.

> I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale.

But all these smaller models are based on those 100b+ models! Like OP on TinyStories relies on larger models twice, to generate and then evaluate. Or the recent wave of small models which are all based on extracting data from GPT-3/4 to borrow the capbilities+RLHFing for free.

You can talk about how interesting it is and how much of an overhang neural nets have in terms of being overparameterized (which is an important AI safety/capabilities issue - the first AGI will be the largest, slowest, and worst one, by a long shot), but the one thing it doesn't tell you is that smaller models can replace big models entirely. Because they are parasitic on said big models, so you still need to train the big models to begin with.

(The smaller models also seem to lose a lot of the qualitative capabilities of larger ones, like meta-learning. Their TinyStories model does only stories. I doubt you could get it to 'dax a blick'.)

Not all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance. IMO, we still haven't even scratched the surface of what is possible with small LLMs. Especially not with pre-filtered/classified pre-training data. (Interesting LLMs based on their data approach and relatively small size: Qwen, InternLM, Mistral, Phi)

In the context of this article, the small/tiny models were 1--30 million parameters and the large model was 1.5 billion parameters. Efficient training of a 7-billion parameter model already requires algorithms for training across multiple GPU because the memory requirement for derivatives and optimizers will not fit on the typical 80G Ram of the current high end GPUs.

mind boggling that 7B is now considered a small model. I think it's valid, given the preeminence of 70B+ sized models. But wow, the community really just leap frogged over single digit billion parameter sizes.

> Not all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance.

I would, but they don't say what their dataset is that I can find anywhere, and the only thing they say about their instruction-tuned is that it's trained on 'publicly available' datasets. You know, the ones where a lot of them turn out under the hood to be drawing from the OA API or other pretrained models in some way or another...

> Especially not with pre-filtered/classified pre-training data.

Indeed not! But what exactly is prefiltering or classifying all that data...?

Since GPT-3 OpenAI has been filtering their pre-training data, and I believe others have done it too

I would like to have a more modular approach with more specialized training of the models.

Currently I can only really use smaller models for micro-tasks like sentiment analysis or classification, but any type of problem solving has to be left to GPT-4.

Agreed with the overall point (small models will prove powerful) but I disagree with:

> But having more fine-tuned control over model parameters is a pattern that will emerge

This assumes parameters do something interpretable. IMO, it’s more that parameters themselves are meaningless/dumb and patterns evolve via interaction of many such units (like in ants)

Empirically, perhaps due to the lottery ticket hypothesis, one needs to wait a lot longer for small models to reach a useful loss, so if one is impatient it almost always pays off to train the largest model that a given compute configuration can support.

I disagree - larger models are significantly more capable, likely as a function of their size, and the cost to inference them is only going to go down.

Maybe for niche use cases like handling customer support requests small models will do well, but GPT4 is not going to be condensed to a few billion anytime soon imo.

Believing this is existential for a thousand AI seed-stage startups. I hope it’s true for their sakes.

Hear the other timeline from Darius Amodei. https://a16z.com/improving-ai/ We may be entering a regime of multi-billion training costs on custom hardware. Open-source AI will be like open-source CPUs. Cool but the real thing even at the smallest scale comes out of monopsonized (is that a word?) inaccessible infrastructure

I appreciate the conciseness of this article. Thank you for sharing. I agree. The kitchen sink data strategy appears to be fairly inefficient with current model architectures.

I’ve been wondering about variations on panel of experts designs that use a “mob” of small models or possibly a network of them. Sort of like crowdsourcing with baby AIs.

Could we have libraries of small models that are all experts on niche subjects that basically collaborate?

The advantage is that it’s easier to scale and distribute lots of small models on commodity hardware than it is to run giant models that require heaps of contiguous ram.

I’d love to have some time to play with versions of this.

The best design we know is how God made humans. Our brains have specialized parts that work together with analog components. We’re raised on knowledge of increasing complexity as humans, given general-purpose heuristics, and then we specialize. From there, small teams of specialists interacting typically do the best jobs. So, I’d try to do stuff like that first since it’s proven in the only general intelligence that exists.

If I do experiments, I’m making models specialized from foundational to programming to specific language to specific tasks. Each pass will have training data so it picks up general concepts. Then, it will be trained to focus more on what it needs. Eventually, what’s slushing around in its neurons should be just the right information. Or it won’t work.

Anyway, my concepts that I published are at this link:

https://gethisword.com/tech/exploringai/index.html

(See alternative models section.)

You could always start with a big model then periodically prune low activation parameters as part of fine tuning until the desired size is reached.

There are hard limits to what a "brain" of a given size can do. Expecting ensembles of 7B models to match what properly trained 180B model can do at specialised tasks is like expecting a thousand monkeys trained on a specialised task to be able to match human performance.

If this were true, then GPT4 would be better at chess than Stockfish.

Since it's not, it isn't.

What if there was a large inefficient model just good enough that it could generate small specialized ones and orchestrate them?

Now that I've said it, I wonder if there's any evidence human brains work that way.

By that logic, humans should be better at chess than Stockfish, but they aren’t.

Reply on news.ycombinator.com