Hacker News Reader

Comment by novaRom | Hacker News Reader

How small can be a LLM transformer in order to be able to understand basic human language and search for answers on the internet? It should not contain all the facts and knowledge, but must be quick (so, it's a small model), understand at least one language, and know how and where to look for answers.

Would it be sufficient to have 1B, 3B or 7B parameters to achieve this? Or is it doable with 100M or even fewer parameters? I mean vocabulary size might be quite small, max context size could also be limited to 256 or 512 tokens. Is there any paper on that maybe?

View on HN

A team at Microsoft Research asked the same question and just published a paper about part of that at least: TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759

"We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities."

They also trained a TinyStories-Instruct instruction following variant. It looks like the 28M parameter 8 layer model had a 9/10 on grammar and consistency. You could probably combine that with something like Jsonformer or parserLLM to enforce valid formatting.

You don't need a 1B+ parameter model for this workflow, but it helps, particularly for the quality of the final output.

> know how and where to look for answers

The point of calling them Agents is that they don't know how. All the examples, including AutoGPT that makes AI influencers go OMG AGI, are operating in a discrete action space with user-specified hints to select which action (or none at all).

I like the idea that a small model could fine tune itself for each task more affordably, so that could be an advantage.

The contents of whatever it found online is fed back in the context, plus the response it generates based on that also counts to the limit.

So 512 is really inadequate unless you just want to make search queries using natural language.

Reply on news.ycombinator.com