Hacker News Reader

Comment by ilaksh | Hacker News Reader

This is not an LLM obviously , it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.

View on HN

It's an slm (small language model) due to number of parameters and it uses the same architecture as an llm, but llms have billions of parameters

Is it? What is the cut off these days in terms of number of parameters? And where do other language models such as BERT/ROBERTA which are encoder only fit?

LLMs are an imprecise, more of a marketing term, to define Transformer models based on the self-attention mechanism, trained with massives amounts of data.

And this implements a transformer. Actually it is a very cool didactic example.

Hasn't it been repeatedly shown that many small models perform worse than a large model of the same total parameters?

I think that is a sort of a reverse scaling fallacy. Given the right resources and environments, many small models can function together in an emergent way. I’ve been on the lookout for an SLM version of Conway’s game of life. SLM always reminds me of slime molds, which demonstrate a form of intelligence which is remarkable.

> Given the right resources and environments, many small models can function together in an emergent way.

What is this based on? Every researcher I've heard talk about this says it's exactly not true, as an uncontested rule, because the larger models will more effectively contain the smaller models, and use them together in ways that the connections between the smaller models can't. Remember, even MOE is to save compute/memory, not to help performance/parameter.

That is my understanding as well. Thousands of monkeys do not equal or surpass a man, intellectually, even if working together. There is some intrinsic super linear scaling in intelligence.

Yes, this is the understanding, specifically that very large models more effectively contain and combine smaller "models" inside them.

Does that mean an African elephant 2.57×10^11 (neurons) is smarter than a human 8.6×10^10 (neurons)?

This is not a good comparison, because the brain doesn't only do "being smart" - it has to do things like innervate muscle and other tissue through the body, elephants will require more neurons given their larger size, just to be able to *walk*

You're comparing completely different training data, harness overhead, and cost function. You're also comparing brains, which aren't really related to this discussion at all.

But, it depends on what you're measuring. By spatial/navigational memory, yes, elephants are far far better. Reasoning, no. It would be interesting to see what an elephant or whale eugenics program could result in, since humans have that pesky (or maybe instrumental?) birth canal problem.

This is not a normal small model. It's in a different category of tiny.

Isn't a MoE model basically a cascading tree of smaller models or some variation of that?

You could think of it as a standard decoder only LLM (almost all modern ones we use everyday), with some layers (experts) having parallel networks and conditionally based on the input token (per token) - the token is routed through some of these layers. In the case of a non MoE (dense) - each token goes through all layers, so the inference engine has to read all the layers and do a matrix (layer) times vector (token) computation, while in the case of MoE the number of layers per token that has to do the compute is substantially lesser, so one can expect much higher tps than a dense model at the same number of parameters (size - 7B, 27B etc)

Is token rate a function of parameter size?

Parameter size and total number of parameters so ultimately the total size of the model in memory.

This leads to some interesting optimizations. You can quantize all the parameters (or certain layers) of a model and halve or quarter the memory requirement but maintain most of the model's intelligence. This increases the token rate inversely with the size reduction.

Popular quantizations for local models are 8-bit and 4-bit parameter sizes. The Blackwell series of nVidia chips now even support native FP4 math making 4-bit quantizations even faster.

Yes, a quick back of the envelope math is 0.65 * (memory bandwidth of the card / (model weights in bytes + kv cache in bytes) ~ practical decode tps. Below context around 32k (depends upon the model but again can be used as a placeholder number) you can ignore the kv cache in bytes and the math becomes just about memory bandwidth and model weights in bytes.

if you think of the depth of a model increasing with parameter count, then the number of steps of parallel operations to percolate data through a model would increase with parameter count, which would be inversely related to rate of computing tokens as a result of flowing data through the model.

Not quite linear, but yes.

they'd just be too small to make sense of other models' outputs

Reply on news.ycombinator.com