Ask HN: GPT-4 has 1.7T parameters. What's a parameter? | Hacker News Reader

Ask HN: GPT-4 has 1.7T parameters. What's a parameter?

13 pointsherodoturtle9 comments

Hi folks, AI newbie here.

Can those of you that are well versed in AI please help me understand what exactly constitutes a parameter?

I often see LLMs being compared / ranked based on their number of parameters, so I'm hoping to better understand this metric.

I have a rudimentary understanding of artificial neural networks (in terms of inputs -> functions -> outputs), and by extension, a very basic understanding of deep learning.

But when it comes to defining a parameter, my google searches thus far have led me to concepts such as 'AI model behaviour' and 'adjustable settings' which, whilst interesting, are still too complex for me to distil into simple terms.

If I were to explain what a parameter is to my child, what might I say?

Thanks!

View on HN

Checkout this tweet https://twitter.com/ylecun/status/1706545305762582580 by Yan LeCun.

"Parameters are coefficients inside the model that are adjusted by the training procedure. The dataset is what you train the model on. Language models are trained with tokens that are subword units (e.g. prefix, root, suffix)."

His comment on GPT-4 parameters count

"Also: a model with more parameters is not necessarily better. It's generally more expensive to run and requires more RAM than a single GPU card can have. GPT-4 is rumored to be a "mixture of experts", i.e. a neural net consisting of multiple specialized modules, only one of which is run on any particular prompt. So the effective number of parameters used at any one time is smaller than the total number."

I thought it was one expert runs per token?

f(x) = ax^2 + bx + c

a, b and c are parameters. So this is a model with 3 parameters. Keep adding parameters and chaining various operations on large inputs f(g(h(...) until 1.7T.

Thank you for this.

I have a basic grasp of algebra (including quadratic functions such as in your example).

So am I on the right track in extrapolating this to say that LLM parameters are the adjustable coefficients and constants in the chain of functions?

Yes, and adjustable here means that they are learned from the training data. They start as random values and get repeatedly adjusted during the training process until the model exhibits useful behaviour.

The explanation you got above is sound but slightly misleading. Let me explain. Typical neurons in a neural network are actually simpler that a quadratic equation. They are actually a linear function y = ax + b, where a and b are vectors. So, y = a_1 * x_1 + a_2 * x_2 + ... a_n * x_n + b.

Here, (a_1, a_2, ..., a_n, b) are trainable, i.e. adjustable, parameters. (x_1, x_2, ..., x_n) are the inputs to the neuron. y is its output. Because linear functions are inflexible, neurons typically pass the output y to a non-linear function. The classical non-linear function is a sigmoid, but there are many choices. So you have z = sigmoid(y) = sigmoid(ax + b) = sigmoid(a_1 * x_1 + a_2 * x_2 + ... a_n * x_n + b). These non-linear functions usually have no trainable parameters.

Such a structure, a linear function composed with a non-linear function is the cornerstone of modern statistics, where it is called a generalized linear model (GLM). Hence, neural networks are just many GLMs or neurons grouped into layers, with layers connected to each other.

A parameter refers to any trained value in the model. If you initialize it to a random number at the start of training - it's a parameter.

In the context of transformer language models, that includes the weights and biases in the feed-forward layers, as well as the input embeddings, positional encodings, and transformers query, key, and value matrixes.

Number of connections between neurons, basically.

>GPT-4 has 1.7T parameters

That's a rumored parameter count. OpenAI has not released technical details of any of their newer models.

Think of it like 1.7T different settings. For the model you're using, they're set in a specific way

Reply on news.ycombinator.com