Google uses int8 for LLM training | Hacker News Reader

Google uses int8 for LLM training

old.reddit.com

48 pointstosh22 comments

This makes me think: the math behind neural nets is kind of arbitrary. The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want. That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals.

> built on "math" or "numbers" but could be built on bits

That reads strange, what is the difference besides the range of numbers?

Maybe bitwise operations instead of arithmetic?

arithmetic is implemented as bitwise operations so i’m still confused. the lower the precision / smaller range of values you choose to represent means fewer bitwise operations are required.

Maybe something like this: https://arxiv.org/abs/1601.06071

I would call it applied statistics :-D

I used to design low-precision hardware to train AIs.

> The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want

That is accurate.

> That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals

Some people have published research where the parameters or activations are binary, sometimes in a stochastic fashion. Typically either the AI doesn't converge or it converges to a rather suboptimal solution.

Also, just because the weights or activations are binary it doesn't mean there's no math involved. You need some sort of a mathematical model to figure out how to tweak the parameters of the model in order to improve its fitness to the data.

The keyword the original comment was missing is differentiability. This is what a) and b) imply because it is how you ensure that you can train your model using gradient descent or whatever. So you need a particular kind of math to train these models using the methods we have today. There's quite a bit of research into other methods for training neural networks that don't come with this constraint, because it's obvious the human brain doesn't work that way. But this is early research and very far from being as useful.

I wouldn't say it's obvious the brain doesn't work this way.

See the free energy principle, one of the leading theories of how our brain works.

> The free energy principle is a theoretical framework suggesting that the brain reduces surprise or uncertainty by making predictions based on internal models and updating them using sensory input. It highlights the brain's objective of aligning its internal model with the external world to enhance prediction accuracy

Making predictions and then correcting for them in some way is a pretty low bar and it's often achieved through differentiability. I wouldn't be surprised if relevant neurological structures exist in our brains that do this.

There are reasons to believe it could work similarly, but they are related to computational efficiency and out lack of knowledge regarding more efficient methods. But I would not bet on our own ignorance as an explanation.

Probably even 4 bit values suffice. Invent some binary operations that exhibit some smoothness or continuousness such that gradient descent still works, because mostly a small change should have only a small effect. I think mathematicians would say functions that are differentiable.

Quantization is done with 4 bits, unless you meant something else?

Perhaps quantization with 4 bits is a special case of what I meant.

With 4 bits you have 16 different weight values but these values must not be quantized. They could be anything provided gradient descent is possible. Have some combination of binary operations instead of mathematical operations. What these operations do exactly, does not really matter, as long as gradient descent/backpropagation stays possible.

In other words, the model will find a way to use the weights correctly.

Can someone explain what is missing in my mental model of the significance of this?

Why does it matter than one is an int and one is a float? I understand that this means a different range of values and that floats are manipulated differently and with different costs on GPU and CPU because of the way they are stored.

Aren't LLMs basically just probability stages, so why does it matter if the values are floats or ints, other than to note that more space for the data gives you more fine grained probabilities?

Is this difference mainly to suggest that Google uses different hardware that is more suitable for processing ints and that is a competitive advantage somehow?

Elon's always prattling on about how Tesla does this for FSD and it is a big advantage but I'm too dumb to know if it matters. Thoughts?

given FSD doesn't actually exist still, I can't think why trying to decode whatever lies the CEO says about the process of failing to make that product is a good use of time

i would like to understand the pros and cons of int8?

Do you have a link to Tesla training with int8?

Oh I guess it’s inference. Not an expert here at all. https://x.com/elonmusk/status/1695376562055774664?s=46

I want to see an example where high bit precision is needed. Could audio be it? Touch?

High precision would matter near the input and output layers, but further into the model you could always substitute more parameters for precision.

Academic in the field: My answer is usually highly complex scientific simulations, military hardware, and non-euclidean mathematics. There's a trade-off space between precision and efficiency, so it depends on the constraints of the solution space.

Reply on news.ycombinator.com