Hacker News Reader

Comment by ianbicking | Hacker News Reader

This makes me think: the math behind neural nets is kind of arbitrary. The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want. That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals.

View on HN

I used to design low-precision hardware to train AIs.

> The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want

That is accurate.

> That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals

Some people have published research where the parameters or activations are binary, sometimes in a stochastic fashion. Typically either the AI doesn't converge or it converges to a rather suboptimal solution.

Also, just because the weights or activations are binary it doesn't mean there's no math involved. You need some sort of a mathematical model to figure out how to tweak the parameters of the model in order to improve its fitness to the data.

The keyword the original comment was missing is differentiability. This is what a) and b) imply because it is how you ensure that you can train your model using gradient descent or whatever. So you need a particular kind of math to train these models using the methods we have today. There's quite a bit of research into other methods for training neural networks that don't come with this constraint, because it's obvious the human brain doesn't work that way. But this is early research and very far from being as useful.

I wouldn't say it's obvious the brain doesn't work this way.

See the free energy principle, one of the leading theories of how our brain works.

> The free energy principle is a theoretical framework suggesting that the brain reduces surprise or uncertainty by making predictions based on internal models and updating them using sensory input. It highlights the brain's objective of aligning its internal model with the external world to enhance prediction accuracy

Making predictions and then correcting for them in some way is a pretty low bar and it's often achieved through differentiability. I wouldn't be surprised if relevant neurological structures exist in our brains that do this.

There are reasons to believe it could work similarly, but they are related to computational efficiency and out lack of knowledge regarding more efficient methods. But I would not bet on our own ignorance as an explanation.

> built on "math" or "numbers" but could be built on bits

That reads strange, what is the difference besides the range of numbers?

Maybe bitwise operations instead of arithmetic?

arithmetic is implemented as bitwise operations so i’m still confused. the lower the precision / smaller range of values you choose to represent means fewer bitwise operations are required.

Maybe something like this: https://arxiv.org/abs/1601.06071

Probably even 4 bit values suffice. Invent some binary operations that exhibit some smoothness or continuousness such that gradient descent still works, because mostly a small change should have only a small effect. I think mathematicians would say functions that are differentiable.

Quantization is done with 4 bits, unless you meant something else?

Perhaps quantization with 4 bits is a special case of what I meant.

With 4 bits you have 16 different weight values but these values must not be quantized. They could be anything provided gradient descent is possible. Have some combination of binary operations instead of mathematical operations. What these operations do exactly, does not really matter, as long as gradient descent/backpropagation stays possible.

In other words, the model will find a way to use the weights correctly.

I would call it applied statistics :-D

Reply on news.ycombinator.com