Hacker News Reader

> neurons in the network are simplified mathematical representations.

That is the important difference. A biological neuron is more like a cell in the cell automaton (Conway's game of life). It is a dynamic process, even if the dynamics change by a small factor, it could have a very different behaviour. Artificial neurons on the other hand are much more uniform and lack the time dimension. If you account for dynamics, then learning in the brain is a completely different thing than learning in neural nets. It has been proven that Hebbian learning (local plasticity of synapses) approximates gradient based learning, though in simulation it attains lower accuracies.

Another observation was that biological neural nets don't have a mechanism to communicate gradients back. But a deep learning experiment proved you can have fixed random back connections and the network will still learn. They don't actually need to be symmetrical connections.

The real mystery is about loss functions. Such a loss function would have to be computed in a distributed fashion in the brain. They are the origin of the learning signal. The brain has many such loss functions that have been discovered by evolution. We have not understood loss functions well enough, and understanding them would be a huge leap forward.

Another important observation is that the brain learns from the environment, not a fixed training set. Environments are richer sources, and our current crop of AI agents are playing all sorts of games, but they never have the same complexity as real world. Btw, tomorrow DeepMind will make a Starcraft demo.

I keep saying this to neural net enthusiasts: brains and neural nets don't have anything in common other than some nouns. Not network topology, not even level encoding (bio neural nets are pulse rate encoded -closest "neural net" to this is something like a liquid state machine, which is not easily simulated on a Von Neumann computer). We actually know an awful lot about how things in the visual cortex work. Nothing like a "deep network."

Which makes the compelling results of deep networks even more mysterious. (https://gandissect.csail.mit.edu/) How could we have captured the seemingly enormously complex act of visual object recognition, with all the neurons and synapses and chemicals, in something so crude and simple as backpropagation and stochastic gradient descent with a bunch of ones and zeros? Or perhaps both processes, biological and artificial, point to a more fundamental emergent phenomenon of the computational universe we inhabit?

I'm pretty sure if you threw the number of grad students and GPUs at PGMs or various ensemble techniques, the results wouldn't be very different. Fundamentally prediction as we do it is just a form of data compression.

I don't think brains work in a way we understand at all (though pieces like visual cortex are well understood).

Hobbyist here as well.

> As far as I know of (and I could be wrong) - there has yet to be any kind of proven method that the human brain uses any form of back-propagation for learning. From what I understand, there isn't a biological equivalent.

Is there research done on how individual or a small group of neurons learn?

> We also seem to learn things faster; natural neural networks - brains - regardless of species - only require relatively few examples to learn and generalize from, while deep learning artificial neural networks require thousands and more, of labeled examples.

When you take the point of your parent's post, then I'd assume having multiple neural nets learning at the same time and seeing things from different perspectives then we might get there as well.

> We haven't even created the equivalent of a mouse brain, and if we did, it would take the energy output of Niagara Falls to run it.

Our brain is specialized architecture. Emulating a brain on a Von Neumann machine designed computer is similar to a brain trying to emulate a computer. To me it makes sense that you'd need specialized hardware for this.

I wonder if we could simulate a neural net of a simple creature, like a worm.