Why AI lags behind the human brain in computational power

You could write the same article in reverse: how many neurons does it take to simulated an ANN? A lot, because neither system is a good basis for representing the other. You need a lot of tanh functions to fit a spiking neuron and a lot of spiking neurons to represent a single tanh. This fact doesn’t say anything about the “computational power” of either system.

I am all starting to feel like it is borderline malpractice to talk about neuron/silicon performance comparisons without acknowledging that perhaps the driving design constraint in wetware is energy usage, and considering the implications this has for optimal architecture.

Realistic simulation of neurons is expensive. Back in my grad school days we ran Genesis and could afford at most 10k neurons - each neuron needs a lot of work to model the corresponding differential equations. However, it's unclear how to translate this into requirements for artificial neural networks -- the type of computation is too different.

A different metric is a more relevant goalpost -- number of synapses. If each of 125 trillion synapses in the brain can adjust its strength independently of others, it loosely corresponds to a parameter in a neural network. So if we get 100 trillion parameter networks training but still no human intelligence, we'll know conclusively that the bottleneck is something else. Currently training 1T parameter networks seem feasible

This is one reason why I think for more general intelligence we want to pair up more computer-natural computer-friendly modeling mechanisms with leading edge neural network architectures.

For example, for computer vision, leverage efficient 3d graphics techniques or knowledge rather than relying entirely on CNNs (for example). Especially take the case of indoor environments which contain mostly man-made objects that are highly regular. Use neural networks to help detect these shapes and then perhaps iteratively refine them. But that high level structure provides a very efficient framework that significantly reduces the amount of computation required.

Some people working on more general purpose AI such as Tenenbaum have been saying something at least a little bit like that for quite a long time. And it's true that although there is a lot of interesting progress, the gap remains significant between human and AI capability in terms of generalization. But I think if you look at the computational numbers suggested by the type of research in this post, and then look at the progress on generalizing AI, the achievements are not insignificant.

It should be noted that this contains pure speculation on the part of how much of the actual information processing in the neurons is optimally required given the total information processing to be done vs what is required given the biological mashup of hundreds of millions of years of evolution.

Lost me at "Traditional neural nets have two input layers."

I’ve recently been interested in reading about neuronal microtubules contributing to consciousness. It’s a theory put forward by Roger Penrose and Stuart Hammeroff.