Jeff Hawkins on the Limitations of Artificial Neural Networks

"I am certain that HTM networks applied to applications like vision will not exhibit the problems talked about in this paper. Brains won’t either."

Put up or shut up, I say. If you have a model that works better, demonstrate it. ANNs simply crush every competitor in real-world performance.

Modeling brains is a cool idea but it doesn't have any practical advantages unless it improves learning performance or has medical applications. Neither is true of any ANN competitor so far.

Honestly, I think we're much more likely to eventually understand the brain by first focusing on building working AI systems of our own design and then comparing them to the brain (with the help of our AIs!), rather than reverse engineering the brain first and building systems that mimic it.

I noticed that Jeff Hawkins quite frequently lacks details and scientific backing on his claims. I read his book "On Intelligence" and followed his talks and I have been inspired by them because his explanations are highly intuitive but when it comes to demonstrating the awesomeness of his approach it's less tangible.

I have chosen deep learning as the topic of my graduation thesis and lately and I had the opportunity to study the evolution of artificial neural networks. I think deep learning is having it's descending period on the hype curve in an interesting way, it is interesting because the prominent figures of the field are delivering it. If my memory serves me well, this is the third article brought to HN within last month on the topic of "deep learning does not represent how real brain works and is actually unsuitable for artificial general intelligence".

On one of his seminars Andrew Ng talks about "the algorithm": he pointed out that human brain may re-wire itself to handle different tasks, for example one part of the brain which is responsible for sight might take up the task for hearing. The idea of ANN undisputedly inspired from research investigating how brain actually works, and now it has set the state-of-the-art on many areas it has been applied, but to do this we had to resort to "ugly hacks" in the perspective of "the algorithm", to name some of them would be Autoencoders and Restricted Boltzmann Machines. It works, yes it works very well, but we had to pay a price of taking a detour from finding "the algorithm".

In my opinion, such banterings as in OP should not be taken as an aggressive stance against deep learning. Just because it is not "the" method doesn't mean it has no value whatsoever, but it has a point; after the limitations of deep learning is explored well, we will need a new idea to push the field further.

I met with the head of the Data Science institute at NYU a few years ago (not LeCun). I don't remember the specifics of our conversation, but I remember at one point I mentioned Jeff Hawkins (I had just read his book "On Intelligence") and I remember he responded with some comments that were politely dismissive of Hawkins.

Some of the other comments here seem to share the same sentiment. I really loved his book, but I haven't really followed his work for the past few years. Anyone know what he's been up to in ML/AI lately?

I am by no means an AI professional or neurobioligist, but I have read recently that a major problem with neural networks is that they do not have glial cells which, among other things, move synapses around and helps with synapse growth.

There are around 100 billion neurons in the brain and 100 trillion synapses. This indicates to me that synapses may be more important to biological neural networks than neurons are, and yet most ANN research has been focused exclusively on neurons (I once asked my teacher which neurons to connect to each other and recommended using a GA, which is not a bad idea but it indicates to me we have a very poor understanding of how synapses aught to work to make better ANNs.)

I disagree with the overly pessimistic view of this article however. To say that our standard neural networks are not nearly as complex as biological ones would be accurate, but this hardly means that ANNs are doomed forever, we just need more research into the things that matter.

Hawkins has no credibility in machine learning so I don't think people should really care about what he says. To the extent that anything he does say about the field makes sense, it isn't novel. Unfortunately our society reveres the super wealthy and treats them as experts on everything.