Jeff Hawkins talk on modeling neocortex and its impact on machine intelligence
numenta.com
numenta.com
At one point Hawkins even says he's not going to waste time by talking about specific data sets they've applied the software to because there are too many! Then pick one! Pick the best!
That being said, I'm finding it very difficult to find any objective comparisons of these algorithms to other, more mainstream machine learning techniques. In the talk, he gave the impression that there was a tremendous amount of data to back up these claims, and that he just didn't have time to present it all. I went through many of the white papers available on the Numenta website. Many were just overall outlines of the approach. A few of them demonstrate tasks for which some form of learning is occurring, however, it was hard for me to know, in the absence of objective comparisons to other techniques, just how good the results really are.
So far, the only objective comparison I could find involved handwritten character recognition, and that was against what appeared to be a standard feed forward neural network with only a single hidden layer. Not exactly state of the art.. why not compare to SVMs, convolutional NNs, deep belief nets, etc..?
So I am at this point hopeful, but fairly skeptical. If nothing else these are some inspiring ideas.
To put it simply, I strongly feel that achieving any kind of biologically-inspired intelligent agent will require a systems biologic approach where we model every minute molecular detail in silico. This isn't an undertaking that we even have the technology for at present. We don't have the raw speed, level of parallelism, or even the molecular/cell physiologic details necessary to model even parts of the brain. (Even of Drosophila!)
The Blue Brain project is nice and is worth following for nothing other than learning best engineering practices for developing the architecture behind something of this scale and complexity--but every simplification we make introduces error. (Imagine patching it! Imagine the "oops" moment, when some molecular mechanism doesn't work as we expected--and that's a regularly occurring event.) I'm not even sure how much simplification we can make before the emergent properties of the brain no longer function. Some of my colleagues say membrane potentials and the cytoskeletal system have key quantum interactions that encode state information--something we don't even understand yet. (I can't comment much on that, since I haven't studied quantum physics.)
I'm actually learning to develop algorithms that will focus on the interplay of the genomic machinery (promoters/enhancers, tx, translation, chaperones/folding, etc), biochemical pathways and kinetics, concentration levels, receptors, etc. in the hopes that one day we will be able to model systems like the brain. But from my limited knowledge, a project on the brain scale will only succeed after we solve the "much less complicated" problems: cancer, alzheimers, and aging, all of which are all cell-level problems. That's where we have to focus at present--and you can see how much more remains to be done.
Algorithms that replicate biological processes are popular because people can easily grasp them. Anyone can understand why evolutionary systems work, or why neural networks work (because it's in nature, dummy!), but they don't work as well as other, purely mathematical methods.
In the end, these sorts of things tend to be toys or marginally useful, where other, more mathematically sound algorithms dominate the landscape. I got very excited about HTMs too, when I didn't know as much about ML as I do now, but I've realised that it hasn't made even a dent in academic circles (you know, the ones with the thousands of people who study these things for a living).
Neural networks are "universal" function approximators in the sense that given sufficient number of hidden layer units and training data, they can approximate any Borel measurable function. This makes them theoretically powerful. (There are issues with this "universalness" claim though when doing cognitive modelling. See "Cognition and the computational power of connectionist networks", Hadley at http://www.informaworld.com/smpp/content~db=all?content=10.1...)
In practice, the best handwritten digits classifier is still neural networks. Lecun keeps scores at http://yann.lecun.com/exdb/mnist/.
Are there challenges with training neural networks? Certainly. But for many applications, there are good reasons to use a neural network. For instance, inference in a neural network is fast, and the learnt model is small in size (in feedforward networks, just a few large matrices).
SVMs on the other hand are "easier" to train, but care has to be taken to use the "right" kernel, and the learnt model in practice has to, in many cases, carry around with it up to half or more of the training data set (ie, the support vectors), making the model bigger in size, and inference slower (although there is work in making a sparser set of support vectors. See "An Effective Method of Pruning Support Vector Machine Classifiers", Liang, at http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5342...).
I know neural network research is kinda old school these days, has been overhyped in the past, and SVM research is kinda "in" nowadays, but that's no reason to think SVM is "much, much better" than ANN.
Neural networks aren't toys. I've seen billion dollar systems based on them. And I believe an RBM was one of the strongest components of the winning Netflix algorithm.
The ML academics are biased against neural nets because they're difficult to analyze mathematically making it hard to fill blackboards with impressive looking equations.
Until you've discovered how it works you won't know which features of the system are actually essential to its functionality and which are mere evolutionary artifacts which can be drastically simplified.
I think the problem will be solved, like other scientific problems have in the past. People will have ideas about what is essential and they will construct hypotheses and build and test models based on those ideas.
I think that arguing that the problem is too complex therefore there's no point in people having ideas until we can model everything is not a productive approach.
The systems we think are simple are only so because we already possess the knowledge necessary to think abstractly. (Or somebody else does.) Now consider this too: our knowledge of cellular biology is infantile. Can we put the entire brain together without understanding its parts? I bet you we can't.
We have to have a model to start with. In this case I don't mean a computational model that simulates everything--just a mental model. A physiologic model. Something that can tell us xyz happens in response to foo or absense of bar. We've been building that for a while now, but we're nowhere near complete. Not even enough to "wing it".
Complex problems can be solved with adequate information. We, unfortunately, are at a loss for that. (At least as far as applications at this scale are concerned.)
I'll make an informed prediction though: you'll see simulated brains working after we've cured cancer and aging.
If by that you mean that science proceeds by first understanding systems in their entirety and then abstracting away the essential elements from that complexity then I can't agree. (Science education may do that but that's an entirely different matter).
Historically we've gained understanding of systems by creating simple models which capture their essential behavior and only later adding more complex effects. One good example is the dynamics of the solar system. The basic phenomenology of central force motion (ie. elliptical orbits) was understood long before N-body interactions were accounted for. I think you would find similar examples in biology. For instance the "rule" that DNA is transcribed to RNA was established long before exceptions like retroviruses were understood.
The challenge is and always has been to somehow "see" or intuit the fundamental behavior in a mass of confusing data then build testable models based on that intuition.
Flash video is sadly unfriendly to my iPad (where I am now)
Even Hawkins himself has admitted that his hierarchical network design hasn't caused the huge bang that he expected it to cause.