So my impression of the way that Face recognition works is that the neural-net part of the system recognizes the faces in a photo, recognizes the features on the faces and provides a map of the features to the system. The system takes these parameters and stores and recovers a given face with them. The ability to recognize individual faces comes because conveniently these parameters don't change much (but clearly it hinges on this). Of course, there are a lot of small enhancements to this process and these evolve but this is still basic outline.

Which is to say that convolutional neural networks, the most common and successful neural networks, fundamentally only operate by recognizing a broad category (face, nose-on-face, mouth-on-face). This limitation seems as inherent in NNs operating with a black-box/training-set/testing-set cycle as it is in any architecture consideration. If all you can "tell" the NN is "this is X, this is not X", then it's plausible that's all it will give back. Moreover, "X/not-X" testing allows a LOT of training to happen in an unambiguous fashion, which seems necessary when what you're doing gradually pushing a giant bunch of semi-arbitrary detectors into place.

Machine learning theorists can do a lot with nothing but this category-recognition tool - like having a go program recognize "good moves" and "good positions". Note, that the more elaborate "neural net within neural net" approaches involve vast levels of computing power - you're brute-forcing brute-force. AlphaGoZero probably involved the largest level of computing power ever harnessed for a single-use app. AlphaGoZero involve megaflop levels close to those estimated for the human brain though such estimates are always debatable (perhaps why we didn't see many games by it).

The thing is there are plenty of ordinary computing algorithms do "more" than neural nets - perform logical operations or sort and search by vague similarity or classify on spectrum of multiple qualities (rather than the binary X or not-X of a neural network). It is just that these break down at an "industrial scale", whereas neural nets are shine at this scale.

However, despite these virtues, it seems likely we'll see neural nets approaching their particular limits reasonably soon, if they haven't (in at least a sense) done so already.

And it seems like any "real" extension/alternative will have to be able to use big data but use it in a more sophisticated fashion than neural nets.

Oh, this relates to the video most in that when Bengio talks about how to extend neural networks, he's mostly just mentioning extension in terms of how to add abstract qualities like "causation", where one can certainly see crude, simplistic things in computation that could be called "working with causation" but you don't have that yet when you go to the level of a large-scale neural network.