Facebook's Memory Networks Demo

Indeed, another weakness is that multi-layer perceptrons and their deep counterparts are susceptible to "catastrophic forgetting" -- training away old memories when new training data is presented. This is one reason why it's necessary to repeatedly run through the training data. This is a result of the way learning works in these neural networks. Like you say, humans have the ability to learn associatively in one-shot, a very different learning method.

There are neural networks like Grossberg's Adaptive Resonance Theory (ART) which do not suffer from this flaw.

I could draw an analogy to short-term memory where a new fact is stored separately from the big neural network and it is gradually self-taught to deeper and deeper levels of the network as long as it keeps making sense and fits into the existing set of knowledge. This is much like an allocation sinks deeper in the garbage collector's generations.

Maybe layers of neural networks with different levels of conservation and adaptability would model it better. The first layer just sticks with whatever you give it. If the bit of information seems to work all right, this topmost layer would eventually keep teaching it to a more conservative level below where it takes more time to "learn" and adapt to the new fact. Eventually the information will, in time, reach the "big net" and carve a "memory" for itself there. But the initial trigger was just to push a new fact to the topmost layer, and let it work it out by itself.

That's how human brain works on some naive level. You have the active, conscious mind that can just decide to know or remember something. But that knowledge is brittle: it is easily forgotten unless reinforced several time and the brain can't integrate the new piece of information in its contextual processes that help it generate answers based on deeper analysis, conceptual intercorrelation, and what we call intuition.

The next level is sort of an intuitive insight that you get if you study new information long enough. You just get it and you can start reasoning about it without going through all the steps involved in a strictly conscious way. But it's still like playing with Legos: you know what you're doing but you still need to move the pieces yourself.

The final level is the subconscious where information ends up codified in a way that allows us to go on autopilot. It just happens, you just know things. It's like being able to build a Lego space ship by thinking about one.

For example, let's take driving.

First stage: you're in the driving school and you're overwhelmed by the hundreds of little details every intersection or patch of road offers. Traffic seems chaotic and you only manage to get by if you follow pre-set traffic laws and driving instructions. This is on the conscious level in your active mind.

Second stage: the first year after you received your license. You already have some intuitive grasp on how the traffic works and you think hey this is easy but you still bump into unexpected things once in a while. You might read the lanes and intersections and signs all right but you're surprised by other motorists or events that happens in traffic. This is the level where you've internalized a lot but you're still learning. Many people stop at this level.

Third stage: seasoned driver, typically with several hundred thousand kilometers under his/her belt. Doesn't think about the minutiae of driving but pays attention to "high-level" things such as reading the traffic ahead, identifying cars that might pose a danger of unexpected maneuvers, and adjusting to the average traffic flow and going against the waves of traffic jams instead of echoing and amplifying them.

Another example. I've seen this happen many times eventhough I'm not quite a good mathematician myself. If you read thousands of pages of mathematical papers they first demand a lot of energy to decipher and understand. You'll eventually develop an intuition for how to read the proofs, formulas, and concepts and also for what is expressed between the lines. But the best mathematicians don't just merely think in terms of mathematics but, for some things closer to their area of expertise, they'll just know how everything maps out in relation to the new information.

Memory Networks: http://arxiv.org/abs/1410.3916

It uses a class of RNN they call a MemNN to do something that looks a lot like inference using phrase embeddings.

It's a pretty revolutionary approach - no one seems to have tried anything very similar since the 1990's. The related work section[1] is worth reading.

Interestingly, the Jeff Dean RNN slide deck mentions using RNNs for doing QA, but doesn't expand on what work Google has done in the field. I don't have a URL to hand for that.

They do experiment with loading the MemNN from a set of statements stored as triples, which looks a lot more like a traditional Knowledge Base.

[1] 4 in http://arxiv.org/pdf/1410.3916v8.pdf