Hacker News Reader

Comment by orthoganol | Hacker News Reader

It appears they are touting 'memory' as the key new feature, but I know at least in the deep learning NLP world there already exists models with 'memory', like LSTMs or RNNs with dynamic memory or 'attention.' I can't imagine this model is too radically different than the others.

Maybe I just feel a bit uneasy with a claim such as:

> We hope DNCs provide a new metaphor for cognitive science and neuroscience.

View on HN

The "memory" in a typical RNN is akin to a human's short term working memory. It only holds a few things and forgets old things quickly as new things come in. This new memory can hold a large number of things and stores them for an unlimited amount of time, more like a human's long term memory or a computer's RAM. It's a big difference, and the implementation is completely different too.

I was not referring to typical RNNs, but LSTMs or RNNs with 'attention'. They are designed to overcome vanishing/ exploding gradient problems and hold arbitrary memory lengths.

They can technically be as long as you want them, but in practice there are still severe constraints. LSTMs alleviate the gradient problems, but you still get real trouble with long-term dependencies.

Alex Graves and some others in DeepMind have focused a lot in the past year or so on developing practical differentiable data structures, so that the LSTM can read and write to an external memory (and save its precious internal state for more immediate needs) yet still be trainable via backpropagation.

The memory provided by an attention architecture is immutable and has only one addressing mode. The memory in this paper is mutable (hence "Neural Turing Machine") and has several different addressing modes.

Basically they're using differentiable memory allocation, which means that training can not just change 'what' is stored in memory but 'where' it's stored.

LSTMs are very different. Think of it this way. LSTMs store information about the current problem you're solving. DNCs store information about the world.

LSTMs are designed to capture long range dependencies, e.g., "this word at the start of the sentence interacts with this word at the end of the sentence."

DNCs are designed to incorporate outside information, e.g., "i happen to know (from background knowledge) that these two people in this sentence are married"

Reply on news.ycombinator.com