Hacker News Reader

Comment by awwaiid | Hacker News Reader

"Previous attempts to combine RL with neural networks had largely failed due to unstable learning. To address these instabilities, our Deep Q-Networks (DQN) algorithm stores all of the agent's experiences and then randomly samples and replays these experiences to provide diverse and decorrelated training data."

... so, made the machines dream. Fancy!

View on HN

For historical background on this part of the algorithm, called "experience replay", see this paper from Long-Ji Lin in 1992:

Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching

http://link.springer.com/article/10.1023/A:1022628806385

as well as his excellent 1993 PhD thesis:

Reinforcement Learning for Robots Using Neural Networks

http://www.dtic.mil/dtic/tr/fulltext/u2/a261434.pdf

Aha, so that's what (human) dreaming is for.

IIRC there is evidence rats replay their experiences, sped up, while they are asleep. Dreaming may be something else entirely though, because my dreams aren't anything like my memories of the day before.

Artificial neural networks can "dream" by predicting what frame it will see next. This is a really cool technique. They've shown slightly blurry videos of atari games being played that comes entirely from the network's dream. With no interaction with the game at all. You can even train the reinforcement learning on the dream sequences and improve it's performance.

But this also doesn't seem quite like what human dreams are. Human dreams are wild and unrealistic, while the NN dreams try to match the training data as closely as possible.

> IIRC there is evidence rats replay their experiences...

https://en.wikipedia.org/wiki/Hippocampal_replay

for anyone that's interested.

Quite a few of my dreams seem to be slightly scary situations (dangerous behavior of other drivers while I'm driving, for example).

This makes me think that we have a "scenario generation" engine as part of our brains, and so we're being presented with novel experiences (often "wild and unrealistic") to react to.

So this seems to be a pretty big step beyond the idea of presenting previously seen situations (which sounds like a really good first step, of course).

You can even train the reinforcement learning on the dream sequences and improve it's performance.

I'm not sure how that would work. Surely you'd be overfitting on your training set by definition?

Reinforcement learning has a problem in that it gets very little labelled data. You may have a million frames, but the only label is the score. Which may only change a few times per game.

Training the net to predict the next frame is sort of unsupervised learning. It can learn the rules of the game without score information at all.

The second thing is that RL is different than prediction. Even if you can predict the next frame exactly, finding the optimal set of moves is still a hard problem. The algorithm needs to learn more than just predicting what will happen, but also what the optimal action is in every situation. That is something that can be practiced in simulations, or "dreams".

i have read somewhere the dreams are for training purposes and present the dreamer situations either from the past or present or combination. that would explain why dreams are mostly dangerous or "wild". this is how our mind tries to figure out how to deal with those situations in the future. good examples of this are same recurring dreams over and over. and also i think there is no sense of time in the brain while dreaming so it does not differentiate childhood "situations" from any current ones. It just keeps fucking trying to solve that damn math exam problem....

i've dreamt up a few possible explanations for dreams

currently my favourite is for converting short term to long term memory

my most vivid dreams are when i experience something new, or have an unexpected reminder about something i last thought about years ago

i imagine the surreal nature of the dreamscape is some consequence of developing neuronal paths

storing a new idea about a boat forms a path to a neuronal pathway referencing fire and your dream has you sailing across a sea of fire

I thought it was for regularization.

The comparison to dreaming reminds me of a comment in Information Theory, Inference, and Learning Algorithms:

"One way of viewing the two terms in the gradient (43.9) is as 'waking' and 'sleeping' rules. While the network is 'awake', it measures the correlation between x_i and x_j in the real world, and weights are increased in proportion. While the network is 'asleep', it 'dreams' about the world using the generative model (43.4), and measures the correlations between x_i and x_j in the model world; these correlations determine a proportional decrease in the weights. If the second-order correlations in the dream world match the correlations in the real world, then the two terms balance and the weights do not change."

next up: make them dream of electric sheep

Let's hope Google doesn't feed it video games like Battlefield, where it learns how to most effectively kill humans.

Reply on news.ycombinator.com