Deep Reinforcement Learning
deepmind.com
deepmind.com
... so, made the machines dream. Fancy!
Artificial neural networks can "dream" by predicting what frame it will see next. This is a really cool technique. They've shown slightly blurry videos of atari games being played that comes entirely from the network's dream. With no interaction with the game at all. You can even train the reinforcement learning on the dream sequences and improve it's performance.
But this also doesn't seem quite like what human dreams are. Human dreams are wild and unrealistic, while the NN dreams try to match the training data as closely as possible.
This makes me think that we have a "scenario generation" engine as part of our brains, and so we're being presented with novel experiences (often "wild and unrealistic") to react to.
So this seems to be a pretty big step beyond the idea of presenting previously seen situations (which sounds like a really good first step, of course).
https://en.wikipedia.org/wiki/Hippocampal_replay
for anyone that's interested.
I'm not sure how that would work. Surely you'd be overfitting on your training set by definition?
Training the net to predict the next frame is sort of unsupervised learning. It can learn the rules of the game without score information at all.
The second thing is that RL is different than prediction. Even if you can predict the next frame exactly, finding the optimal set of moves is still a hard problem. The algorithm needs to learn more than just predicting what will happen, but also what the optimal action is in every situation. That is something that can be practiced in simulations, or "dreams".
currently my favourite is for converting short term to long term memory
my most vivid dreams are when i experience something new, or have an unexpected reminder about something i last thought about years ago
i imagine the surreal nature of the dreamscape is some consequence of developing neuronal paths
storing a new idea about a boat forms a path to a neuronal pathway referencing fire and your dream has you sailing across a sea of fire
Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching
http://link.springer.com/article/10.1023/A:1022628806385
as well as his excellent 1993 PhD thesis:
Reinforcement Learning for Robots Using Neural Networks
"One way of viewing the two terms in the gradient (43.9) is as 'waking' and 'sleeping' rules. While the network is 'awake', it measures the correlation between x_i and x_j in the real world, and weights are increased in proportion. While the network is 'asleep', it 'dreams' about the world using the generative model (43.4), and measures the correlations between x_i and x_j in the model world; these correlations determine a proportional decrease in the weights. If the second-order correlations in the dream world match the correlations in the real world, then the two terms balance and the weights do not change."
For example, Breakout saves score in address 76 and 77. Arcade Learning Environment has code to read the score, one per game. Code for Breakout is here: https://github.com/mgbellemare/Arcade-Learning-Environment/b...
The reward seems to be only the game score which I believe is meant to be problematic for this game because your score doesn't go up very often (so you have to perform a lot of actions to get any feedback)? The lives are recorded but aren't part of the "get reward" method...are the lives factored into decision making somewhere else? Seems like knowing you just lost a life would really help decision making for such a game.
EDIT: It could be that the agent will just stand there the first few plays around, enjoying the novelty reward gained from simply watching the timer tick up. Haha
Anyway, it would seem to me that novelty functions that would allow the agent to ignore periodic changes in state such as timers going up, can be quite simple. A function that estimates novelty of individual bit values in the bitstring of the gamestate and then aggregates it, could quite easily account for timers, or generally elements changing periodically regardless of agent actions. A baseline novelty reward would seem relatively easy to predict by the agent and thus result in low prediction errors and low reinforcement of actions by agents. This function would have a linear space and time complexity to the length of the gamestate, and fairly naive & simple to use, but would get the job done I think?
P.S Just wanted to thank you for the work you've put into your website, it's very informative and always a great starting point to dive deeper into topics you cover!
'gamestate' is illegal. It's pointless to suppose an agent which has access to the true groundtruth RAM of the Atari games, because that generalizes to vanishingly few other domains. The goal is to create a general agent which can be used elsewhere, such as in recommender systems. (And if you did have access to the raw RAM, that would reduce the problem from an extremely challenging POMDP or harder, to a fully-observed deterministic MDP, because you could then construct a game-tree of each individual RAM state and the possible actions taken in it; in which case, you would use a much faster and more powerful MDP solver like MCTS rather than DQN.)
One can come up with hand-crafted heuristics which might improve over the naive bitstring equality approach, but your suggestion still doesn't do the trick, assuming you could figure out how to meaningfully define 'periodic changes' and teach the NN to ignore them. Imagine a game in which the overall screen lighting varies (perhaps it's set at night or during rain, or perhaps each level has different color themes). As all the bits keep flipping with changes in lighting/intensity, you'd be in about the same place.
[1] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
[2] http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_...
This would be really interesting to see. I'm curious if Google would avoid this game though since that the media would likely have a field day reporting about a murderous Google developed AI if they did this. Technically we already have this in games anyway though (e.g. bots in FPS games).