Human-level control through deep reinforcement learning

Interesting that they're using Torch7. The code is pretty concise and readable, very cool stuff.

The No Free Lunch Theorems don't apply in reality anyway, as explained at https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...

Essentially: NFL says the performance (1 / number of trials are required to find a solution), when averaged over all fitness functions, will be the same for all optimisation algorithms. However, most fitness functions require more computational power than the Universe to evaluate; making even a single trial impossible to perform.

Since those problems are already impossible, it doesn't matter how many trials our optimisation algorithm would require when tackling them. We can use this unobservable bad performance to offset some observable good performance, to remain within the average dictated by NFL whilst still becoming objectively better.

In fact, one of this paper's authors (Shane Legg) has used this fact to define a general intelligence test which favours performance on simple functions http://arxiv.org/abs/0712.3329

> the algorithm was able to generalize across multiple problem sets

Did it really? I think they reset it and retrained it for each game.

I'd like to know how much more is needed to make one instance of the AI that can successfully play any of the games. To play all 49 games that it could learn, does it need to be an extra level deep? Or 49 times larger? Or 2^49 times more?

This is so cool. I'd love to work on this stuff...

Anyone know how hard it would be for someone who is fairly good at programming (works as a full stack developer and feels quite comfortable learning new things) and has strong math skills (undergrad degree) to break into this field? Is going back to school for a masters/phd the best way?

Only the first page, the rest is blurred and has to be paid for.

I expect it could go far. Mind you I only did parts of Artificial Intellegence: A modern approach but the Q-learning algorthim seems very flexible. https://en.wikipedia.org/wiki/Q-learning It basically keeps doing good stuff, while exploring to get out of local minimas.

Seems to be critiquing the claim that this is "groundbreaking" and not much else. Nice to get some further context though. :)

Yes, I can't access the full paper but at least the figures are different :)

Edit: Ars Technica has a summary of this new paper. https://arstechnica.com/science/2015/02/ai-masters-49-atari-...

Yes, they addressed this on the conference call I was on. That was a sort of 'initial results' they wanted to get out there, this is the full research. Same concept, explored more deeply.

They claim good results on 49 games while the original paper only worked on a few IIRC.

I do not think that it figures out "what the win is" as the score parameter is explicitly made available to the algorithm.

In some sense this paper even demonstrates that how simple the problem actually is.

I think more important question is that what else can be modelled as such problem.

Reinforcement learning is not new. The same algorithm was used in the 90s to beat backgammon. The only notable thing is they are using raw pixels and vision to play the games.

The amazing part of what DeepMind has achieved is its capability to learn from raw pixel input with deep convolutional neural networks, which as I understand it quite different from what you do.

Still the reinforcement learning part is the same, but reinforcement learning was not the main contribution of this nature paper.

They built a system that automatically learned to play 49 different games, without any human tweaking (aside from the original setup).

Here is a very rough translation from my POV:

> The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment.

Reinforcement (rewards/punishments) is a highly effective way to train autonomous individuals to succeed in arbitrary environments.

> To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations.

This kind of open-ended learning in a simulation is hard: Think of the number of inputs from all your senses being continually processed by the nervous system at a given time. Being able to take all those inputs, each of which changes meaning depending on context, to figure out what to do right now (while learning from the past in the process) is a hard problem, especially for a computer to solve.

> Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms.

Humans and other animals have this figured out through our own brains ("dopaminergic neurons") and the combination of our senses and various parts of our nervous systems/biology (e.g. our reflexes respond faster than our cognition due to the hierarchy of autonomic nervous system response).

> While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces.

Past reinforcement-based algorithms have worked well, but require thorough understanding of the problem being solved, or for the problem to be relatively simple and predictable.

> Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning.

By combining advances in deep neural network training with reinforcement learning in a "novel artificial agent" ("a deep Q-network"), our agent can learn sophisticated problems through only reinforcement learning.

> We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters.

Our new approach works across 49 games using the same approach for each game (where each game presumably has different rules and dynamics), and is able to perform at the same level as a professional human being.

> This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

We've created a universal learning algorithm that can take a multitude of inputs and consistently respond correctly without having to re-define the model for each game or problem.

No, this is a new paper.

All sorts, we have a machine learning service which lets you do something similar with JSON messages.

https://aiseedo.com/demos/cookiemonster/

>> I am wondering that what kind of real life problems could be modelled this way.

Driving a car? Making a taco? Working the checkout counter?

The biggest differences are this:

1. The Mario algo has direct access to the game state, and will only work for games where it has that game state access. The DeepMind algo plays directly from the screen pixels. That means DeepMind has to first learn to interpret time varying (!) visual information correctly, then deduce rules and good play strategy on top of that leaky abstraction. That's hard. It also means the algorithm can be applied to any game with a screen output, not just to an Atari.

2. The Mario algo is doing a direct search through move space. It can back up and explore a different branch of the tree and play differently to see a different outcome. When the DeepMind algo plays Atari, it can't undo a move that it just did. It has to make good choices, using intelligence, just like a human player would.

The impressive thing here is not that it plays Atari games. You're right, we have had AIs that can do this for a long time, even better than this. The impressive thing is that it's a single AI algorithm that works for many games, and that is learning directly from the screen. We have not had anything like this before.

That's pretty cool (especially the ending of Tetris!), but the crucial difference is that it's trying to learn by observing what happens during a known-good run, then it sounds like some game-specific learning is performed. In contrast, the Arcade Learning Environment (the benchmark used in TFA) is completely autonomous; there is no human guidance or game-specific learning: anything it learns must work well across all of the games (ie. it's either generally useful, or it gets surrounded by a bunch of conditionals).

In particular, the author of your link says he's surprised that the algorithm can sometimes get further than twice the length of his pre-recorded training sequence.