Preserving Outputs Precisely while Adaptively Rescaling Targets

Slightly off-topic, but David Silver did work on this project and discusses it briefly in his really great reinforcement learning course [0]. It's worth checking out if anyone is looking for a well-paced, well-explained introduction to some of the basic reinforcement learning algorithms and theory.

[0] http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

Edit: The slides are fine but the gem is the YouTube lectures playlist!

Since the Atari era, video games changed significantly in as much as they no longer tend to be built around trying to "kill" the human player and end their game. This structure made sense in a video game arcade (you die, you put in another quarter, the more often this happens without you losing interest the more money the game makes) and it got inherited into most early video games _because_ of the arcades, but it's not actually fun.

In most modern games the human will invariably "win" eventually, so the "try to get most points before dying" approach makes less sense in general. But, _Speed running_ has become a very popular activity even in games you may not think of as being about "speed". I will be interested to see multi-purpose AIs take on speed running for two reasons:

1. The evaluation function is easier for the machine AND the human assessing it to understand, especially for "Any%" runs in most games, where the rule is basically you don't care how the game is supposed to be played, you just want to go from the start to the end in the least time using the controls (e.g. Mario Kart speed runs are often mostly about convincing the game that your shorter route counts as a "lap" of the track).

2. TAS (Tool Assisted Speed-running) is already a thing, TAS players don't compete with live human players, they choose and play back a precise sequence of inputs to cause the game to play out in a particular way, so they can do hundreds of frame-precise movements, or even influence hidden internal variables (with the effect that seemingly "random" behaviour from a game often proves to be deterministic). An AI can do everything the TAS player can, moment by moment, but it reacts to changes, which a TAS run cannot, on the other hand the TAS players often have deep insight into the implementation details of the game which would not be available to a general purpose AI player.

This makes me wonder if games that "require" the player to read the manual first can be played by a similar agent with/without reading the manual.

Or perhaps even more difficult are the games that even humans struggle to play through to the end due to "cryptic gameplay elements" (of which there are a lot of examples in Atari games).