OpenAI Retro Contest

This should be required reading for anyone wanting to do this contest.

Of course it's a difficult problem, but why try and dissuade people from attempting it? Nothing novel ever came from somebody who didn't fail first : )

> Current approaches such as DQNs or god forbid DRL

You say that like you're talking about two separate things. DQN is just one type of DRL.

> Two months ... Good luck

Well, it needs to be 2 months so Musk can transfer this learning to Autopilot... /s, I think

Do note that most RL methods including DQN were designed to be very general. While this contest doesn't really require generality. You could design a model specialized to the characteristics of the game. This is what the top winners of the Vizdoom challenge did.

IMO, the best approach would be to jump right in, without looking into what other people are doing. People have tried to solve these problems using the current trend (deep reinforcement learning), and so far it has not worked out. Most such contests are usually won by methods which are somewhat different from the current paradigm. For instance, Imagenet 2012, and to some extent, the Vizdoom challenge was won by a method which departed significantly from the paradigm of the time.

If you have the time to work on this contest, I'd recommend trying a method which is either not deep learning or not reinforcement learning or not both. There will be enough submissions from people who will try obvious ideas, and so you're better off trying something unique. Good luck and have fun!

If you want to do ok and understand what's up, watch the awesome free lectures by Alphabet's DeepMind here (the people at the origin of AlphaZero): https://www.youtube.com/playlist?list=PLweqsIcZJac7PfiyYMvYi...

The challenge is using all 3 Genesis games.

I'd imagine part of any reinforcement learning approach would be randomly tweaking the physics for each run, to make it robust against small changes and errors. Handling the different games (assuming they're similar enough) should just be an extension of that.

No, you're not wrong. That said, I don't think this is a competition aimed at people with a beginner level knowledge of machine/deep learning. I don't believe there is currently any MOOC that is going to hold your hand from 0 to expert. Given the current state of the field, I believe the only surefire path towards this level of knowledge is either university education or a /lot/ of reading.

Anyone have an intuition to the next step after parent?

Acknowledge its a hugely complicated subject.

I think some of the "transfer" part comes from the fact that there are actually three different games involved (Sonic 1, 2 and 3).

Yeah, any side scroller...so dying is bad, but getting hit and shrinking or losing all your rings is not as bad, killing enemies is good, and collecting a power-up is good, but only when it matters.

Anything else?

Like BDD for AI.

Transfer learning is what makes this contest currently unique. Usually in these machine learning contests, you hand off your model to the contest runner, and they just run it and evaluate its performance. It has to be already good at the task.

In this contest, you're not just submitting a trained model, you're submitted a docker environment capable of performing training, which they will run with their secret levels, with specified constraints. So you want to make a highly trained model whose capabilities can be transferred to secret levels.

The loophole suggested in the instructions is to use the ROMs provided by legally-purchased copies of the game from Steam.

The problem is that reinforcement learning is far from solved and doesn't work all that well yet, so these toy problems are probably what researchers will stick with for some time to come.

Video games are a great environment because they are quite complex and they are already coded and we understand them. For basic research it doesn't matter whether is it a game, test cases or 3d design. They use games because it is less work as they already work and are easy to integrante. Moreover each game is a different domain.

> Each timestep advances the game by 4 frames, and each observation is the pixels on the screen for the current frame, a shape [224, 320, 3] array of uint8 values. Each action is which buttons to hold down until the next frame (a shape [12] array of bool values, one for each button on the Genesis controller where invalid button combinations (Up+Down or accessing the start menu) are ignored ... During training you can access a few variables from the memory of the game through the info dictionary. During testing, these variables are not available.

So yep, the RGB image.

It's fair because you get better at gaming in general when you play games. For instance, your performance on Mario will be better after playing a lot of Sonic.

This is in contrast to current models which generally need to be trained up from scratch on each new game, even if the games are mechanically similar.

It's Sonic, clearing it is really easy for humans.

That's what makes this contest interesting, they're not going to just evaluate a static agent, you submit a docker environment capable of performing learning, and they'll let your agent learn on their secret levels, on their servers.

That provides a rather unique challenge, of making a good general baseline agent, but one that also is flexible for further training.

Best score

Check out https://contest.openai.com/details

I'm not sure how different those skills are from an AI development point of view though, why do you think they are?

> winners will receive some pretty cool trophies.

Second to last paragraph:

  The contest will run from April 5 to June 5 (2 months)
  and *winners will receive some pretty cool trophies.*

Yep!