Show HN: AIsaac – Learning to Beat Bosses. Live Reinforcement Learning on Twitch

This has been my main side-project / hobby for the last few months. I am building a framework to facilitate the creation of agents that can interact with games.

You can think of it as a minimal, one-man OpenAI Universe. The main difference is that you can set it up to interact with native games you own. No VM + VNC setup or depending on available games/environments like Universe! It takes less than 10 minutes to write a plugin to add support for any Steam game.

The framework provides 2 major features: First, it sets up the game agent loop for you. Frames are captured and fed to your agent, you process them the way you want (can be anything from pure computer vision to reinforcement learning) and are able to send actions back to the game. Second, it ships with multiple blackbox implementations of algorithms that are compatible with the architecture of the framework. This second part is a total WIP.

On stream right now, I set up an experiment in The Binding of Isaac: Rebirth to see if we can learn to defeat bosses in game using the DQN family of algorithms. The current version uses 2 DDQNs each using Prioritized Experience Replay. 1 controls the movement and the other is in charge of the shooting. This is already v11 so there are a lot of iterations and chat has been great for discussing possible improvements.

Code is in Python. Linux only for now but Windows support is on the roadmap. Pre-release repo: https://github.com/SerpentAI/Serpent

Happy to answer any question!