Comment by formalsystem | Hacker News Reader

Comment by formalsystem | Hacker News Reader

Parent Full thread

I think any game with strategy is going to have to use reinforcement learning to have any hopes of being understood by its players. Concerns around cheesing strategies can be mitigated for real time games using constraints and cheesing in turn based strategy games is a symptom that maybe the game you're working on isn't that strategic.

Ultimately Chess, Go, Dota, Starcraft are compelling games to watch and play because the skill cap is so high. No hand coded AI for such a game could ever prepare you to enjoy playing any of those games professionally or get to a point where you can understand professional play. High level strategy in those games would never even be discovered had we not had a large population of players playing those games for long hours.

Odds are most strategy games won't be as successful as any of the above games but the newcomers could still have compelling strategic elements that gamers could explore had they been able to find other players that are slightly better than they are so they learn but at the same time aren't frustrated by too big a skill gap. A small multiplayer community is why the strategy game market is pretty bleak except for a few large players.

Reinforcement learning can provide a better experience to your players because you the game developer don't need be the world expert at the strategies of your game to create a top level AI. You just make sure the rules of the game are there and self play will take care of the rest. You can constrain the AI such that it's enjoyable by both your newest and most devoted players. You can discover OP strategies and better guide your design process. Excel is great for balancing but if you want to make a game complex enough then balancing will be a tremendous challenge without a larger team.

If you're interested in leveraging reinforcement learning for your game, I run yuri.ai which helps game developers do exactly that. So reach out to me at mark@yuri.ai if I can help!

Thanks for your response, I had actually heard about Yuri before and was hoping you would respond to this post!

I have a lot of questions for you if you don't mind, maybe you don't want to answer any/all of them that's fine.

1. How do you handle constraining the AI such that it's enjoyable? If you read through the comments here you'll notice just about everyone is saying the hard part is not making a good AI but in making a fun AI. Do you know of any research in this area? most examples I see are all about making the best AI possible. It seems like it's really hard to find a good reward signal to use for "fun". Have you seen this [0,1] paper on learning from human preferences? does Yuri attempt to do something similar? It seems like it would be labor intensive for the play testers.

2. Does Yuri make use of imitation learning at all? I.E learning from lots of human data when it's available to bootstrap the learning process?

3. Do you let the game designers impose any structure on the AI? I.E. "I want three phases for the boss where he progressively gets more aggressive" or use RL just for certain small parts of the behavior? A well defined path finding algorithm with the RL deciding where the agent tries to go? stuff like that?

I have a ton of questions about the technical details side of Yuri too, what frameworks are you using, which RL algorithms etc etc, not sure if you are willing to share those or not.

[0] https://arxiv.org/abs/1706.03741 [1] https://blog.openai.com/deep-reinforcement-learning-from-hum...

1. We use a couple of tricks for this, there's ideas from human preferences. There's modeling a reward function to maximize fun instead of pure performance, there's handicaps and a host of other tricks. Different tricks work better for different games but game developers can easily configure them

2. It can, imitation learning turns RL into a supervised learning problem though from my experience it's not really needed. It's more of a if you already have it then you can make the training a bit faster

3. Yes all the RL part becomes tuning a config file where the game developer needs to describe the task it wants to be accomplished

Regarding technical details, the infra is agnostic to the specific deep learning library you'd like to use and the algorithms are chosen based on the types of inputs (continuous vs discrete), size of the space (the end all of strategy games or a casual puzzle game) and type of game (full information vs partial information).

Reply on news.ycombinator.com