Hacker News Reader

Comment by dontreact | Hacker News Reader

Are we done with games yet? I thought it’s been well shown that you can beat any computer game using RL. Would like to see people move on from these types of things and engage more with the difficulty of solving real world problems with AI.

At this point, I’m not really sure that the progress in playing games is carrying over to much in terms of solving real world problems, but curious to see if anyone has good counterexamples.

View on HN

In theory, with accurate passing / historical data you could build out strategies that could translate to the real world from this setting. Football has two parts to it, one being the strategy aspect of moving the ball and beating the opposition position, and the other of actually executing these instructions physically. While the Google framework lacks on the latter (no sports simulation game is really close to the real world in terms of player models), the former can probably be learnt via RL, and maybe open itself to some neat counters to traditionally established strategy.

There are still lots of things that are more easily answerable in games to see if they exist.

For example, can we get AI to play deceptive strategies without being explicitly rewarded (Open AI hide and seek game, StarCraft fog of war plays, etc)?

As to whether it can translate to solving real world problems is arguable (there are some cases yes), but it definitely helps out in weeding inference models or training strategies that are not viable in games (and probably not the real world).

edit: Forgot to list an example. Using the same reinforcement learning strategies not to play games, but to design them.

Thanks for the perspective. I have been asking this question for a few years now, and I think Deepmind's marketing has been pretty deceptive on this. What examples do you have in mind where the research from playing games with RL has carried over into solving a real-world problem? Probably the state of the art has advanced as I have focused in applying supervised learning on some applied problems that interest me.

"can we get AI to play deceptive strategies without being explicitly rewarded"

Haven't we seen quite a bit of that? I think there have been multiple items on HN about instances where ill-defined goals led to AI finding bugs in a simulation or unintended solutions. Deception is really just about human expectations.

> it’s been well shown that you can beat any computer game using RL.

Starcraft has not been beaten yet. DeepMind's AlphaStar is a grandmaster, but is nowhere near world champion levels.

It's probably not even grandmaster if you restrict it to human-level mechanics. I know what DeepMind has said about APM, but if you look at how it functions it's still superhuman (like the fact that it never uses control groups since it can always arbitrarily select whatever subgroup of units it wants instantly with no mistakes).

Deepmind's AI would be ranked in 5800-6000 MMR range by Blizzard if they would actually use the matchmaking instead of random opponents and a ELO calculation formula which wasn't suited for the circumstances nor used by Blizzard. It's actual win-loss ratios against opponents below and above 6000 were: T — 15-4 and 3-8, Z — 15-8 and 3-4, P — 26-1 and 0-3 (which produced whooping 6352 by formula DM used).

This completely ignores what I just said.

It's really hard to say with AlphaStar, because it depends enormously on what type of APM constraints the developers put on it. As much as I love the strategy of StarCraft, brute force is incredibly important. Serral, the best player in the world at the time by a wide margin, played some fun games against a few lower level European pros controlling the same opponent and it was so lopsided he quit after ~2 games.

If AlphaStar is allowed just slightly too much micro it's easily world champion beating (eg the version that killed TLO in the original demo, it was winning fights that should have been blunders), and too little and it can get squashed by regular GMs.

I don't know how the competition can ever really be calibrated to be fair. The latest version of AlphaStar was fun because they tuned it to be good but not unstoppable, but I have no idea if those were 'fair' settings or not. Maybe AlphaStar was too handicapped.

It's great to see new football strategies. But I agree, personally I'm most excited about proving/discovering new math theorems using ML.

Reply on news.ycombinator.com