Artificial Intelligence Uses Videogame Footage to Recreate Game Engine

Tribes 1 was an absolute favourite in my household growing up. It was the first real viable, extendable MMO fighter I'd played. I don't know if I've had as much fun in a similar format (and I enjoy [like the odd ice cream treat] some modern ones like BF1). Maybe it's just sentimentalism. It was the first game I got my younger siblings into, and they became obsessed. More than me... haha...

SHAZBOT!

Having in game mail, IRC, and an actual clan system I think was awesome and ahead of its time as well.

The set of facts is worthless for anything of complexity. It does not really generate the rules itself. (They are directly derived from the facts.)

What they did is only a small improvement over a typical expert system or CNN for a very limited case.

Choice quote: "Notably each fact can be linked back to the characteristics of a sprite that it arose from." Wrong. When you pick up a flower your sprite changes, but how does it know you can suddenly shoot bullets? Etc. And for more complex games a lot of data requires exploration well past the GUI. An action might change acceleration (suddenly nonlinear ice physics with momentum), or direction handling, or you can start flying, or many other things. What if the thing moves in a circle? What if there is just some probability that something results?

The approach will fail at modelling as soon as Mario level 1-4. (The one with rotating fireballs.) Or produce an insane representation of the engine. Note how it even cannot model the dampened triangle wave motion of the fireballs in the example - assumes they're a sparse line.

The paper presents no way to reduce these huge number of "if-then rules" to something actually useful either.

Since this doesn't even attempt to explore the state space, it also requires a huge database.

Calling this "recreate game engine" is akin to saying that since we have an algorithm that can solve checkers, it will solve poker, go and also whodunit. And can play Jeopardy too.

I even suspect it's not useful as a preprocessor to something that can actually play a game, as it will break later cases.

In addition, this is wrong to having been said to be new, such attempts have been made before and even stronger in results and generality. For example this (relatively dumb) approach from 2013 generalized kinda well, much better than I've seen a silly even deep network generalize: http://www.cs.cmu.edu/~tom7/mario/

So yes, they are overselling it a lot. I am 100% not impressed by this paper as it lacks critical detail. That it can parse stuff from 2D frames is not interesting, it is basic motion analysis which can be done even by a supremely stupid algorithm, not even a CNN.

I mean, Google best AI can play 15 rooms of a simple game...

Section 3.1 of the paper outlines a list of 'hand-authored' functions the agent used to derive events from images. They include animation, sprite-entity relationships, motion, collision, and camera movement. Which is to say, every component of Super Mario level 1-1.

That doesn't mean the paper is uninteresting, or useless. Defining facts based on those possible rules is still an intriguing result. I'm having real trouble working out from the paper how well their agent understood conditional changes like size and fire flowers - if it accurately recreated those rules, then I am impressed.

But "modeled without accessing the code" is a dubious claim about an agent that started with a list of the core rules included in its code. The Engine Learning section (3.2) mentions that automatic derivation of possible facts is a key area for future work. That is to say "this would be flexible if it did feature learning instead of needing feature engineering". Unfortunately, that's the problem in agent design, and the value of CNNs isn't unbeatable performance but the capacity for flexible feature learning. The press release here elides the issue of feature learning entirely when comparing performance.

It's waaaayy less impressive if it's just programmatically processing video frames, tracking pixels to generate coalescence, generating a library of sprites based on pixel coalescence, and then playing back the same sequence of sprites programmatically...