I agree... I feel like advanced games will always need guidance to alter the game such that they AI don't reach some local maxima through an exploit.

This happens all the time when making games for humans and is evident by how many balancing patches are made to new, highly competitive games (such as any Blizzard title).

The next logical step would be for another, impartial AI to observe the games and changing the rules and parameters intelligently as they evolve to guide the player AI toward the actual goal.

So, I'll just dive sideways right into a religious/philosophical thought based on the simulation discussion I've been having all over the place:

A universe-sized simulation built for a purpose, which requires simulated intelligence to carry out that purpose, would almost certainly include a God intelligence to alter parameters and induce suffering/hardship to direct the simulated intelligence toward that purpose.