Bad Universal Priors and Notions of Optimality (2015)

AIXI is an AI algorithm which is uncomputable, but has interesting theoretical properties, like making optimal decisions across a wide variety of environments. Computable approximations like AIXItl satisfy weaker versions of these properties.

This paper shows that the definition of AIXI and what is "optimal" are 'subjective': they depend heavily on which language(/turing machine) we decide to use. We can choose a language/TM which (a) causes AIXI to perform arbitrarily badly (worse than random); and (b) we can make "optimality" so trivial that it applies to every algorithm.