As hugh3 mentioned in a sibling comment (http://news.ycombinator.com/item?id=2479211), 'making "optimal decisions" in some defined state space where the quality of various options is evaluable is a really different problem to general intelligence'. While I definitely agree with this statement to some extent (namely, a powerful MCAIXI setup is not necessarily going to display any intelligence that's remotely human, at least without a lot of other stuff going on in the system), the concerning thing is that it should almost certainly be enough to get a system reasoning about its own design, since its code is a well defined state space where quality is evaluable (depending how the programmer decides to have it evaluate quality).
To end up with a dangerous runaway "AI" on our hands, we don't need AI that we'd consider intelligent or useful. All it takes for a runaway is an AI that is good at improving itself, working effectively at optimizing a metric that approximates "get better at improving yourself". AIXI approximations should be plenty powerful to do this with the amount of computing power we'll have in ~20 or 30 years (at the very least, there's a big enough chance that we really have to take it seriously).
This is one of the reasons Eliezer Yudkowsky is so keen on extending decision theory, so that we can get some idea what we should be actually be trying to approximate in order to have a decent shot at doing self-improvement safely.
The best way to sum up my concern is that (unboundedly) self-improving programs make up a tiny fraction of program-space that we can't quite hit with today's technology. Of that sliver of program space, there's a much smaller sliver that contains "programs that won't kill us." There's another sliver that contains "programs that have useful side effects". We need to make sure that the first "AI" that we create lies in the miniscule intersection, "self improving programs that do something useful [1] and won't kill us", and that's a terrifyingly small target to shoot at, so we had better work strenuously to make sure that when once it's feasible to create any of these programs our aim is good enough to hit the safe and useful ones.
[1] We need to find self improving programs that are useful early on because we'll need to use them as our "shield" against any malicious self-improvers that will inevitably be developed later. There's a significant first-mover advantage in AI, and even a small head start would probably make it difficult or impossible for a second AI to become a global threat if the first AI didn't want to allow it.
For practice, there have been related ideas from D. Bertsekas and R. Rockafellar.
For actual computing, the problem remains the curse of dimensionality: This curse is so bad that, for a brute force approach, which is what AIXI is, or really nearly anything general in stochastic optimal control on big problems, a few more decades of Moore's law still won't scratch the surface.