This is the group of problems we need to solve in order to design effective government and stable markets.
Is there actually any demand for "effective govt"? (Yes, lots of people claim to want it, but do you really think that there's large scale agreement on what that means?)
In politics, 'effective' really means 'does what I want.' When there is a lot of conflict about what people want, the government is not 'effective' because (A) deadlock is frustrating and (B) yelling louder is a way to get more and intimidate the opposition.
A government is less like a "singular intelligent self-modifying system" and more like a war on controlled burn.
G = SUM(Fi(W)) from i to WorldPopulation
Where W is the physical configuration of matter in the world. Fi is the function associated with citizen i that defines his notion of 'goodness'.
Bad government is one which tries to maximise an alternative function:
G = SUM(Fi(W) * Di) from i to WorldPopulation, where Di is some factor for each individual and where some Di are much greater than other Di. i.e. some individuals have much greater say over the configuration of the world which is chosen...
almost certainly not.
For example, many people think that "effective govt" involves some notion of "justice" and/or "fairness".
One concrete example is Obama's position wrt capital gains taxes. He wants higher rates even if that results in less revenue. (Higher rates with less revenue means that there's less capital gain, which means less wealth produced, aka less total stuff. Since there's less tax revenue, there's less govt spending.)
A significant number think that "effective govt" propagates certain values/behaviors and discourages others.
Yes, there is disagreement on what "justice" and "fairness" mean and there's also disagreement as to the values/behaviors to be encouraged/discouraged.
To think about the world, you must first have a model of the world. Then you reason about the model, finally you take action.
Somewhere in there you have a motivation for following this observe-decide-act loop. Motivation provides a reference point towards which you want the observed system to evolve.
But there's a problem. The easiest way to satisfy the motivation component is to lie to it. Tell it things are just hunky-dory.
Any singularity-style intelligence will necessarily need to be built with some kind of anchoring motivations to try and stop it from getting out of control. But what's to stop it simply lying to itself and ignoring the outside world?
Trying to add a meta-motivation be as realistic as possible won't work, such a system will seize up with analysis paralysis.
One of the things that Hayekian economists argue is that knowledge about reality cannot be centralised; it is unevenly, lumpily distributed across the whole of humankind. No one actor does, and no one actor could, perceive the entire system. But it works, because no one actor has to.
How do you build efficient markets? How do you build an effective AI? How do you design effective distributed systems? How do you build effective languages/models with which to compress the world?
Under what circumstances do such 'meta' searches fail and succeed? It's all beyond me but this is what my nose is saying...