Cracks in Reality: How our Systems Fool Themselves
ieet.org
Third installment in the series of articles that ask if intelligence is self-limiting because motivation ultimately undermines survival.
ieet.org
There is a one nice way around the intractable problems of maintaining a huge model of the world: don't. Rodney Brooks' famous saying: "the world is its own best representation." And it isn't necessary or usually even helpful to translate all incoming data into some propositional model. Again: what is your use case? If you are just trying to satisfy some essentialist intuition of what it means to be intelligent and what 'must be' inside minds, then you will be lucky to ever get anything meaningful done.
If you want to write an agent to do something without ongoing guidance, and you have not thought out the task from the beginning to get a specific algorithm, do not start by making a monolithic program that makes lots of vague high-level decisions like "how much reality" based on abstruse constructions of data. Start with the data which is always available, processed minimally, and see how little you can do. Implement walking before you implement steering and implement steering before you implement path-finding.
This essay was just a little too unfocused for my tastes. As an observer of many types of organizations of people, the predilection to spin is endemic. It's not just "in extreme cases"
I scanned the article and kept trying to find some meat, some kind of clear and forceful statement that the author was supporting. Instead it seemed like he was saying a lot of good things, but lacked the intestinal fortitude or insight necessary to actually get anywhere. I was left with a version of "it may be a property of monolithic systems that their motivations are at odds with an accurate perception of reality." That's almost useful, but not quite. There are a lot of interesting places to go from that statement, but we didn't go there. Hell we could have started deconstructing Aristotle and talking about how the tension between the usefulness of categorization versus all abstractions are leaky creates holes in what is knowable. Or we could have moved forward to sociology and discussed the role of the individual in large groups. Or we could have talked AI. Lots of cool places to take this train of thought.
That's a shame. It's obvious the author is well-educated and has thought this through. Sure wish he would have taken us somewhere a bit more interesting.
all of the exciting stuff isn't in the fundamentals and principles, it's in the exact details of how you train model X to do task Y, and few practitioners are going to give away information that's so commercially valuable.
To think about the world, you must first have a model of the world. Then you reason about the model, finally you take action.
Somewhere in there you have a motivation for following this observe-decide-act loop. Motivation provides a reference point towards which you want the observed system to evolve.
But there's a problem. The easiest way to satisfy the motivation component is to lie to it. Tell it things are just hunky-dory.
Any singularity-style intelligence will necessarily need to be built with some kind of anchoring motivations to try and stop it from getting out of control. But what's to stop it simply lying to itself and ignoring the outside world?
Trying to add a meta-motivation be as realistic as possible won't work, such a system will seize up with analysis paralysis.
One of the things that Hayekian economists argue is that knowledge about reality cannot be centralised; it is unevenly, lumpily distributed across the whole of humankind. No one actor does, and no one actor could, perceive the entire system. But it works, because no one actor has to.
How do you build efficient markets? How do you build an effective AI? How do you design effective distributed systems? How do you build effective languages/models with which to compress the world?
Under what circumstances do such 'meta' searches fail and succeed? It's all beyond me but this is what my nose is saying...
Is there actually any demand for "effective govt"? (Yes, lots of people claim to want it, but do you really think that there's large scale agreement on what that means?)
In politics, 'effective' really means 'does what I want.' When there is a lot of conflict about what people want, the government is not 'effective' because (A) deadlock is frustrating and (B) yelling louder is a way to get more and intimidate the opposition.
A government is less like a "singular intelligent self-modifying system" and more like a war on controlled burn.
G = SUM(Fi(W)) from i to WorldPopulation
Where W is the physical configuration of matter in the world. Fi is the function associated with citizen i that defines his notion of 'goodness'.
Bad government is one which tries to maximise an alternative function:
G = SUM(Fi(W) * Di) from i to WorldPopulation, where Di is some factor for each individual and where some Di are much greater than other Di. i.e. some individuals have much greater say over the configuration of the world which is chosen...
almost certainly not.
For example, many people think that "effective govt" involves some notion of "justice" and/or "fairness".
One concrete example is Obama's position wrt capital gains taxes. He wants higher rates even if that results in less revenue. (Higher rates with less revenue means that there's less capital gain, which means less wealth produced, aka less total stuff. Since there's less tax revenue, there's less govt spending.)
A significant number think that "effective govt" propagates certain values/behaviors and discourages others.
Yes, there is disagreement on what "justice" and "fairness" mean and there's also disagreement as to the values/behaviors to be encouraged/discouraged.