Reward Is Unnecessary [pdf]
tilde.town
tilde.town
So a perfect world model is enough to win at reinforcement learning. Can you show that if you maximized reward in some RL problem, it means you necessarily built a perfect world-model?
[1] A Theory of Universal Artificial Intelligence based on Algorithmic Complexity, 2000 - https://arxiv.org/pdf/cs/0004001.pdf Side-note: since building a perfect model is uncomputable, this is all a theoretical discussion. The paper also discusses time-bounded computation and has some interesting things to say about optimality in this case.
No? Because maximizing reward for a specific problem may mean avoiding some states entirely, so your model has no need of understanding transitions out of those states, only how to avoid landing in them.
E.g. if you have a medical application about interventions to improve outcomes for patients with disease X, it's unnecessary to refine the part of your model which would predict how fast a patient would die after you administer a toxic dose of A followed by a toxic dose of B. Your model only need to know that administering a toxic dose of A always leads to lower value states than some other action.
I think a "perfect" world model is required by a "universal" AI in the sense that the range of problems it can handle must be solved by optimal policies which together "cover" all state transitions (in some universe of states).
For those who don't know, a tilde server is a community-operated server distributing shell accounts (via SSH) to its members, and sometimes other services. See tildeverse.org for a small federation of such operators.
I like my text to fit my screen/window rather than an arbitrary piece of paper.
In an era of ultra-conformity on the web i find it refreshing to see that some people still use HTTP as a means to share documents of their choice, not just single-page applications.
For example to count how many bananas are in a box properly, without weird edge cases, you need to know what a banana is and what it looks like and what a box is and what it looks like.
When speaking about AI as far as I understand it should be human like in order to be called artificial intelligence. If it mimics human intelligence and solves trivial problems then it is nothing than calculator with primitive intelligence that I mentioned before.
The thing you are talking is evolution, you acquire model of the world during millions of years of the evolution step by step but today computer scientists are trying to hardcode human like intelligence into computers which is extremely hard taking in consideration how long it took for humans to become highly intelligent. I think better approach would be to write evolutionary algorithms which perhaps can yield human like intelligence.
There is some secret ingredient in human evolution which made us the most intelligent specie on Earth no specie is even remotely close to be as intelligent as we are. I think nobody really knows what that secret ingredient is.
I think anatomy of humans played the crucial role in our evolution. Hands are powerful tools which enabled us to create and develop other tools and technologies.
Whilst physics can be modelled, and hence kinematics and dynamics can be modelled, intelligence, in the human sense, is different. Intelligence, for humans, is sociological, and driven by biology.
Computers cannot parse culture because culture is comprised of arbitrary, contradictory paradigms. Cultural values can only be resolved in an individual context as an integration of a lifetime of experiences.
Computers cannot do this because they cannot feel pleasure or pain, fear or optimism, joy or sorrow, opportunity, disinterest or attraction. They cannot grow older, give birth, or die. As a consequence, they lack the evaluative tools of emotion and experience that humans use to participate in culture.
But wait, you may protest. Computers don't need to feel emotions since these can be modelled. A computer can recognise a man pointing a gun at it demanding money, which is as good as the ability to feel fear, right?
A computer can recognise faces, so surely its only a small step further to recognise beauty, which is enough to simulate the feeling of attraction, right?
A computer won't feel sorrow, but it can know that the death of a loved one or the loss of money are appropriate cues, so that is as good as the feeling of sorrow, right?
The limitation of this substitution of emotions with modelling is that the modelling, and remodelling has to take place externally to the computer. In biological organisms that are in the world, each experience yields an emotional response that is incorporated into the organism. The organism is the sum of its experiences, mediated by its biology.
Consider this question: In a room full of people, who should you talk to? What should you say to them? What should you not say?
A computer can only be programmed to operate in that environment with respect to some externally programmed objective. e.g. if the computer were programmed to maximise its chances of being offered a ride home from a party, it might select which person to talk to based on factors such as sobriety, and an observation of factors indicating who had driven to the party in their own vehicle.
But without the externally programmed objective, how is the computer, or AGI agent to navigate the questions?
Humans, of course, have those questions built-in to the fabric of their thoughts, which spring from their biological desires, and the answers come from their cumulative experiences in the world.
Eh, GPT-3 is pretty good at imitating (some parts of) culture already.
> A computer can only be programmed to operate in that environment with respect to some externally programmed objective. e.g. if the computer were programmed to maximise its chances of being offered a ride home from a party, it might select which person to talk to based on factors such as sobriety, and an observation of factors indicating who had driven to the party in their own vehicle.
Have you ever tried debugging a program? Programs can do stuff that's really hard to predict, even if you wrote them specifically to be easy to predict (ie understandable).
1) multiplayer games with "bots"
2) all these things, on a lower level, serve to make groups of entities communicate and cooperate. Even in solitary animals like cats these emotions serve to facilitate cooperation, to produce offspring or share territory optimally. There is no problem with creating that artificially: just have multiplayer environments with multiple artificial entities cooperating, resource constraints, "happiness", "pain" and "sorrow".
It's going to take a while before we see these entities compose poetry when their mate dies, but it'll go in the same direction.
Solving intelligence is a highly complex problem, in part because it is nearly impossible to get any significant number of people to agree about what intelligence actually means. We eliminate this dilemma by choosing to ignore any kind of consensus, instead defining it as “the ability to predict unknown information given known information”.
To put it more put it more simply, we define intelligence as a model of the world.
So while I'd agree that a world model is necessary, I seriously doubt it's sufficient for anything that I'd call intelligence.
Even if it was (which I doubt, except implicitly, which is not the same thing), there can be no planning without a goal.
It's a kind of mirrored Chinese Room fallacy: In that case, the complaint is that the performance of the system cannot be ascribed to any distinct part of the whole, concluding that the whole cannot perform. In this case, the performance of the system is falsely ascribed to one distinct part, ignoring the contribution of the other.