884 karma · joined August 15, 2009
I founded an AI/LLM startup you can check out mieza.ai
I think it only makes sense to think like you are if you've adopted equilibrium assumptions; if you haven't then I find this sort of reasoning to be a conjunction fallacy causing an epistemic closure.
In my opinion if you really care about this topic the right thing to do is ask someone at Twitter when the change was made. Getting more information would make us converge on the true estimate faster than arguing the odds IMO. Feel free to update me with the results if you do end up doing that so I can adjust my beliefs accordingly. I'm not going to try to gain this information, because I don't think the question matters much.
I'll explain my mistake and why I made it.
I think I thought to use the estimate I did because Elon claimed he didn't know. The prior probability of not knowing something in a code base with millions of lines is very high, but contingent on his involvement in the change the prior that he is aware of it is much higher. So I started the estimate attempt with the probability that I thought better predicted the production of evidence claiming he didn't know.
Your point does raise my estimate substantially, but I think it probably raises it less than you would expect. I don't agree with your 1/300M prior, because I'm aware that hot users get special treatment. I've seen Elon's account thrown around in interview-style questions about hot users before and used as an example of a hot account that needs special treatment. This is something I've witnessed, but it wasn't contingent on Twitter being acquired and it happened prior to Twitter being acquired.
I also don't particularly assign high odds to wanting it, based on the evidence that he claimed to not want it implicitly by wanting it removed. I don't think it seems appropriate to get to near certain probability the he wanted it with the evidence being that he stated that he didn't want it. In my view there isn't a compelling reason for him to lie about this. He owns Twitter, so if he wanted them to have his account monitored that would be a reasonable thing well within his authority. If he wanted it, he doesn't need to pretend to not want it in order to appease someone.
It does seem to me that the odds that the change was added in response to someone thinking he wanted it is much higher than 1%.
When you add the additional information that Elon wants the code removed, but existing Twitter engineers think it appropriate to keep this actually increases the probability of it being added by the existing Twitter engineers and decreases the probability it was added due to Elon.
Obviously, these are rough numbers, but hopefully seeing any numbers at all helps you to get an intuition for the math.
You are arguing on the basis of possibility of imprecision in language that the choice to be more precise does not contribute anything. That structure - whether you want it to or not - as a direct consequence of logic applies to every thinker who ever argued for precision due to the possibility of ambiguity. It is an argument against formal systems, programming languages, measurement, and more. Some of the time it will turn out that your conclusion was true. Other times it will not. So the argument structure itself is invalid. Your conclusions do not follow from your premises.
Try your blade - your argument structure - against steel rather than straw. I saw you slice through straw with it. So I picked up the blade after you set it down and tried to slice it through steel. The blade failed to do so. The blade is cheap, prone to shattering, and unsuited for use in a serious contest between ideas.
For what it is worth - I do happen to agree with you that Lanier is making a mistake here. I think it is in the logical equivalence mismatch. He wants intelligence to be comparable to running, not to motion more generally, but since intelligence is actually more comparable to compression we can talk of different implementations of the process using terms like artificial or natural intelligence without being fallacious for much the same reason we can talk about different compression algorithms and still be talking about compression. So instead of trying to argue from his distinction between motion in general and motion in humans, I would think the place to point to for contradiction is the existence of cheetah runners versus human runners. Directly contradicting his insinuation is that we actually do say that cheetah are faster runners than humans.
This isn't an accident of language. An example of an actual accident of language would be giving tanks instead of giving thanks.
Are runners runners? Yes, according to you. A walker is a runner is a missile is a bowling ball rolling between places is light moving through a medium. No, according to a fitness coach, because a runner is not a tank is not a plane. When they say that a person should take up running they don't mean the person should melt down their body in a furnace and sprinkle their atoms into metal which is then pressed into iron plates that are attached to a tank which will then go running.
Sometimes we need to be careful in language. For example, we probably don't want to confuse the process of being incinerated and pressed into iron plates with the process of a human exercising their muscles. The choice to be careful in this way is not an accident of language. It is a very deliberate thing when, for example, John Von Nuemann carefully explains why he thinks the laymen use of the word game has perilous impact on our ability to think about the field of game theory which he starts in his book about the same.
I think you should make your point so as to disprove Nuemann, not pick on the straw man of running. Or you should argue against the use of the term radiator instead of car parts. It will better highlight your fallacy, because with running I have to make your position seem much more farcial then it is. We do gain something from thinking imprecisely. We gain speed. That can really get our thoughts running, so long as we don't trip up, but it calls to attention that when someone chooses to stop running due to the claim that the terrain isn't runnable, the correct response is not to tell them that running is accidental property. It is to be careful as you move over the more complicated terrain. Otherwise you might be incinerating yourself without noticing your error.
H0: Free will predicts in advance of a decision problem that when a decision problem is resolved you cannot always predict the result.
H1: A lack of free will predicts of a decision problem that when a decision problem is resolved you can always have predicted the result in advance of the result, because there was only one possible resolution.
Complete Refutation of H1: The existence of halting problems. A self-reference in a decision problem refutes H1, because the self-reference
Additional strong evidence against H1: All the most successful epistemological frameworks suppose a superposition over positions which on attaining information about the next position resolves, not to a position, but to a new superposition. For example science does this. It believes we don't know for certain the theories, but we conjecture guesses and use observations to test them.
Implication of failure of H1 and the failure to refute H0: There exists a self-reference, a sort of self-concept, which prevents the resolution of a prediction prior to the resolution. Self-reference exists as a thing which must be computed.
Okay. So now we pick back up in the mathematics that deals with self-reference in game theory. In it we find equilibrium concepts which are defined with respect to self-reference and proven relative to self-reference considerations. These proofs show that the solution structure contingent on self-reference is non-deterministic.
Implication: Not only does self-reference exist, but the consequences of self-reference is the use of self-reference to refute self-referential prediction or to support self-referential prediction on the basis of utility of doing so or not doing so.
So now we know a self-reference must exist, we also know it predicts a refutation in some cases of the ability to predict it. We can also see that if the self-reference didn't exist, it would imply that there was no structure which wasn't computationally irreducible like self-reference structures are.
H3: We will see everything is predictable from the information contexts available, because agents don't exist. Example predictions: that all physics will be determinable like the position of a planet is determinable.
H4: We will that some things are not predictable, because of the existence of illusions that prevent resolution and which contain a self-reference consideration.
What we actually observe is H4. We observe H4 in at least one case, so we can't infer from H3 that we can reject H4, because H4 already rejects the safety of H3.
Let me give an example: Some deer in Africa don't see orange. Tigers, to us, are orange. To the deer they are green. The tiger color is decided by a self-reference consideration with the deer perceptual system. The deer is not going to be able to predict the future state of all tiger like we could predict the planets, because the deer is caught in a superposition with respect to a tiger's presence. When it observes a tiger, this looks the same as observing a bush. Therefore, even if it sees a bush, it can't predict that it can always model the consequence of a bush. Therefore, if we see a planet, it is incorrect to assume we can always predict the consequences of seeing a planet.
So now you can understand, if my other comment didn't click, why appeals to illusion are actually evidence for self-reference, not evidence against it. So feel free to read that comment again to see if it now clicks.
There is this story about a village of blind men and elephants. You are like a blind man from that village, saying that the elephant must be abstract because you felt a footprint of the elephant. Try playing with an elephant carving. It predicts footprints when you press it it into the ground. You can totally guess before you touch the elephant on the basis of the carving. The decision to focus on the untestable is a commitment to a hindsight fallacy which experimental investigation would likely refute.
An anti-enlightenment koan could be: The student came to the master and asked, “Why are tigers green?” The master responded, “The deer they are hunting can’t see orange.” The student then asked, “It is not therefore it is? How mysterious and inscrutable your answers!” But from that moment onwards the master was de-enlightened.
The compilation of each possible Play in an Extensive Normal Form Game into a cellular automata can constitute a refutation by counterexample of your claim. It satisfies the agent condition through congruence with the technical language defining the concept of agent in artificial intelligence literature as it relates to agents in multi-agent multi-step decision theory problems. The condition of non-deterministic policies is found and shown to be optimal by Nash's justification for an mixed strategy equilibrium. The underlying mathematics that leads there is proven via Russel and there exists various proofs of the desiderata of probability from others which can begin atop that. Furthermore by compiling the formal system to get a physical system we create the ability for a logical analogy between the formal system and the physical system through Aristotle's analogical congruence concepts. We can count the cell configurations and we can count the transitional structure, but this is mostly a formality since we know through the definition of the compilation that they must agree. As far as I can tell at every single step in the laddering up there are proofs from the basics of logic to the mathematics to the use of probability to the justification through utility theory and even to the computational universality necessary to create the physical embedding of the agent in our reality.
The prior conception of guess appears laughable posterior to computation and comparison with evidence, but is just science prior and seeing it is wrong is hindsight bias posterior. It might sound laughable, but it is actually the case that it is not a laughing matter [1]. In like manner, but slightly different and perilously close to being anti-scientific is that you don't know until the observation whether guesses would go one way or another.
We sometimes don't know what will happen until it does. The best we can do is to make a guess. Sometimes this guess will be very often good, but it is still a guess and we need to be open to the idea that our guess was wrong in some aspects.
Suppose we could have a perfect guess though.
We can use the guess to figure out what would happen and it would always work. Okay, lets use it to determine what happens in the future in which we use it to determine what happens. Do you see the issue here? The guess references itself. So it can't predict the future in advance of it. There comes a point where it is looping and giving you the leading edge of what is knowable given the amount of time that has passed so far.
So we don't have the perfect guess, even when we have the perfect guess. We don't know what will happen before it happens, even if we know the rules that tell us what happens before they happen.
There are a bunch of wierd things that get implied by the math that is consequent to this, especially when you get into transpositional structures, but it isn't silly so much as it is non-sensory.
I mean, it does sound laughable. When Feynman said we guess when we do science and then validate the guesses with experiments people did laugh. Yet this is not because it is silly. It is because the non-sensory cases we can't differentiate from can only be differentiated from the sensory cases via the experiment.
What often happens though is that directly after the moment people pretend that it was all known before hand. It is a tricky moment, the present.
This should be "Eventually it should be abstracted or else you are at risk of being computationally reducible." I was trying to say it isn't computationally reduced in the unabstracted form, but didn't use the words precisely enough.
You have a world state in your mental model of the Turing Machine state, right? That is why you talk about a specific Turing Machine as if you could look at it from outside it. What do you think you get when you pause, not from within the Turing Machine, but in the space of Turing Machines implied by your observation so far?
How many programs do you think I could write which would output "Hello" midway through the computation? One? Two? Could I suggest to you that there are an infinite number of such potential programs? The observation space corresponds with a sort of superposition of potential world states.
It is worse than that though. Your observation space? It contains more statements as more computation takes place. Eventually it has to be abstracted or it isn't computationally reducible. So there are actually more than one observation per information state. When you think about things from within the universe, even though chair is atoms, you are better off having the superposition of chair being highly variable. It corresponds to many different things.
So when you are entering decision problems you are in actuality in multiple potential universes, in multiple potential observation spaces, and you can't even tell one way or another except by expanding the computation.
In order to think about this sensibly you have to think through the consequences of the infinite self-reference. Logic fails here. So does evidence. For example if you observe that your opponent is playing badly, modeling them as a bad player doesn't work, because if you choose to do this, they could exploit the bad model by playing better than that model would predict. So even though the evidence suggests that they are a bad player, the self-reference considerations forces you away from acting as if they actually are.
So lets say you play rock? They play paper. Lets say you play scissors? They play rock. Lets say you play paper. They play scissors.
Lets say you fully determine everything they are going to do using your complete knowledge of physics and you play rock because your model told you that they would play scissors... They will play paper; obviously.
Lets say you anticipate that and you play scissors. Then they play rock. As long as you claim that you can figure everything out, it means they could figure everything out. The harder you push against this, the harder it pushes back. You have to move with the force, diverting it, not fight against it. You've got to take advantage of the inevitability of being unable to overpower, not try to overpower. You have to act as if you can do everything at once, because the undecidability allows you to do so. Not try to determine everything, because if you try to, then you can't, because others try to do the same thing.
Not quite.
The choice of perfect knowledge of state is irrelevant. Even with perfect knowledge you can't, for example, predict the thought you think next. If you did, you didn't predict it, you experienced it, because the very act of predicting it was the act of thinking it.
There are leading edges to the computation. It is a hindsight bias upon thinking them to think they were predictable.
> I definitely think there is probably a contradiction between perfect knowledge and self reference, thus it is impossible to have perfect knowledge of a system from within the system.
This is a lot closer to what I'm actually trying to say. Critically, that leading edge of information is a barrier that is very important and agents can use it to cause other agents to be unable to predict what will happen before they experience it.
> I don't think I understand why game theory is relevant
The reason it becomes relevant is because of optimal decision structure. From the math of game theory we learn that the optimal structure of an agent's decision making isn't to play rock or to play paper or to play scissors. If you choose any of these then an argument from the equilibrium considerations - which is basically an argument from the consequences of self-reference - shows the choice is exploitable. So in game theory we don't select rock and we don't select paper and we don't select scissors. Instead we select a probability vector over those choices.
Basically it turns out that when things become unknowable the appropriate response is to change the level of abstraction over which actions are taken.
Huh. This is interesting. In multiple intelligent agent designs I know how to create this improved lucidity would come as a consequence of estimation of values not needing to be short-circuited due to intractability. One design has this happen because the limit of the monte carlo rollouts converges to the accurate result due to reduced computation time. The other design has this happen because it stops taking the limit with respect to a value near but below one such that the limit of future expectation gets discounted to nothing. A final design, which is far less optimistic with respect to recovery from this, just has literal cached values for end of life scenarios.
It would foreshadow death because the world modeling suggests that death is happening and is still functional. It would produce lucidity because the erroneous heuristic functions wouldn't be relied upon as heavily as the world model's actual results. Or in the final case... it would have been precomputed before reaching that scenario and didn't decay but it was cache lookups.
Do you happen to know if putting the non-lucid people in life threatening and novel such that they would not have anticipated them situations - for example, standing near the edge of a cliff on a windy day - tend to increase lucidity?
Now that I'm thinking about this I notice that free soloists have been claiming that they gain increased lucidity during their climbs. So my model that predicts non-lucidity gaining lucidity actually successfully predicts an improved lucidity in free soloists too, which is not something I realized before I started thinking about this.
I am not a neuroscientist so I don't know how biologically plausible this is as an explanation goes.
It isn't silly, but it is hard to discuss it with people because logic breaks down under self-reference and logical debate has been the most popular mode of our debates since Aristotle. Even empiricism doesn't save from this weakness, because it implies we ought to update based on the observed evidence. The Halting Problem and the Incompleteness Theorems are very important to recognize. They lead to a recognition of the infinite self-reference that occurs quite naturally as physics reaches its limit and is employed to model an agent which uses that model to model another agent that is modeling them. This produces computationally irreducible phenomenon. From there we start to reach into game theoretic concerns wherein evidence denial on the basis of equilibrium consideration becomes normal. When extended to imperfect information settings we end up discovering that non-deterministic policies are optimal. This optimality proof and the guarantee of the ability to confound through undecidability give us a grip on what selection and variation ought to select for through an appeal to the central limit theorem.
The funniest thing to me is that the decision to deny this corresponds with choosing an unfactored and unsimplified representation for reality as being more correct. Yet this backfires in the most beautiful way: it is slower to compute then the factored and simpler representation. Which means it is computationally reducible. Which means other agents can know your output before you do. Which makes you victim to Halting Problem attacks - or rather it makes you determined but your world indeterminable for you. But the even more amusing irony is that the entire reason we even use logic and empiricism is because we recognize the proxy relationship advantage as a function of the proxy not being the actual thing. So it is a self-refuting position, because it tries to reject the underlying motive for both the use of logic and the use of evidence. Which, well, when you see it - now that is rather silly!
What it is on the free will side isn't silly, but non-sense. As in, literally non-sensory. When you really realize that is what is happening though it is a mistake to laugh at it. After all, how many fingers am I holding up right now? You aren't sensing it. Non-sense as a belief about your sensory data is actually quite valid, because you aren't sensing it actually corresponds to your actual states. It is congruent, not in-congruent, with relevant states.
In formal reasoning about this topic we therefore differentiate between three things: world state, observation state, and information state. This is starting to get into the game theory aspects of the problem, which can end up being very motivating when you notice the proof of optimality of non-deterministic policy functions under imperfect information, but if you really want to understand the undecidability you probably are better off checking out something like https://www.wolframscience.com/nks/p750--the-phenomenon-of-f...
In computability theory and computational complexity theory, an undecidable problem is a decision problem for which it is proved to be impossible to construct an algorithm that always leads to a correct yes-or-no answer. The halting problem is an example: it can be proven that there is no algorithm that correctly determines whether arbitrary programs eventually halt when run. [1]
You are predicting my state in advance of it having been achieved. I'm fully capable of intentionally disrupting your prediction, for example, by drugging the nerves in my arms such that they cannot send signals to my brain. Your claim of knowledge is a false claim and I deny it.
This is not a pedantic point. It relates directly to the concept of computationally irreducible systems [2][3][4]. These processes create the condition for non-deterministic outcomes as a consequence of deterministic systems. We then have to ask: since it is possible, should it actually be that way? Which leads to research on optimal solving of games under imperfect information. Nash has shown optimal mixed/impure strategies [5][6][7][8].
> If everything can be explained by a physical processes, then it seems like if you had perfect knowledge about the current state, then you could predict the next, and if you can predict one state...
This is false. Even if you know the deterministic rules of a system it is not the case that you can predict the state of that system [1][2][3][4][5][6][7][8].
> I feel like freewill is a made up idea entirely and while we might have the experience of free will, the reality is different.
Humans seem to struggle with thinking about this for several reasons, but two important ones are that logic breaks down under self-reference and humans are cooperative with each other [9]. The first is a problem because most of our tradition of debate descends from argumentative traditions descending from Aristotle which is logical tradition of debate [10]. The second is a problem because cooperative agents tend to make themselves predictable. They make themselves stand out from "the world" rather than appearing as if "of the world". This is not actually generally the case, but because we have exceptionally capable senses we don't always realize what the actual decision problems really look like and how nice we have it due to our cooperative tendencies. For a more representative example try to spot the predators in the two cited images which pursue the "of the world" competitive equilibrium [11][12].
[1]: https://en.wikipedia.org/wiki/Undecidable_problem
[2]: https://mathworld.wolfram.com/ComputationalIrreducibility.ht...
[3]: https://www.reddit.com/r/philosophy/comments/4lbck4/computat...
[4]: https://www.wolframscience.com/nks/p750--the-phenomenon-of-f...
[5]: https://en.wikipedia.org/wiki/Strategy_(game_theory)
[6]: https://en.wikipedia.org/wiki/John_Forbes_Nash_Jr.
[7]: https://en.wikipedia.org/wiki/Nash_equilibrium
[8]: https://en.wikipedia.org/wiki/Strategy_(game_theory)#Mixed_s...
[9]: https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
[10]: https://plato.stanford.edu/entries/aristotle-logic/
[11]: https://www.boredpanda.com/blog/wp-content/uploads/2017/04/s...
[12]: https://www.researchgate.net/profile/Michael-Kuba-2/publicat...
Is A now deterministic? No, A isn't anything anymore.
Nothing is happening.
A isn't now deterministic; A was determined and now our physics is stopped. Where is the determinism at? Not within A, because clearly during A B wasn't deterministic because of self-reference to C. Not afterward either, because now it isn't doing anything.
It seems like it works, but lets talk about the parable of the time you were given a deterministic dynamics function and you computed it according to what it was because you wanted to claim it was deterministic by treating it as what it was.
So you start calculating B + B + B + BC + BC + BC. And you see me looking at the same problem and idiotic like I am, you see me write down A = A' = BC. Then I get ready to solve it. And you are like, pfft, what an idiot. He isn't even talking about, like, actual reality.
unfortunately, in the calculations that follow, you realize something strange: Though you may B, I C you. And because I see you before you be such that I can determine what C should be such that you cannot see C while you B be, I choose a C for your B so that your B doesn't see C.
Basically, in choosing the slowest path, you also choose to be determinable, but in choosing being determinable, you implicitly make yourself vulnerable to an A' that makes your context - during the process - undecidable.
You thought choosing A saved you, but actually it was your curse:
You let me set A = B + B + B + A' because you didn't want to claim that we could divide A into the pieces which it was in actuality composed. But because it was in actuality composed of those pieces BC is now calculatable by A' such that your conceit is actually what traps you in the non-deterministic perspective. If B was calculated, not via A's conception of B, but A' then now it isn't just B that can't predict C. C probably can't predict B either.
Try considering a Turing Machine which is running several Turing Machines within it. Call it A. Within it are B and C. B gets an input state that is the entire Turing Machine. Meanwhile C gets the output of B. There are multiple B and multiple BC. So we get something like this when we write it out:
A = B + B + B + BC + BC + BC
You might be tempted to say that A = A, therefore A is deterministic. This would even be logically true. However, it is actually far more dangerous than it appears. Why? Well, you aren't A. You are within A. Lets say you are B within A. Does it matter that you know the dynamics function? It is deterministic, but does it follow from it being deterministic that it is deterministic? No! Even though B is within the deterministic system A, B cannot claim that A is deterministic, because B cannot claim that C is deterministic, because C is indeterminable by B through self-reference.
Now the standard mistake is to let B run then let C run then to pretend B could determine C, because now obviously we can tell that A was determined by B because C was determined by B. Which sounds really compelling, but ask yourself this to see why it isn't as great as you think it is: is the universe still subject to a deterministic dynamics function right now? If so, can you tell me what C is, not in the toy problem, but in the real world?
Like, I realize this is an impossible problem: I'm asking you for the current configuration over the course of the next second for all matter in our universe even the parts you can't model because you haven't observed it. But that is kind of my point. The universe hasn't stopped running yet. You can't determine C from the information context of B.
I also think philosophical literacy is less relevant than it was in the past. For example, we are talking about how an agent's decision making algorithm is in actuality. Is philosophy the best subject to discuss this? Control theory, learning theory, game theory, scientific investigation into the brain's processes are all mature enough that their usage produces more rewarding outcomes.
So lets say we see what at first glance appears to be degenerate twitch chat zoomer spam about philosophy sus no cap? Are they really philosophically unsophisticated? Or were they raised in a world of such greater philosophical sophistication that self-replicating knowledge structures - like, say, memes, were something they had extreme and constant exposure to? What if they are engaging in some sort of coordinated omegalul gambit?
I'm joking, but I'm kind of serious.
I often find that explaining observed incompetence with genius works better than explaining observed incompetence with incompetence. So I'm more fond of claiming a problem is hard for good reasons then that people just weren't educated - which is also often true, by the way, just explaining why I ended up putting the emphasis somewhere else.
It really isn't a problem. The analogical congruence holds.
Typically free will is defined according to the action being taken without the constraint of necessity based on a person's own desire.
In the game theoretic model that Nash showed optimal the decision making act that the agent has to do isn't about deciding over actions. It is over strategies. That might be a bit confusing so lets just use an example.
If you play rock paper scissors in the agent's modeling of the problem it isn't modeling it as an action choice of rock, paper, or scissors. It is actually modeling it as an action choice over probability vectors. The action choice of rock corresponds with the probability vector of [1 0 0], paper with [0 1 0], and rock with [0 0 1]. There are an infinite number of different policies, but it turns out that the rational one to pick is [1/3 1/3 1/3] playing each option with equal probability.
Notice that here the agent is taking an action, but not one that is constrained by necessity. Notice that the choice of it is due to the modeling of what is in the preference of the agent. Compare that with the definition of free will. The same thing is happening.
So the congruence does hold, but what makes it appear to not hold is that most people rejecting free will neglect computationally irreducible phenomenon. In actuality, computationally irreducible functions which allow for stochastic signals show up in cellular automata without the requirements of dualism. Selection and variation are then much more then enough to show that surviving agents will protect these information sources so as to make them unobserved, because failure to do that isn't optimal and so agents which don't aren't selected for.
> Ah, this is where the problem starts.
We could try and reject, not the analogical congruence which holds, but the system of using analogical reasoning in the first place, but this goes badly. The first big problem is that all of our knowledge comes via theory-laden proxies and removing analogical validity removes the validity of evidence in general. A stranger result is that compression is justified through analogical congruence. So you can no longer claim to have knowledge over the state and dynamics function, because you have knowledge over the compressed form of it. To claim knowledge, you now need to physically be it.
> Ah, this is where the problem starts.
There are many invisible octopuses. Fish are going to encounter decision problems wherein these camouflaged predators are both on their right and left yet the decision context shows them the exact same thing in both cases. So which should they pick? Always left? Then they always die. Always right? Then they always die. The only winning solution is to pick both, because then sometimes the agent lives to reproduce. How do we pick both? Well, if we do it in a way that is computationally reducible then the octopus which observes the fish invisibly can anticipate it. So now it is wherever the fish decides to go. So it has to decide to do both, but it has to do so in a way that is unobservable. This decision problem doesn't go away when someone chants the magic word of dualism. The agent still needs to decide in an unpredictable manner or is going to die. So when optimization processes build things? They end up approximating a solution to this problem.
This type of decision problem has played out billions of times. It has played out over billions of years. I don't know which time it happened which was the first, but where it was, that is where the problem really starts. And since that problem and onward a filter has been killing the things that answer incorrectly.
Why difficult?
Discussion that is considered rational culturally descends from argument techniques rooted in Aristolean ideas about logic. However, as shown by Kurt Gödel formal systems break down under self-reference. Through Turing we see can see this type of thing starting to happen when two Turing Machines need to predict the output of each other.
Through John Von Nuemann and Nash we can get a proof of non-determistic policy functions in multi-agent decision problems being optimal. So we find an analogous concept to free will and we find it justified in this abstract domain not on the basis of some wishy washy concept, but on the basis of analogical reasoning (logic) and causal outcome modeling (dynamic functions).
So what do we reject to claim it is an illusion? Logic? Or causality?
"To deal with hyper-planes in a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly. Everyone does it."
– Geoffrey Hinton, A geometrical view of perceptrons
Well, you actually can come up with Norvig's spellchecker very easily with a similar trick. Just tell yourself: to deal with a really hard problem, just argmax with an okayish heuristic. Everyone does it.
Solutions like this seem impossibly elegant because they tolerate a whole lot of error, but they are actually extremely obvious when you tolerate error.