Now, imagine that there was a computer program that makes the same choices you do in the same situations, and the Predictor can read that source code before filling the boxes.
That's the intuition pump I use for thinking about Newcomb-like problems.
The minimal way to think about choice is to require the existence of an agent that is capable of producing binary outputs. Each output signifies a decision the agent has made. The agent is treated as a black box and whether the agent is deterministic or not remains unspecified and irrelevant.
Another approach is to attempt to endow the agents with some sort of free will.
An interesting realization is that there is no way to know whether humans or anything else for that matter qualify for the second option. There may exist a deity, alien or Matrix root who already knows all our choices before we do.
The funny thing is: we couldn't even tell the difference.
But you can only perform B-AB if you manage to fool the predictor and especially you can not do B-AB in a deterministic universe without free will (let's ignore that you don't even have control over the first step in that scenario). B is definitely the right choice in the first step and then changing to AB is the best choice in the second step, but whether you can actually do this depends on your stance on free will and the like.
>(Incidentally, don’t imagine you can wiggle out of this by basing your decision on a coin flip! For suppose the Predictor predicts you’ll open only the first box with probability p. Then he’ll put the $1,000,000 in that box with the same probability p. So your expected payoff is 1,000,000p2 + 1,001,000p(1-p) + 1,000(1-p)2 = 1,000,000p + 1,000(1-p), and you’re stuck with the same paradox as before.)
Even if you have "free will" and your actions are semi-random, that's not different than flipping a coin to make your decisions.
Since you have imperfect information about yourself (this neuron talked to this neuron by these rules) you don't attribute the cause of your own behavior to anything other than "I did that".
I think the example from 1.3 is much more clarifying than the one-box vs two-box problem, there's no "predictor" involved