Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.
Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture.
> A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. http://edge.org/conversation/jaron_lanier-the-myth-of-ai
I fail to see how the first part of this sentence implies the second.
The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement.
Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases.
I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?
- If you prefer 100000000000000000000000000000000000000 specks to 1 torture
- and you follow the axiom that A > B implies x% A > x% B, and value people equally
- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture
- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%
- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day
- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that
I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".
Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:
* some people spontaneously torture others
* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them
* so: eyeball dust storms commencing in 3, 2, 1...
One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.
That doesn't seem to be Yudkowsky's argument.
At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.
When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.
Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?
Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?
The only reasonable outcome in which the dust specks would be worse would be, admittedly, if there's a non-negligible portion of the population that, as a result of getting the dust speck while operating machinery or doing something important like surgery, ends up making some terrible mistake that causes lasting harm to someone or themselves. In that case, the outcome is "hey, some huge number of people got maimed or killed, so the torture of one is OK in comparison."
But my thought is that your line of reasoning would still be held even if all 3^^^3 people were given the dust speck when they are sitting alone in their room away from any potentially-dangerous objects. Is that the case? In that particular case I don't see what the dust speck would actually do to any person in the future (beyond a single sensory experience in the present).
And to answer your question, I wouldn't pay a penny to avoid the dust speck. Maybe I'm just particularly frugal, being a grad student at the moment.
I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects.
But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny.
Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die.
Which do you choose? Sixty years from now, which do you wish you'd chosen?
> I wouldn't pay a penny to avoid the dust speck
Do you mean just your own dust speck? The question was all 3^^^3 of them.
If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)
a) would you press the 7bn specks button for an individual 'reward' of $70million?
b) would you press the N-specks button if given a choice between that and the 1-torture button?
Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.
Example (this is actually true!) -- I have a life-long inflammation in both eyes that started in one eye when I was about 5, and eventually spread to the other.
The specialists weren't able to figure out the original cause; their best guess was basically a speck of dust -- just the wrong speck of dust (maybe with a microorganism in it?), in the wrong place, that my immune system attacked and then just kept on attacking even after the original speck had been gone for years. Medication plus a series of unpleasant surgeries have managed to keep my eyes relatively functional, so far, but others with the same condition certainly do lose all vision.
Obviously this isn't a common result; but given enough specks of dust.... Well, you have to start doing some math.
I think the dust speck argument would be better if first framed as a preference for yourself:
If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.
I think most people would answer no. So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/), that our emotional reaction to things isn't diminished by even infinitesimally small quantities (see also Useless Medical Disclaimers http://lesswrong.com/lw/h4/useless_medical_disclaimers/).
I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.
There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.
And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.
> I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.