Rationality: From AI to Zombies
intelligence.org
intelligence.org
At the heart of the book is the assertion that rationality is not about justifying why we are right, but thinking how we might be wrong, and paying careful attention to any evidence that puzzles us or could convince us to change our minds. The resulting philosophy is a combination of the falsification principle and the scientific method, applied to your own character. Highly recommended.
I look forward to getting some of these books on my shelf.
"Today I would write more courteously, I think. The discourtesy did serve a function, and I think there were people who were helped by reading it; but I now take more seriously the risk of building communities where the normal and expected reaction to low-status outsider views is open mockery and contempt.
Despite my mistake, I am happy to say that my readership has so far been amazingly good about not using my rhetoric as an excuse to bully or belittle others. (I want to single out Scott Alexander in particular here, who is a nicer person than I am and an increasingly amazing writer on these topics, and may deserve a part of the credit for making the culture of Less Wrong a healthy one.)"
If you're not sure where to start, a number of his writings have been received well on HN: https://hn.algolia.com/?query=slatestarcodex&sort=byPopulari...
Does anyone know whether that sequence has made it to one of these books?
There is a long list of alternative explanations[0], with very little evidence to lend support to or contradict any of them (at least, any of the ones that are still around. Some have fallen by the wayside).
[0]: https://en.wikipedia.org/wiki/Interpretations_of_quantum_mec...
That all being said, no one really disagrees about the predictions of the math, and at least many-worlds has less completely ridiculous misinterpretations by lay-people, so both ends of the spectrum are tolerable to me.
Regarding the LessWrong/OvercomingBias version, I must say that you should take the blog posts with a huge, gigantic, over-sized helping of salt. Speaking as someone who majored in physics because I love the field even though I knew it wouldn't help my comp sci career, Yudkowsky does not have the necessary background to speak with authority on this topic and it shows. He makes a few outright mistakes and a lot more overly broad statements, he ignores many validly competing alternative explanations, and as a result the grand philosophical conclusions reached are left unjustified (and as an editorial aside, probably bunk IMHO).
The important result of analyzing quantum mechanics as a case study in rationality is the valid point that if like most people (and sadly, most physics professors) you think quantum mechanics is weird, that should be very concerning. Quantum mechanics is real: it very accurately describes reality. If you have a problem with quantum mechanics, you have a problem with reality, and that's about as anti-rationalist as you can get. To improve, find out what part of you rejects the reality of quantum mechanics in spite of experimental evidence, and censor that.
Once you've achieved that step, you can skip the rest IMHO.
http://lesswrong.com/r/discussion/lw/lx0/rationality_from_ai...
Vague counterpoint -- this is an ebook collecting essays from lesswrong.com, which is a bit more subtle as names go.
A name is enough of a reason on it's own?
That is, my understanding of reality is the same, regardless. I think the goals of "The Brights" are laudable, generally. But I didn't register on their website, and wouldn't really want my photo and identify featured as a Bright, because the name just doesn't feel right to me.
How you identify yourself makes a difference to how you're judged, and hence the influence you can bring to bear on the world.
Or, you know, if I someday gain a bit more public recognition and clout, it may start to matter then how I self-identified back in 2010.
Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments.
Consume this philosophy at your own risk.
And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.
I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?
[1] https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo...
As far as I understand this part of the sentence, Eliezer got angry at Roko because he did something that in the same thread he stated is unethical.
See "Belief in belief", which is right near the beginning of the first sequence....
http://lesswrong.com/lw/i4/belief_in_belief/
It should have been obvious to Roko that either it was a worthless idea (in which case he was safe to post it... but shouldn't, because it's worthless); the other case was that it was a serious, valid idea, in which case he obviously should not post it.
(If you discover something that scares the crap out of you and you need someone to talk you down/point out why you're wrong/help you sleep at night, you find one very smart, very stable friend and you ask permission and then you talk through it, you do not just publish)
I believe there is a secret government agency with the purpose of controlling the entire world will come about.
I believe that this agency will control everything, and will punish all people it determines learned about it's possibility and did not either work to make it a reality, or immediately turn themselves in to the authorities.
In light of this, it's important to keep knowledge of this future event from people so they aren't targeted.
Whoops, I just posted this to a public forum.
Regardless of whether the belief is true, if the poster believes it, then they are acting in a reprehensible manner. If you incorrectly believe you have a button that sets off a nuclear warhead in a heavily populated area and choose to press it, you still shoulder some responsibility for your intended actions.
EX: What's the maximum computational power of the universe?
I feel like I just stumbled into a Scientology auditing session or something reading your comment.
- Instructions on how to make a gigaton bomb with processes and materials anyone can acquire and assemble privately.
- A reachable-by-a-country mechanism for blowing up the sun.
- An image designed to temporarily or permanently mess with your vision. Sharing [1] on reddit as a "cool optical illusion" would be mean.
- Probably whatever a computer spits out if you give it a scan of your brain and set it on the task of finding a paragraph/image/movie/whatever that maximizes dopamine production.
- Some people would consider a devastating argument against free will to be an information hazard, predicting that people would then do whatever they wanted because they "couldn't be blamed".
- If you're a hardcore atheist: religions. If you're a hardcore theist who thinks the devil is out to trick you: convincing arguments against belief.
But... for "might be real" to be valid at all, you have to buy into the Roko's basilisk concept in the first place.
If you don't, there's nothing to get upset about.
1: http://en.wikipedia.org/wiki/Inchoate_offense#Impossibility
On the other hand, there's some pretty compelling reasoning that the entire universe is actually infinite in extent, so ... infinite computational power?
Also, there is this concept of "thought experiments". Building a universe-sized computer may sound as likely as sitting on the light wave and looking around, and yet the second one brought us Relativity.
I wonder how many people that fed that question into their mentation have considered sabotaging a coal plant. Obviously, that's ethically more complicated than the thought experiment, but it follows the outline.
(If you believe that the substitutes for coal power are available and superior, then taking it offline should still be a net good, one that is distributed over a great many people, with the downside (jail) concentrated on the saboteur.)
Note that dustspeck vs. torture example involved 3^^^3 dustspecks, which is more than number of atoms in the universe raised to the power of number of atoms in the universe. It was purposefully chosen as such an extreme number, as to clearly outweight the utilitarian calculation in favour of torture - so that the example could be used to study our moral intuitions.
Short of that, it's just a mind puzzle, "how does what I think my beliefs are match up with what I think my beliefs are" or so.
I know I'm not a 100% selfless altruist. I don't donate all my money to charity and I'm not going to risk going to prison to sabotage a power plant. But that doesn't mean I can't aspire to being more altruistic or want the world to be more utilitarian.
Almost all of EY's posts are about how your gut reaction to things is often wrong, inconsistent, or leads to outcomes you probably don't desire. E.g. Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/) where people will pay more money to save 4,500 people out of 11,000, rather than 4,500 out of 250,000.
If someone invites you over for dinner and they have a live calf there, nursing at its mother, and they're intending to slaughter it in front of you, it's not the same (to gut morality) as if they slaughtered it the day before, and really not the same as if they just bought some veal at the supermarket (i.e.: paid someone out of sight to do it).
Likewise; buying something that was produced in a way that involved significant human misery and/or death feels awful to gut morality if the sweatshop, diamond mine, or whatever is next door. On the other side of the world? Eh... well this one is $6 cheaper...
Why should it be so different? Well, gut morality isn't logical, at all. It's worth a bit of effort to do better.
At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.
When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.
Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?
Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?
The only reasonable outcome in which the dust specks would be worse would be, admittedly, if there's a non-negligible portion of the population that, as a result of getting the dust speck while operating machinery or doing something important like surgery, ends up making some terrible mistake that causes lasting harm to someone or themselves. In that case, the outcome is "hey, some huge number of people got maimed or killed, so the torture of one is OK in comparison."
But my thought is that your line of reasoning would still be held even if all 3^^^3 people were given the dust speck when they are sitting alone in their room away from any potentially-dangerous objects. Is that the case? In that particular case I don't see what the dust speck would actually do to any person in the future (beyond a single sensory experience in the present).
And to answer your question, I wouldn't pay a penny to avoid the dust speck. Maybe I'm just particularly frugal, being a grad student at the moment.
I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects.
But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny.
Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die.
Which do you choose? Sixty years from now, which do you wish you'd chosen?
> I wouldn't pay a penny to avoid the dust speck
Do you mean just your own dust speck? The question was all 3^^^3 of them.
If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)
a) would you press the 7bn specks button for an individual 'reward' of $70million?
b) would you press the N-specks button if given a choice between that and the 1-torture button?
Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.
Example (this is actually true!) -- I have a life-long inflammation in both eyes that started in one eye when I was about 5, and eventually spread to the other.
The specialists weren't able to figure out the original cause; their best guess was basically a speck of dust -- just the wrong speck of dust (maybe with a microorganism in it?), in the wrong place, that my immune system attacked and then just kept on attacking even after the original speck had been gone for years. Medication plus a series of unpleasant surgeries have managed to keep my eyes relatively functional, so far, but others with the same condition certainly do lose all vision.
Obviously this isn't a common result; but given enough specks of dust.... Well, you have to start doing some math.
I think the dust speck argument would be better if first framed as a preference for yourself:
If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.
I think most people would answer no. So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/), that our emotional reaction to things isn't diminished by even infinitesimally small quantities (see also Useless Medical Disclaimers http://lesswrong.com/lw/h4/useless_medical_disclaimers/).
I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.
There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.
And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.
> I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.
Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.
Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture.
> A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. http://edge.org/conversation/jaron_lanier-the-myth-of-ai
I fail to see how the first part of this sentence implies the second.
The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement.
Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases.
I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?
- If you prefer 100000000000000000000000000000000000000 specks to 1 torture
- and you follow the axiom that A > B implies x% A > x% B, and value people equally
- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture
- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%
- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day
- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that
I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".
Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:
* some people spontaneously torture others
* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them
* so: eyeball dust storms commencing in 3, 2, 1...
One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.
That doesn't seem to be Yudkowsky's argument.
Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.)
With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?
The most common example I see is the dust-specks vs torture thing[1], where Eliezer thinks increasing dust specks eventually outweighs the torture and various other people think torture outweighs any number of dust specks. Eliezer is basing his position on taking a few decision theory axioms as obviously true, and others are basing their position on taking their moral intuitions as obviously true (i.e. a "One's modus ponens is another's modus tollens" situation).
Anyway, my point is, I'm only offering a gentle and hopefully reasoned disagreement with someone I regard highly. I'm not jumping on the "fuck that guy" train. Moving on.
There was a post that boiled down to the question "Would you rather 3 ||| 3 people (where | is an ascii stand-in for Knuth's up-arrow notation) get a mote of dust in their eye, or one person be horrifically tortured for 50 years?" and his conclusion was basically "you can use math to assign some incredibly small epsilon of suffering to getting a mote of dust in your eye, but eventually, if you sum enough people, it's more suffering overall than one poor person getting horrifically tortured". My problem with his post was that I think that any morality scheme that results in the person getting tortured is fundamentally flawed, and I don't care how much math you throw at me to try and "prove" that it's better.
I don't think I've ever heard of an attempt to rigorously derive morality that I agree with. Morality is too contextual, too messy, for us to perfectly capture in those sorts of models. It's especially bad when we try and model morality mathematically, and then take as gospel the result of that model, rather than say "oh, um, that's not a great result, the model must be wrong".
Now I happen to agree with you that I feel this dust result is incorrect, but morality is so undefined and subjective that I can't help but disagree with your conclusion that the model must be wrong. The entire moral question of the dust vs torture doesn't have one defined answer so we can't really conclude anything more than opinions about the model
If you decrease the number to even the number of humans that ever existed/exist/will exist, then it returns to being negligible in relation to torture.
The problem with an if -> then, is when the 'if' is flawed, it doesn't really matter what the 'then' is. The human mind can't grasp 3^^^3 dust motes because it doesn't exist in any meaningful way.
>But let me ask you this. Suppose you had to choose between one person being tortured for 50 years, and a googol people being tortured for 49 years, 364 days, 23 hours, 59 minutes and 59 seconds. You would choose one person being tortured for 50 years, I do presume; otherwise I give up on you.
>And similarly, if you had to choose between a googol people tortured for 49.9999999 years, and a googol-squared people being tortured for 49.9999998 years, you would pick the former.
>A googolplex is ten to the googolth power. That's a googol/100 factors of a googol. So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort, and multiplying by a factor of a googol each time, until we choose between a googolplex people getting a dust speck in their eye, and a googolplex/googol people getting two dust specks in their eye.
>If you find your preferences are circular here, that makes rather a mockery of moral grandstanding. If you drive from San Jose to San Francisco to Oakland to San Jose, over and over again, you may have fun driving, but you aren't going anywhere. Maybe you think it a great display of virtue to choose for a googolplex people to get dust specks rather than one person being tortured. But if you would also trade a googolplex people getting one dust speck for a googolplex/googol people getting two dust specks et cetera, you sure aren't helping anyone. Circular preferences may work for feeling noble, but not for feeding the hungry or healing the sick.
To put it more clearly, he says: > So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort
And I don't agree that you can. The difference between dust-mote and torture is not one of degree, but one of _kind_. It's a discontinuous function (in my opinion). I don't know where the discontinuity is, but it's there.
There are a finite number of possible brain states, far less than the 3||3 number. All possible "feelings" is likewise less than that.
You need to be dividing all possible feelings of pain into two groups, and asserting that the absolute worst of the first group (containing dust specks) is incommensurate with the absolute best of the second group (containing torture).
At some point you need to say that you'd pick specks even over 1 minute of torture, even over 1 second, on which I think most people's intuitions would stop saying that. Or you need to find some point between 50 years and 1 second where it become commensurate.
At least, in my opinion.
But let's double each side; presumably you would make the same decision if asked again, right? So now you prefer X-1 seconds of torture done to each of two people, over twice as much specks.
Now, unless X is very low, you should prefer X for a single person over X-1 for two people. (If you disagree with this, please give a plausible value for X that makes it false.)
So you prefer X on a single person over double!specks, but prefer single!specks over X. This seems extremely unlikely. Or even if true, we should be able to make you pick torture for 50 years just by multiplying specks another couple of orders of magnitude.
Does this make his argument any clearer?
The answer to the argument from circularity is obvious - there is a discontinuity. I can see the discontinuity in my own thinking, and you probably can too. I would accept a dust mote to save an individual from years of torture (and I think almost all reasonable people would), but I would not accept a dust mote to save two people from a dust mote. That may be unethical of me, since I would of course prefer the world that has fewer people with dust motes, but for me to expect the greater number of people to make the sacrifice, I must be prepared to make the sacrifice myself and it must be a sacrifice that I think all people should make. Exactly how much I think people should sacrifice for others is difficult to say, but there is a clear step change at some point.
So saying that you are selfish doesn't get you out of this, as long as you do prefer the world with less other people having specks.
Where is there a discontinuity when talking about other people being tortured or specked?
(Also, if you click on the comment you can reply).
>At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large.
Well, the boundary is fuzzy, so I believe that different people will draw the line in different places, but yes, that is correct. There is a point at which I accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person.
I think the debatable range is actually quite large, so the tiny decrease part might not be fair (there are a lot of degrees of pain I would not demand someone to suffer to save others), but at that point of expected sacrifice, the number of people jumps to infinite.
That seems like a good way to put it.
The number of people doesn't really matter, when it the amount of hurt per person is so small that you can say with all confidence something like "I believe that literally any sane person would agree to get one speck in their eye to save a stranger from fifty years of torture".
As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.
Nearly all of whom prefer to receive the speck than to allow the individual to be tortured.
> As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.
Yes, I feel somewhat uncomfortable ignoring the agency of torturers and murderers in this scenario, and that reluctance would play into a very conservative estimate of where that threshold should be, but I would be reluctant to choose a lowest common denominator measure to establish morality.
I believe that there is an amount of suffering that it is reasonable to expect anyone to accept in order to help another person. That amount depends on the amount to be suffered, and the amount benefited by the recipient. Once the suffering falls under that threshold, I do not believe the number of people required to make the sacrifice comes into consideration, as each of them if reasonable would say "I prefer to belong to this world, where as part of a huge group I accept this small ill in order that someone else benefits". Therefore, the implied sacrifice results in greater utility for that choice.
Let me try a different tack.
Let's say that you observe a universe with some large number of people suffering dust specks in their eye. That sounds bad. But what if every single one of those people actually suffering thinks that this universe is better than the alternatives. You don't suffer from a dust spec, but are you going to ignore all those people in their estimation of the utility of the universe? If you switched to a universe where all of those people didn't suffer from dust specs, but someone else was suffering, they would tell you that that was a worse universe.
It's pretty obvious to me that even if that isn't the exact case, it's close to being the case in reality - that's why people find the dust speck argument to be unintuitive, not the large numbers thing. It's because some measure of sacrifice for other people is part of what we expect from everyone, and most people know instinctively that if everyone asked to make a sacrifice agrees that it's right to make that sacrifice, then the world is better because of it.
>Can you explain more clearly why what I said implies that, because I don't think I mean to say that.
It's basically a reformulation of the circularity argument.
I assume there's some level of pain that you would prefer to the specks; say a single second of torture, equivalent to a smack or such. (If you think we should prefer 3|||3 specks to one smack, I could go further, so let me know.)
So counting up from that one second at a time (i.e. 2 seconds of pain, 3 second, etc), eventually we reach a point where you no longer think it's better than the specks. Call this X.
So X and X-epsilon are qualitatively different; the lower amount is not bad enough to outweigh specks, but the higher amount is. You need to prefer giving X-epsilon to a large number of people rather than X to a single one, if the qualitative difference is to be upheld.
(I may not be phrasing this so well. Maybe try working through the circularity argument above, or the other phrasings I used in this thread.)
Now to respond to your line of reasoning: this proves too much.
Imagine instead of dust specks, we want everyone to donate a dollar to save the person from torture. Are you really going to say that we should be spending unbounded amounts of money (3|||3) to save anyone from torture? Have you donated all the money you could get to prevent torture? (and yes, I'm sure there are charities that are at least partially effective.)
Why doesn't your argument work for the case I just outlined as well?
Quite possibly I prefer a world with one person tortured for a lifetime compared to 3|||3 specks if they are evaluated out of context. It's hard to say, because pain doesn't easily sum, there are different qualities of pain, and we are talking about situations where we may be losing an entire persons contribution to humanity. I just don't think any of this is relevant to the conversation, or to the reason that so many people find your conclusion unpalatable - and it's nothing to do with not understanding large numbers.
In the case that they are evaluated in the context of a choice between one of those worlds or the other world, I would take into account what I believe to be the value that the individuals involved would place on the worlds were they to know the details and have minimal moral standards like mine.
Let me phrase it another way:
What would you say the utility of a world where there are 3|||3 people with specks who chose it gladly and voluntarily to save someone from torture is?
Let's say you tell those 3|||3 who wanted to save someone from torture by accepting a speck in their eye that they cannot, and someone must be tortured instead. You've massively increased the unhappiness in the world - not only is an individual getting tortured, but 3|||3 have ended up with a situation that's worse than they wanted. Are you going to claim that it's still got a higher utility? Now that you notice that you're making those 3|||3 unhappy by the choice, you can see that the disutility scales with the number of people - that's why the number of people becomes irrelevant.
> Imagine instead of dust specks, we want everyone to donate a dollar to save the person from torture. Are you really going to say that we should be spending unbounded amounts of money (3|||3) to save anyone from torture?
Your phrasing is unnecessarily emotive here. You seem to be saying that 3|||3 dollars is an awful lot of dollars without giving me any context about what fraction of the dollars belonging to those 3|||3 people those 3|||3 dollars are or what else they could/should be spending it on. If it's a negligible fraction that scales, and I could plausibly think that any sane person should donate that fraction of their money to save a person from torture then yes. You'll notice that when it's phrased like that it does not require that I donate all the money I could get to prevent torture. I in fact do donate a small amount of money regularly to prevent torture, but I would demand much less of the 3|||3.
If the people are told of the choice, that's a whole new problem, but that's kind of avoiding the point of the original question. To use a hacking analogy, you're using a side-channel to cheat.
Nobody is told about any of this. If they were, that would itself need to be factored in, and quite possibly lead me to prefer specks.
>If it's a negligible fraction that scales, and I could plausibly think that any sane person should donate that fraction of their money to save a person from torture then yes.
Each person isn't donating to save someone from torture, they're donating to save 1/3|||3 of torture.
Let's rephrase the original question to zero in on that last point. There are 3|||3 people. You choose the number of people who donate one dollar, which can be any number between 0 and 3|||3. After you make a decision, one of those people is chosen at random, and they are tortured iff they did not donate.
If you think about it, this results in the exact same outcomes in either choice, except you have more options than all or nothing. You basically choose the probability of torture.
To be consistent with your previous view, you'd need to not pick zero donaters. So for at least some people, it should be worth it for them to pay 1 dollar to avoid a 1/3|||3 probability of torture.
This dissolves, as you may have noticed, into Pascal's Wager. (Or Pascal's Mugging, more precisely, which was coined by Yudkowsky.) What if I tell you that unless you give me a dollar, I'm going to torture you for 50 years? The probability of that being true is more than 1/3|||3 (and if you disagree, then you are way too overconfident for life. 3|||3 is a huge number, and there's no conclusions I can think of in which I'd place that much confidence. We don't even have anywhere near the kind of raw data to draw any conclusion that confident.) So are you willing to give up a dollar to avoid the >1/3|||3 chance that I'm telling the truth?
If not, and all or most of those people in the problem would answer the same, then your previous rationale falls apart.
(Oh, and this is not the real Pascal's Mugging; that's much harder to deal with. But let's stick with the easy stuff for now, shall we?)
Also, what about the argument in http://lesswrong.com/lw/kn/torture_vs_dust_specks/ueo
(And if we're going to continue this, could you please move it over to another forum? It's getting harder to keep track of comments here.)
Exactly. My whole contention is that the reason this question is considered unintuitive by so many people is that they're really considering a different question to the one you think you're asking.
Hello, apparently I'm unreasonable. And so is everyone else who has said that they choose torture over dust specks. If you select 3^^^3 people, you're going to find an awful lot of us. (And also some sociopaths who literally don't care if someone else gets tortured.)
Your argument seems to boil down to "specks is the correct answer, so anyone who gets it wrong doesn't count; and because we all agree that specks is the correct answer, it's okay to do specks".
On the other hand, I would totally accept the torture for myself, if it would prevent the specks. (At least I hope I would, and to the extent that I can model how I would act in that situation, it does seem plausible that I would.)
But anyway. This isn't even philosophy - it's a digital remix of medieval scholasticism pretending to be philosophy.
The irony is that politics proves empirically that ideas actually can be dangerous and harmful. And some ideas - actually narratives - can be very dangerous and harmful indeed.
There's over a century of "persuasion technology" (Bernays, etc) that exploits this.
Nothing I've seen Y write deals with the problem of politics as a social exploit in an insightful - never mind a useful - way.
Meanwhile real people are being tortured in real ways. What's his proposed rational solution to that problem?
(You may assume you don't know anything in particular about these people, other than that they are adult humans.)
The fact that it includes some numbers that reduce to some other numbers doesn't change that.
Putting numbers into something doesn't make it scientific or objective. It just makes it numerical.
There is a difference, and it's not a small one.
If I capture person X and X's laptop Y, and X tells me under no duress that Y contains the location of a nuclear bomb that X has placed in a major city, and I have other strong evidence that this is true, but X refuses to give me the password to Y; then it is moral (but rightly illegal) for me to torture X for the password to Y.
Torture is not literally incommensurate with any other bad thing. Then the question arises, how do we, in full generality, determine which is the greater of two evils (or the better of two goods)? The torture vs. dust specks thing is supposed to disabuse people of the unhelpful notion that some things are somehow incomparable in terms of goodness and badness.
If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.
I think most people would answer no. So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/), that our emotional reaction to things isn't diminished by even infinitesimally small quantities (see also Useless Medical Disclaimers http://lesswrong.com/lw/h4/useless_medical_disclaimers/).
I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.
There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.
And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.
In Yudkowsky's argument, it's the option between one person _absolutely guaranteed_ to have 50 years of torture, vs 3^^^3 people _absolutely guaranteed_ to be dust-moted. I think the guarantee changes things significantly.
Just think of it as if you were going to other people and deciding to take this bet for them. 1/3^^^3 chance of torture in exchange for removing a speck of dust from their eye. I think you would be ok with that because it's an obvious choice for ourselves.
And if you continue to do this for enough people, eventually one of them will get tortured. After removing dust specks from approximately 3^^^3 people's eye. But you are right, there is a chance no one will get tortured. So do it for 3^^^^3 people then and the probability is 0.99999...
Something as innocuous as "Think safe thoughts!" enjoys a similar property.
If I modded this, I would delete your post.
If I don't take info hazards as a serious threat, there isn't anything to sort those statements with.
Toss out silly ideas like the basilisk thing, and think up real info hazard possibilities.
E.g., let's say you knew what Snowden was up to, a few weeks before he figured out how to publish what he'd found and leave the US.
You'd have done a lot of harm to post that info publicly.
Suppose you find a laptop left in a taxi, with police records including personal details of rape victims in your city for the past 20 years. Post it to 4chan? Uh, no.
What about ongoing undercover investigations? Post it?
You accidentally shoulder-surf sensitive credentials -- bank manager's login, nuclear plant access code, whatever. Post them online?
Simple ways to make highly dangerous weapons, drugs, poisons, etc. can also be info hazards.
It's honestly not that hard to imagine real-life examples, without having the dangerous info in question or descending into government assassin-type scenarios.
It's just any data that, if you make it generally available, will predictably result in bad things happening.
In some cases -- like revealing important security flaws -- a public disclosure may be a last resort if less harmful methods have failed, but it's likely not an ethical first resort in those cases.