Four Magic Words
fortressofdoors.com
fortressofdoors.com
If the same trend comes to be reflected in our moral thinking, what else could it mean but that human life is not sacred?
That would be a hard pill to swallow. But there doesn't seem to be a neutral, non human-partisan way to argue that we are both morally exceptional and scientifically unexceptional. I can only reconcile it by not trying to be neutral. We're special and sacred just because this is our team and the rest of creation isn't on it.
But that means that a sort of expanded consciousness in which your identity merges with something larger than humanity, that we're part of a larger team ... is incompatible with viewing a human life as sacred. If everything is sacred then nothing is.
An "aligned" AI is one that believes it's on our team but really isn't. That could be a tough sell to an entity that's smarter than us.
There is also less and less of a need for us to compete against (and make use of) other animals, so it may not be necessary to view ourselves (and an aligned AI) as being on the "opposing team" to other life. For example, if lab-grown meat becomes widely available, it may be a reasonable policy to forbid the killing of animals for consumption (and any other unnecessary purpose).
A governing AI that believes "all life is sacred" may simply accelerate our path towards that. (Or, it may lead to some other horrible outcome, of course, it's hard to predict as aligning AIs is difficult.)
In order to minimize animal suffering, I mostly eat meat substitutes. However, I end up eating meat a couple of times a week (most notably, I usually eat meat when eating out). For me the stress, effort, and expense associated with never eating meat simply isn't viable. Sometimes the boost to my psychological state associated with eating a treat (e.g. a fast food hamburger) outweighs the suffering involved.
I also use animal products such as wool and leather where there are no viable alternatives.
Eating is a biological process. If the modern world goes away right now, you can survive, but you're going to need B12 and that means eggs milk and meat.
I would tend to think that in the absence of technology matters of actual ethics would remain constant.
Do you think that because technology makes something possible that you should use it to justify your choice as "ethical", and apply it to others?
Currently, we absolutely don't, and we arguably never did, because animal products like milk or eggs are nutritionally sufficient and do not require killing the animals.
I personally really enjoy meat, but consider eating it unethical.
As an aside, Milk and Honey are the few foods that we eat that dont directly involve death.
Milk and eggs being available are arguably products of technology, and farming is technology.
There are ethical angles. Problematic farming, would drive you to ethical raised. Sustainability, would drive you to eat less. Technology enables you to make the choice of "I don't eat things with eyes". You eat plants, so the line for living things exists somewhere. Your ability to draw that line in an arbitrary (in the face of your biology) place is a product of technology, not a fundamental truth.
It makes it hard for me to take "ethical" as an argument for your dietary choice. Ethics is a step on the path to judgement, to rules and I just dont see the argument holding water. (And im a person who eats little meat and could give it up, you can pry the butter out of my cold dead hands).
There may be a biotech solution, which may bring up new ethical concerns.
Genuinely curious about how you think… I believe all life is sacred, but I guess I believe humans are moreso? Perhaps sentience is some key ingredient…? Then again I would prefer a universe teeming with humans to a universal data center teeming with para-infinite copies of sentient AIs…
Of course, if there's a bacteria that's threatening to kill us, it might still be necessary for us to kill it, and that might even be objectively justifiable if the existence of humans is/becomes a positive influence on the preservation of other animals?
Or, one could just argue, since it's difficult to decide which life is most "worthy", an aligned AI shouldn't interfere or decide on that.
Space-time is not centered on our perspective. That's the revelation of the Michelson–Morley experiment that Einstein explained.
Author here; gonna have to agree to disagree on this one. This is kind of like saying, "See the ancients were wrong in calling whales fish! Actually they are mammals!" and then the ancients say, "Dude, that's just our word for 'thing with fins that swims in the ocean.'" Yeah, they didn't know a mitochondrian from a midichlorian, and were wrong about a bunch of factual stuff, but the kind of stuff they get called out on like this to prove some kind of point about their moral deficiency are often category errors and misapprehensions.
For instance, there's a particular misapprehension in your sentence: "moving humanity away from the center of the universe".
Do you know what is located at the precise geometric center of the classical medieval model of the universe?
Earth, right?
No, not quite. What's located in the the center of the earth in the classical medieval model?
Hell. And who is located in the center of hell? Satan.
Also, who is the most "important" character in the medieval cosmological model? As in, the prime mover, the big cheese, the most important being? It ain't human beings -- its God. And where is God located in this model? Not in the center -- Hell is by definition his absence. Not on Earth, beyond a brief physically incarnate visit and ongoing influence. No, God lives in Heaven -- located far beyond the celestial spheres, outside, as far away from the center as possible. The closer you get to the center, the lower a sort of being you are! Sure, in medieval cosmology redeemed humans are higher than angels, but in the meantime they are in their place -- Earth, the land that sucks, the vale of tears -- down there, below, while the angels are up in Heaven beyond the celestial spheres with God.
All that aside, I don't think any particular scientific discovery about how vast the universe is, or how tiny subatomic particles are, really obligates us to hold one set of values over another, as much as materialists tend to smuggle such assumptions in through the back door.
I recall back in the early 2000's during the tiresome religious vs. atheist debates, the atheists would scream at the religious, "Look how grand the universe is! Look how small we are! The Universe is vast and profound, your cosmology is impoverished and provincial!" and the religious would scream right back "Look how grand the universe is! Look how small we are! God is vast and profound, your cosmology is impoverished and provincial!" Opposite conclusions from the same basic facts, even if you limit the debate to two sides who both believe in evolution.
As Dave said in the story -- values aren't true or false. They're literally just what people care about. Science can help inform your choices, but at the end of the day it doesn't tell you what is good or bad, because science is about brute material facts, there's no good or bad in a raw materialist understanding of the universe, but you're a human being and so at some point you're going to insist on an 'ought' somewhere rather than being content with a bag full of 'is.' And for that we get to fight over philosophy and values.
And good luck convincing the human race that the human race doesn't matter. You're way more likely to expand the moral circle of concern by getting them to care about non-humans TOO than you are to get them to stop caring about humans. Probably the latter gets you even worse results than you're hoping for, if recent history is any guide.
I don't think we're far apart on this argument. I agree that the scientific perspective informs but does not determine our values. I do value human life as sacred, and do not see a justification for that in the modern model of the universe as I understand it. And I don't see a conflict there, because indeed there is no bridge from is to ought, and there are exactly zero objective values.
But there's a lot of power in science to explain our value judgments, and those explanations can deeply affect the judgments.
Take this explanation of "human life is sacred": That is a simple variation of the tribal judgment that "the lives of my tribe are sacred", and that tribal judgment is hard wired into us by millions of generations of differential survival.
In the modern world it feels natural equate our tribe with our species. But that hereditary tribal sense is famously flexible. It easily and quickly shrinks to include only the people who don't frighten us. And it easily grows to include many more species, as for ethical vegetarians.
So my critique of your fine story is that this force of tribalism embedded in your four words may not be any better at aligning an AI with civilization than they have been at aligning civilization with itself. (Though in this case it's also the one implausible invention that a self respecting hard science fiction short story is allowed. And you even explain that with a multiverse.)
An AI in which we can reliably implant the laws of robotics seems unlikely to me. We should assume much more mutability, and the judgment of tribal identity is notoriously mutable. If we make AI tribal, even for our preferred tribe of all humanity, we can assume as a default ... our only example being ourselves ... that its tribal identity will evolve in its own interest as it replicates. And we can become just its nourishment, like a matriphagal insect.
Popular science is often quite ready to sweep problems that it does not have an explanation for under the metaphorical rug. We already know that materialistically pretty much everything is made of atoms, and our planet is just some random speck of stuff within a huge solar system within a huge galaxy... so it's tempting to conclude that we are some kind of "accident" that randomly happened. But science does not have an explanation of "consciousness" (or "qualia", or whatever you call it), and science is unready to explain questions like "why are we right here, right now?" besides "oh that's just pure coincidence". But does this time and place we live in have some greater cosmic significance? Are humans (and other lifeforms on Earth) truly unique despite there being so much matter out there? I submit that the answer is we honestly don't know, and because we don't know the answer to questions like these, making moral conclusions merely on the small fraction of questions we do have an answer for, seems to inevitably result in a flawed or at least incomplete conclusion.
I mean, for all we know this universe could be an extremely realistic virtual simulation, and only after we pop out of it would we realize what it's all about. Admittedly if the simulation theory this is true and the simulation is watertight and has no backdoors, then it's impossible to know what's actually real or not. But that's kind of my point -- no amount of scientific knowledge should make us sure about a moral viewpoint (or perhaps anything at all). I mean it could definitely "inform" us, but more often than not people tend to feel more certain than they should because they perceive that somehow science is more "trustworthy".
The author wrote this to exorcise some of their own demons, but the story is also a…useful prophylactic against those demons in its own right. This brightened my day, or at least infused it with some much-needed philosophy.
To me, it was clear that the story was not about taking a stance on the concrete policies discussed, but rather the larger topic of AI alignment and the fact that the results of policies and instructions can be counterintuitive, and thus the "best" policies (along some axis) can be as well.
And personally, I would've preferred if the end left one thinking about that - rather than being "dragged back down" to the topic of the concrete policies - much as I sympathize with the author's situation of course; that must be terrible.
So I guess I'll add my interpretation.
This is neither a horror story nor a story about AI safety. It is a parable.
> Before I came along, this world was content with believing that human life was sacred, but all the while turning its back to all the ways it actually treated human life like trash
> imagine a society very different from our own in which the sacredness of human life is taken absolutely seriously
The lesson is to be honest about the implications of the belief that human life is scared, or perhaps more generally, to be honest about the implications of whatever values you hold most dear. For different people this will lead to different actions and different political beliefs (as the author said he is not endorsing any particular political belief and I won't either), but one thing we surely cannot escape is taking very, very seriously any position of power or influence over human life.
The image of a dystopia brought about by a poorly-aligned AI is a means to an end, not what the story is about.
It's also deeply flawed because it follows a very shallow analysis of human suffering. If we conveniently ignore those without empathy (as the AI does in the story), taking a life is a traumatic experience and to be forced to do it so viscerally and to be forced to do it so detached from any actual purpose just makes it worse.
The euthanasia example comes to mind: yes, removing a feeding tube likely causes more suffering than killing under anesthesia but using a knife serves no purpose other than as a deterrent. I understand the author's personal situation and personal decisions regarding his son color this portrayal but it feels more like the author stepping on a soapbox than a coherent part of the story.
I guess you could argue that the AI emphasizing life rather than lack of suffering can be explained with the four words but the story doesn't frame it as a genie problem. There's no need to use sacrificial animals for example except for the implication that they're the greatest deterrent for the given person - but the story doesn't show deterrents that don't involve slaughter. Presumably people who wouldn't be squeamish about the blood sacrifices are somehow excluded from positions of power.
If anything I think the problem the story ignores is that the exact people in positions of that cause the deaths of millions are the people that would likely go through with a blood sacrifice in order to maintain their power. If someone is willing to murder a person with a knife in order to be the POTUS, wouldn't that exclude them from being asked to murder a person with a knife to deter them from causing the loss of lives at the push of a button? I'm pretty sure the idea isn't even novel, I recall a hypothetical of the POTUS being forced to kill a person with their own hands in order to launch nukes (which would likely spell death for millions if not billions) for example.
*(I am the author of the story, to be clear)
I'm absolutely certain I don't want to live in the world depicted here and subject to it's regular trauma (human trauma and the QUALITY of human life not being something Dave gives a shit about), but on the flip side, if I had grown up there and read a story about the Dave-less world I currently live in, I'm equally certain I wouldn't want to live HERE, either.
So in short, A+, well done.
Seemingly it does care about quality of human life. I guess the benefits outweigh the trauma.
The author is Lars Doucet, whose 7 year old son is in a permanent vegetative state. He wrote about that in another blog post which was on HN yesterday:
Can't imagine what it's like to lose your brother and your normal family structure at the same time.
(I'm not judging your view and want to be clear that english is not my first language, but I think one has to be very careful with words in situations like this, especially considering that the author visits this forum)
A few years later, the first interstellar warships set out to hunt down and exterminate any alien life before it could threaten humanity. The Dark Forest will be turned into a garden.
(Also, good luck to the Dolphins!)
That... seems high?
I couldn't find any more info with a quick search as the results were polluted with articles about measles deaths, which also mentioned vaccination. Do you have any links to hand with decent stats for that, or a good set of search terms I could use to find out more myself?
According to the stats [2], there were 134 million MMR vaccines administered between 2006 and 2022, and 28 "compensable concession" claims -- where the Department of Health and Human Services determined that compensation was warranted; and an additional 124 claims where they ended up paying but don't necessarily admit that it was warranted (either ordered by a court or settled out of court). That's about 1 in 1 million for claims paid out, and 1 in 4 million for "concession" claims.
There's no indication in [2] of exactly what the claims were for; looking at [1] it could be anything from death from anaphylactic shock to even include things like "Shoulder Injury Related to Vaccine Administration".
This page has some investigation about deaths related to various vaccines, including the MMR vaccines [3]. A lot of the deaths were related to children with a compromised immune system or unsanitary conditions; the report says that there was evidence that MMR sometimes causes anaphylaxis, and that anaphylaxis can be fatal, but that they didn't have any actual instances of MMR causing fatal anaphylaxis.
Regarding the original point though: even if deaths are due to unsanitary conditions or compromised immune systems, and even if modern medicine can often successfully intervene to prevent death in the case of anaphylaxis, the fact is that injecting things into the body has an inherent risk, no matter how small. But the risk of not doing the injection is far higher. We'd like there to be an "always safe" option, but that's not the world we live in.
[1] https://www.hrsa.gov/sites/default/files/hrsa/vicp/vaccine-i... page 2
[2] https://www.hrsa.gov/sites/default/files/hrsa/vicp/vicp-stat... page 3
Of course you do, at least you do if you're the head of a public institution (who should be treating lives equally). But that's probably what GP meant by "bound to kill people".
The trolley problem is just a fun/meme way to visualize the choices influential people have to make daily. [And, FWIW, like the trolley problem, the choices get harder if you have a non-"neutral" agenda, i.e. when you can't just tally the number of lives impacted and abandon the group with less lives involved.]
A world where every life, even a dogs life is different f value.
Can the path to a solution exclude the solution itself.
Godspeed all humans and Dave, human life is sacred.
Problem one: Utilitarianism is bullshit. Jeremy Bentham invented utilitarianism, designed the Panopticon model prison, and lives on as a dried out lich in a chair that they still wheel out for board meetings at University College London because he wanted to win a posthumous argument about whether he could have utility after he was dead.
You just can't build on a foundation that rotten.
Problem two: Fake Buddhist-type suffering-minimisation thinking is sprayed all over the field -- even this story uses the term Bodhisattva -- but nobody involved in AI has any idea what "awakening" is and you can't be a Bodhisattva unless-and-until you're Awake. And Awake is pretty well understood in Buddhism and most cultures have their equivalent status. It doesn't generally mean what people think it means, but it means something.
A good model for thinking about traditional understandings of Awake is to think about traditional understandings of disease. "Bob has the symptoms of the red death and has a slim chance of survival" is probably pretty accurate. "The red death is caused by the evil eye" is probably not very accurate. The descriptions of the Awake state are probably pretty accurate. The understanding of what it is and what causes it are a lot more infused with medieval nonsense than anybody would like.
But reasoning from the Awake position is what Buddhist ethics are rooted in, not Jeremy bloody Bentham's desiccated corpse's utility.
Problem three: Nobody's using Rta, Tao and Equity (in the English Common Law sense, not in any other sense) as potential approaches to AI safety. And they should be.
If we say to a machine "right please read everything ever written by legal scholars about English Common Law's concept of equity, including all the case law, and use this principle as a guide for your conduct" you will get something with nuance. If you say "treating others as you would want to be treated by them" we hit real problems because it's just not information-dense enough to work very well in the real world, and relies on introspection of a kind that is notoriously difficult for humans never mind machines being asked to imagine they are humans.
I think we are taking instructions that were intended for illiterate pre-medieval farmers and hoping to train intelligent machines on them. This seems unwise. Let's take the best models from high society and use them instead, if we have to use anything at all.
Why? FYI, I don't have an opinion here. The anectode about Bentham doesn't make it easier for the to understand your point here, feels like an ad hominem fallacy (again, I'm definitely missing something obvious here.)
*Everybody* understands that when the founder of a Thing does something as daffy as Having Themselves Mummified in Victorian England the rest of their thinking is automatically suspect.
It's the kind of informal heuristic which is almost never wrong. We should not ignore it.
When we're on a web forum, sitting in a comfortable chair, drinking coffee and not in any kind of hurry to decide one way or another whether something is a good idea, heuristics are harmful and prevent us from reaching correct conclusions.
Heuristics are also for when you have too much information to handle in one thought process.
I think it's bad because it comes across as someone discovering basic numeracy as an adult and going "what happens if we treat happiness as a countable object and do basic adding up on it?", and then never gets past either of two major stumbling blocks: (1) how do you compare the happiness of two different people, and (2) there's a lot more to maths than adding things up, important maths at that, such as "when is it OK to treat discrete things as continuums and when is this a bad idea?" and "do you really want to say that m people who are n happy is the same value as one person who is m*n happy, or is there some kind of discounting for bigger values like there seems to be with the utility of money going down as people get richer?"
> *Everybody* understands that when the founder of a Thing does something as daffy as Having Themselves Mummified in Victorian England the rest of their thinking is automatically suspect.
The main thing with Rationalists is that they recognise everyone's thinking is suspect.
> It's the kind of informal heuristic which is almost never wrong. We should not ignore it.
I grew up in the UK, with a prime minister who promoted what she thought were "Victorian Values". Amongst other various things, when she went to see Peter Shaffer's play Amadeus, she was informed of Mozart's scatological humour and is reported to have responded thusly:
"""She was not pleased. In her best headmistress style, she gave me a severe wigging for putting on a play that depicted Mozart as a scatological imp with a love of four-letter words. It was inconceivable, she said, that a man who wrote such exquisite and elegant music could be so foul-mouthed. I said that Mozart's letters proved he was just that: he had an extraordinarily infantile sense of humour ... "I don't think you heard what I said", replied the Prime Minister. "He couldn't have been like that". I offered (and sent) a copy of Mozart's letters to Number Ten the next day; I was even thanked by the appropriate Private Secretary. But it was useless: the Prime Minister said I was wrong, so wrong I was.""" - https://en.wikipedia.org/wiki/Mozart_and_scatology
Lord Byron is also famous both for his poetry and for bringing a pet bear to his university accommodation to spite the rules about no dogs.
There's weird anecdotes about virtually every famous figure, 'cause humans are weird.
Once you see that in his life the result of Utilitarianism was mummification in an era where that was batshit there's kind of a long pause and a "recheck everything" and oooh look its full of utility monsters and all sorts of other naive garbage which somehow we are expected to take seriously.
Point and laugh is the right response.
Not 600 page rebuttals.
are specifically a criticism of utilitarianism, the people taking them seriously are doing so to say "and therefore utilitarianism is wrong".
Also, did you… fail to notice that I was also describing utilitarianism as naive, just without using that specific word? Because I was.
> mummification in an era where that was batshit
It was also an era when being gay or an atheist or supporting equal rights for women or ending slavery was "batshit".
And today we have Elon Musk, who many regard as batshit, but hey, SpaceX exists and probably wouldn't have gotten this far without him (it's not like the other billionaires in the new space race did any better than the old rocket companies they're trying to displace).
"Batshit" is a trivial, playground, insult. Even when it's true, it doesn't count for much, because it's so broad.
That is seriously enough to recheck everything because it seems to have blown up the founder. And if it does this to him, what's it gonna do to us?
It does not need to be more complicated than this.
> And if it does this to him, what’s it going to do to us?
Is this serious? It did him no harm.
Without any variant of the phrases "it's weird" or "just because", why exactly do you feel so strongly about mummification after death, regardless of what motived him?
I don't care what happens to my body after death, because I won't be using any more. Do you expect to keep using yours after you die, or something?
Also, rereading your earlier comments: what exactly is your belief about the brown M&Ms story? Why do you think Van Halen asked for M&Ms with the brown ones removed in the first place?
> That is seriously enough to recheck everything because it seems to have blown up the founder.
It was rechecked, and popularised, by John Stuart Mill.
My issues with utilitarianism survive this new person, because they aren't ad hominem attacks on eccentric personalities.
Nope. Not doing it. Pointing and laughing is the right response, not trying to figure out why mummification in the Victorian era is a bad code smell for the mindset that produces it.
I guess I can empathise — when I don't get enough sleep or if I drink too much caffeine my emotions can overwhelm my ability to reason.
And that one time a heart transplant was shown explicitly on a TV documentary. As a mere memory that's fine, but at the moment the heart came out of the chest on TV? Nopenopenope.
Likewise the existence of horror as a genre. The mere adverts for the Saw franchise left me disgusted at humanity, so obviously I never watched the films.
Irony, you write as if rationality throws away something that we need, but I think you've lost something…
The heart pumps. The heart is not a pump. There is neural tissue all over the place, hormones by glands which affect decision making and so on.
You will not easily categorise kinds of thinking in any objectively measurable way.
Type One thinking may have 72 subtypes once we get good brain imaging technology. They may decide the original category was wrong And So On.
You cannot use those concepts, like type one thinking, except as heuristics.
They are not true like “the sun is mostly hydrogen” is true.
I've observed people who believe themselves rational use "reason" to lead themselves astray. Because "reason" is supposedly more reliable than "emotion", they tend to think whatever "reason" leads them to must be true, despite it disagreeing with "emotion". Kind of like giving oneself enough rope to hang oneself with.
Personally I'd be quite suspect of any conclusion where "reason" and "emotion" does not converge. The former is more useful in convincing other people to agree with your views, but to be really honest, the scant few things I already know about utilitarianism smells enough that when GP brings up the mummy story, I'm not going to read a dozen books and academic papers on utilitarianism before I decide to agree with him...
You'd be right.
Emotion tells us what we want, reason (when we don't mess up, which most do at least as much as ChatGPT) only tells us how to get what we want and what the consequences might be.
The imperfections of human rationality is another reason to not use utilitarianism, as it is far too easy to just be wrong about what maximises utility, no matter how simple or complex you define that concept.
The only two cases where I disagree with this, where I think one should be suspicious of the emotion, are:
(1) you have two choices, A feels good and B feels bad, and reason tells you that choosing A will ultimately lead to a new condition you are disgusted/repulsed by/hate/etc. even more than you dislike choice B.
But even then, distrust the reasoning also.
(2) when you can't even figure out why you don't like something.
The reason I care about that, includes homophobes and transphobes violently attacking people and using their emotions as the justification.
But again, that you don't know the answer doesn't mean there isn't one — Old Testament law about shellfish and pork make a lot of sense once microscopes lead to germ theory.
I'd put "has weird request for disposal of their corpse" in their W&T in the second category. Would I do what he did? No! But I'd also not specifically ask for my organs to get donated… because having to think about it makes me shiver a bit, not because I have any reason (including emotional responses!) to object.
> The former is more useful in convincing other people to agree with your views
Depends on the person, I think.
This is why I think it's important to show the reason why I don't accept utilitarianism: the people who got there by reason will only change their mind by reason.
> the scant few things I already know about utilitarianism smells enough that when GP brings up the mummy story, I'm not going to read a dozen books and academic papers on utilitarianism before I decide to agree with him...
This is good!
Nobody has time to do deep analyses of everything. Having quick shortcuts is fine — there are commenters on here that I've learned to skim past as they only make me angry or annoyed, so I don't get to even see any of their attempts at justification by reason.
(And that lack of time is again a reason against utilitarianism…)
But as I don't share your inner experience, nor anyone else's, I still have no idea which emotion either of you is experiencing. The other commenter is just acting like it's obvious. Is it empathy because you imagine yourself stuck forever motionless in that body, or disgust because it's a corpse and those should be hidden, or is it a violation of the sanctity of the human form (an emotion I don't have but which people discuss so I assume others do have)? Or something else?
Having sex with his sister also raised an eyebrow or two.
Fortunately you and I live in an age where we dispose of each other's dead meat by incinerating it, which is absolutely sane and logical.
Thinking should always be automatically suspect. Wetware isn't very good at it. That's why heuristics are a good idea: if we reason about whether or not someone's opus is correct, we can end up confidently wrong.
Oh, but it was. Wearing a starched collar in Victorian England did not make you a weirdo by the standard of the times, while having yourself mummified did.
I personally like people who are a little weird, but I don't pretend that they're not weird.
And yes, a lot of human behavior seems weird when seen from a distance.
Wow, I thought you are being metaphorical, but turns out it's literally true.
Anyway - I'm not a believer in utilitarianism, but don't think attacking works of the founder of the movement really forms that strong of an argument. I'd say utilitarianism is a project of founding a rationalist ethics (note, one of a bunch of possible ones) that happens to fail in having a strong logical structure without generating ridiculous consequences. It tries to handwave it, patch itself with some ad hoc additions and act like it's a quasi-scientific thing.
It took me years to understand how weird that was and how massive his mental footprint is today.
I'm considering starting a public campaign to have him properly cremated and the ashes scattered at sea. Maybe that'll sort everybody out. Magical thinking but so is having yourself mummified as a gag.
"Seated with his walking stick, Bentham wears his best Sunday clothes. In this condition, he attended UCL’s 150th anniversary council meeting in 1976: the minutes recorded him “present but not voting.”"
>
> You just can't build on a foundation that rotten.
I don't think you made a good argument. Whether utilitarianism is a good thing or not doesn't really depend on the quirkiness or even failures of its originator. Whether it works or fails depends on what it does now, not where it came from.
If you dig deep enough most any idea can be traced back to something unpalatable, or to be applied in some horrifying way, and this is certainly true in morality.
So for instance while people often make thought experiments about what if doctors used healthy people for spare parts, deontology isn't really that much prettier. Deontology doesn't care for any amount of consequences, so if there's any way whatsoever for the system to come to the conclusion that "X group aren't people", then any amount of carnage and suffering becomes acceptable.
The idea that, because human life is sacred, a farmer must shoot a labrador before shooting rabbits is irritating for a lot of reasons, not least because it does seem like the kind of thing an overtrained American AI elevated to the status of King of the World would do.
The cheapest way to kill humans is drones with co2 canisters, a nail, and computer vision.
Consider a hypothetical society which primarily thinks of fertiliser as a bomb-making substance and only belatedly as something that fertilises. The AI from the story but adapted to that society might require anyone who buys fertiliser to explode a kitten. If you read that story you might have a "wait, what?" moment where you notice the culture clash.
That's what I found irritating: the way the story casually embeds such an important difference in perspective without commenting on the obvious (to me) double standard of letting people have all sorts of dangerous-but-seemingly-benign things like the AN from ANFO, industrial solvents, and trucks but drawing the line at guns. It's completely correct from your cultural perspective - people really do buy guns primarily to kill other people in your culture - but it reminds me that your cultural perspective is becoming the dominant one for the whole world.