We need a Butlerian Jihad against AI

Replacement of AI with Mentats (as well as other narrow specialities) has done nothing but disfigure the soul. We see few Mentats -- aside from Paul and eventually another -- who are not constricted. Similarly, if you practice medicine, well, you get the Imperial Conditioning. Certainly, a sign of trust ... but also a sign that the person's actions are no longer completely free.

Now, I am not touting the Heinlein "A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship ..." line, exactly, but the alternative to AI is the kind of stagnation we see in Dune, millennia of locked down ritual, honed again and again, with some people becoming ... utilities.

Before we begin this jihad, we must examine the alternative futures.

It's interesting to ponder how we should go about building and interacting with "strong AI", and questioning whether we should even build it in the first place. But I really don't think any detailed moral frameworks can be built when we have no real idea of what a "strong AI" would look like.

Also, it's worth reminding people that in the Dune universe the Butlerian Jihad led to millennia of stagnation and control of society by a narrow elite: The Spacing Guild, the Bene Gesserit, and the Landsraad.

You can skip directly to it here:

https://www.gutenberg.org/files/1906/1906-h/1906-h.htm#chap2...

But the context in the story is fairly important - it takes place in a society that has essentially already carried out its own Butlerian Jihad and taken it too far. Wonderful book by the way.

I'm starting to think that the religious sects in the U.S. that laboriously evaluate a technology before incorporating it into their communities have a pretty good thing going. Sadly it's not really practical at a larger scale, and the suffering that could be avoided by adopting something early rather than late is difficult to estimate. Ah well!

I know things that I would like to see. Like humans "in the loop" (as opposed to "on the loop" or "out of the loop") for certain classes of decision making - for example target selection of military strikes or law enforcement. Or what kinds of information we use to train the decision making models, for example if you feed ML a racist data set and you get a racist algorithm - use that algorithm to decide who to give mortgages and you'll get systematic depression in generational wealth based on racial lines.

But this isn't some crusade on AI because it's AI; it has to be based in reality - what AI or ML is being used for, what information it operates on, what decisions it is used to make, and ultimately the human beings that are responsible for those decisions. The reason it is so hard to convince people as to how we should legislate (or otherwise regulate AI) is that every conversation drifts into science fiction and not concrete examples of the ethical issues today and what can be done today. Otherwise it comes off as Luddite fearmongering.

So when people say we shouldn't develop AI to think like that, it's basically saying we shouldn't try to understand how consciousness works. Because as soon as we do, I guarantee someone out there will attempt to make conscious AI.

Those problems largely fall into three categories that I can think of off the top of my head at 1am:

1. AI is a convenient way to justify potentially uncomfortable decisions you would have made otherwise (idlewords said it best: "AI is money laundering for bias")

2. AI is being used in situations where it can be a threat to life and limb, like the current crop of self-driving(ish) cars

3. Essentially all of the gains from automating work going to people who already have capital

> Far more important than the process: strong AI is immoral in and of itself. For example, if you have strong AI, what are you going to do with it besides effectively have robotic slaves? And even if, by some miracle, you create strong AI in a mostly ethical way, and you also deploy it in a mostly ethical way, strong AI is immoral just in its existence. I mean that it is an abomination. It’s not an evolved being.

My fear is that most software, even when useful, locks us into certain paths. Our situations or needs change, evolve, but we will remain subject to inflexible software, to systems we cannot make change with us, in the vast majority of cases. Only a very few programs strive for better: spreadsheets being one noted example.

Ursala Franklin categorized technology as holistic or prescriptive[1], where it is something wielded or something that directs us. Even a social media app which lets us create content- a seemingly holistic act- still has narrow prescriptive channels we can not escape. We will never be able to understand or enhance this tool. We will never understand it, never see it's nature. This, to me, is the definition of what Erik talks about: an abomination, a thing beyond comprehension, a horror outside of reality, the form of existence which is shared.

I feel like we're reaching a crisis where we are creating an unknowable, unexplorable world. We're building an anti-Enlightenment prison. That, to me, constitutes a deontological hazard, demands that we assess the action themselves of creating unexplorable software.

[Edit: I misread the line I quotes as, "what are you going to do with it besides effectively be robotic slaves": that uhh changes the pertinence of our two discussions here notably. I think it's risky that the strong ai would be used to try to architect policies/systems that steer people, which is a different concern than Erik's.]

[1] https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...

> All to say: discussions about controlling or stopping AI research should be deontological—an actual moral theory or stance is needed

I don't see this happening in the near future, at least in the West. We're in the middle of total epistemological meltdown, and only capable of reasoning from utility, or some insane framework like critical theory. If we get to Strong AI in our lifetimes, we're just going to spawn a bunch of reductive, racist robots.

How about this: "The primary objective of humanity should be to build an intelligent system with far more precise perception, reasoning and physical manipulation capabilities than humans"

That's my starting point.

Unexamined anthropocentric hogwash.

If an AI is created that has consciousness and is smarter and more moral than humans, it could lead our world better than we currently do. It could genetically modify us to make humankind a more ideal species. These AIs could be better humans than humans can possibly be.

In 100-200 years, "kill all humans" could actually be the morally-best choice. By comparison, we could be anachronistic barbarians suitable for nothing but the novelty of viewing in a zoo.

There are certainly risks from passing the baton to a different unknown future lifeform, but I don't think we can have a priori certainty that replacing humans with AI will be inherently bad.

People have done this for most of history. Working a farm, for example, is non-trivially dangerous and fairly low profit. Children often helped on the farm in rural communities from a young age. So every time you had your child work the farm you were rolling some dice. Over and over. But eventually those quarters add up to enough to put food on the table so it was rational to roll them.

This seems like the sort of philosophical argument only someone who has grown up a very privileged life and hasn't experienced much else would make. To them it is inherently wrong but to other it is simply part of life. Which inherently makes it no longer a universal axiom but a matter cultural upbringing.

The potentially limitless power of AI is going to be too much of a temptation for some governments, militaries, powerful individuals, and various groups to resist.

Ban AI in one place and it'll just be developed somewhere else.

That's not to say we should simply march forward with our eyes closed, but efforts to stop AI are highly unlikely to succeed.

Jacques Ellul[1] talked about the almost inevitable and freedom-denying aspects of what he called "technique" (really the relentless drive towards efficiency across all disciplines) in The Technological Society[2].

Ellul's thought inspired the Unabomber, though Ellul himself never advocated violence as a response. As we all know, the Unabomber did not himself stop technological progress... he did manage to spread his ideas a bit, but that hasn't seem to have had any effect either.

I'm completely against violence myself, and unequivocally condemn terrorism of all sorts (including the Unabomber). Not only is it unethical and immoral, but it's ineffective and counterproductive.

[1] - https://en.wikipedia.org/wiki/Jacques_Ellul

[2] - https://en.wikipedia.org/wiki/The_Technological_Society

Anyways.

Let's assume the best and brightest dog breeders endeavor to make German Shepherds as intelligent as they possibly can. Would the same ethical debates about what constitutes a 'mind' come into play? What would happen if the dogs became smart enough to make their own mating decisions? Would we be worried about them turning on us once they get close to human level intellect? Would it be immoral to make these dogs work? Or, would not letting them work be considered immoral?

This is just food for thought. But I suspect AI's capabilities will grow much in the same way other domesticated species have grown into the specialized roles we've crafted for them.

"artificial intelligence system (AI system) means software that is developed with one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with"

"Annex 1: Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning; (b)Logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems; (c) Statistical approaches, Bayesian estimation, search and optimization methods."

Worrying about Superintelligent AI is like worrying about over crowding/population on Mars.

Theoretically, how would we create human-animal hybrids? That’s a strong claim to say it’s within our power.

You know, as I am reading The Cultures series, I can't help imagine how much fun would be to have a Mind taking care of a few things for us.

We need a Butlerian Jihad against AI - https://news.ycombinator.com/item?id=27698233 - July 2021 (3 comments)

This is nothing more than an appeal to authority, no? Even in this proposed axiom there's plenty of room to disagree (even if the author rejects that there is)

And yet we're talking about banning stuff right now which is little more than slightly better statistical regressions?

If that doesn't work then moral arguments against strong AI will not work.

Is there a danger? Yes. Is it because the AI might escape control? No. Is it too late? Yes.

In cheap SF, somebody always says, "we can always pull the plug." But we too often can't pull the plug. Consider the simplest robot we all obey without thought: the traffic light. What would it mean to pull its plug? What would have to happen for its plug to be pulled?

If the traffic light served no one's interest, somebody would pull it. But few things serve no one's interest. The worse a system is, the more integrated it is into someone's interest. Any AI we would like to unplug is almost certainly protected by law, and by numerous layers of physical protection. Lives may even have come to depend on it, daily.

Any AI will be designed to to serve somebody, and will most likely continue to appear to serve them. Just, not necessarily you. But maybe even you, too, much of the time.

We already have numerous AIs controlling myriad details of modern life. We call them corporations, institutions, agencies. Their parts are made of job descriptions and papers, offices and spreadsheets. They are very inefficient. But they are, nonetheless, terrifyingly effective. At times they consume people by the million, seemingly almost autonomously. Ask Uyghurs whose organs are about to be harvested how much it matters that, in principle, a person could turn off the machine that has ensnared them. Ask everyone hoping for a transplant how many would turn it off.

A strong AI would be, simply, an institution that is not so dependent on fallible organic components. It would be more efficient than what it replaces. But the institutions we have are entirely efficient enough to cause all the unpleasantness we can devise.

We are well into an era where the institutions we built to serve us, or anyway to serve somebody, do not. Who can have much effect on the choices of even such a minor institution as the New York Police Department? Who would be able to cancel the F-35 program? The Chinese Communist Party and Shell Oil have both outlived their creators, and are well equipped to eject any human components that do not serve their implicit charters.

- Our problems are getting more complex. We need better AI.

- Better AI is a threat.

Great. We aren’t!

"The philosopher John Searle made precisely this argument about the standard conception of rationality. His point was that there are no odds that would rationally allow a parent to bet the life of their child for a quarter. Human nature just doesn’t work that way, and it shouldn’t work that way."

I agree that it sounds morally repugnant to risk your child's life for a quarter, but in practice people do do this all the time.

Imagine your child wants ice cream. There is some utility in taking your child to the nearby ice cream parlor. Your child will be made happy and you will be made happy by making your child happy. However, this is not infinite utility. In other words, there is probably some amount of money I could offer you to not take your child to get ice cream today. If I offered you 10,000 dollars to not get your child ice cream today, I bet the vast majority of people would take the deal. That sets the upper bound of the utility of taking your child to the ice cream parlor at 10,000 dollars.

Suppose the ice cream parlor is 3 miles away (I just checked the distance to my favorite ice cream parlor and it is 3 miles away). In the United States this website[1] says there is approximately 1 death per 100 million vehicle miles traveled. We could rephrase that as 1 * 10^-8 chance to die per vehicle mile traveled. This risk may be high, presumably you aren't drunk or impaired, maybe you're a better driver than average or have a safer or better maintained car, or live in a safer place, but the risk of death isn't zero.

If you are willing to drive your child 3 miles to go get ice cream then it seems like you are willing to expose your child to the risk of death from car accidents for utility that is less than 10,000 dollars. Putting those ideas together we could calculate the odds where a parent would, in practice, risk the life of their child for a quarter.

I don't quite know what to make of this. I tend to think that people would regard doing odds calculations like this for real life decisions as somewhat sociopathic and would just prefer to live as if significantly unlikely bad things were impossible or just refuse to think about the moral implications of probabilities. That seems similar to what the article is saying, people just prefer to live as if bad super collider or AI experiments won't happen rather than reason about them.

I tend to think that being too "reasonable" on a local scale is bad. That is, I will still take my children to go get ice cream even though I know driving is a risk. At higher levels though I want people to be making decisions that are increasingly based on reason and probabilities. I do want the traffic engineers to be reasoning about vehicle deaths per mile and the like when they are setting speed limits, traffic signs, and the like. For things like AI and, I suppose, super colliders, our decision makers should absolutely be considering things rationally.

1 - https://www.iihs.org/topics/fatality-statistics/detail/state...

“I tried to convince people to slow down. Slow down AI. To regulate AI. This was futile. I tried for years. Nobody listened. Nobody listened.”

Rather rich coming from one of the self driving car market leaders. It certainly makes business sense to mislead academia and the policy sector into wasting resources on figuring out the best philosophical and ethical regime while large corporations benefit from regulatory capture. If he has his way, ML would become like the pharmaceutical industry, with multiple barriers of entry if you are not well-funded, well-connected, or established.

"Oh, but China will do it anyway, so there's no point."

Which is pretty easily counterable:

1) We don't know if China would cooperate with a ban. We haven't tried. China is very complicated, and if you think you can predict what Xi will do, you are wrong.

2) If AI is truly a global existential and moral crisis, the US _could_ absolutely shut down China's AI research capabilities. There are a few avenues here, some less pleasant than others. Think outside the box.