Yoshua Bengio: How Rogue AIs May Arise
yoshuabengio.org
yoshuabengio.org
> Now imagine AI systems like corporations that (a) could be even smarter than our largest corporations and (b) can run without humans to perform their actions (or without humans understanding how their actions could contribute to a nefarious outcome)
Wait - but it’s those very corporations who currently control the most powerful AI on the planet - so in a sense this is an argument that we already have rogue AI. Corporations with goals that are not aligned with society, not controlled by any individual and outside the effective control of government entities, who now provide essentially unlimited compute and human researcher/programmer resources to improve the AI. Who’s in control? Nobody.
The thrust of this argument is that centralized authority -- perhaps uniquely centralized, as Bengio proposes an agreement between Russia, China and the USA as an example -- is the best safeguard against rogue AI. He goes as far as calling out open source development of AI as a contributing threat:
> [if] the open-source community continues to play a leading role in the software development of LLMs, then it is likely that any hacker will have the ability to design their own pre-prompt (general instructions in natural language) on top of open-source pre-trained models...
The concern is logical, but the solution is self-contradictory (as many intellectually honest explorations turn out): a section later, we are appropriately exploring nefarious corporations as examples of mis-aligned intelligences -- and how larger corporations may be even more adept at circumventing the rules.
Whatever their philosophical foundations, our governments are also corporations (and so will be any new trans-national organization). They are fragmented into smaller groups with their own instrumental goals, and subject to the same risks of corruption that afflict private corporations, and the same risks of runaway optimization that may afflict AI. Appealing to them as a higher authority is at best a case of "the devil you know".
Doomism always has a non-falsifiable counterargument: AI is new and might become arbitrarily more powerful than the most powerful organizations we've ever created ("super"-intelligence). For me this fails on several counts: we are still exploring the limits of power (and of corruption) that large organizations can achieve, progress in AI remains observable as long as it is public, and our primary experience with AI has been colonizing a frontier of capabilities rather than a sequence of new systems dominating at every task.
To be clear, I do NOT personally conclude that we should avoid all regulation and oversight. But I think that constructing regulation around doomist theories serves disproportionately towards regulatory capture, and acts as a distraction from immediate issues of algorithmic inequality, disruption of political speech, and intentional weaponization of AI.
Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences.
AI is out and about. You cannot prevent humans from developping it.
4 to 8 million dead: https://en.m.wikipedia.org/wiki/Thirty_Years'_War
Ridiculous take. AI ain't nukes, you don't need a thousand centrifuges to separate it from raw AI ore. A huge amount of tiny actors are sufficiently equipped to work on it.
What? No. They spend their money fighting with other wealthy people over who gets the passive income from the paperclip machine. The game they play is about maximizing this income. Whether or not this also maximizes paperclips is completely immaterial to them. If they had an opportunity to form a paperclip cartel and profit from cutting supply, they would be happy to do that instead. Yes, the idea of a paperclip cartel is ridiculous -- but no less ridiculous than the idea that maximizing paperclips maximizes investor profit.
Companies are about investor profit. Everything else is a side effect.
The author is trying to convey that the situation with corporations and human labor doesn't have to be an extreme of either using human labor or not using it at all. They argue that corporations are gradually relying less on human labor, but that doesn't mean it has to drop to zero for negative consequences to occur within society.
They also bring up the stock market as an example of a system that focuses on optimizing for profit (paperclips) rather than human welfare. The author implies that many wealthy individuals primarily focus on accumulating more wealth (paperclips) and may only contribute a small portion (guilt tax) to alleviate any sense of responsibility for the negative consequences of their actions on society.
Paperclips are an extremely poor metaphor for financial assets. Rich people want to get paid for being rich. That's what financial assets do. Paperclips do not do that.
My point here isn't so much to pass moral judgement on capitalism as it is to assert that money and assets have fundamentally different character. They are easy to confuse due to the liquid market for exchanging the two, but the lines of reasoning that lead us to a concept of "enough money" do not lead us to a concept of "enough assets." We might independently arrive at a concept of "enough assets" by hypothesizing a situation where one person controls a 10% of world GDP or something but world GDP is 100T and the richest people own about 100B in assets so at a 10x multiple they are probably at least three orders of magnitude removed from a self-serving definition of "enough assets" (and that would be the relevant definition, since they would be the ones defining "enough.")
tl;dr Rich people DO stop maximizing money -- but then they start maximizing power, which can be bought with money, and which we measure in units of money.
> "By the end of 1933, the Reich agreed to buy all of the Leuna factory's output that couldn't be sold on the market, Jeffreys says. That same year, Farben donated more than 4.5 million Reich marks to Nazi Party funds." (Diarmuid Jeffreys, "Hell's Cartel", 2008)"
https://web.archive.org/web/20201101015513/https://www.chica...
They were certainly the power behind the throne, and active participants as well.
Corporations might have had good business etc., but that again isn't the same as being in control.
> "Business Collaboration within the Nazi War Machine: Corporations and the State in the Austrian Semiperiphery, November 2019, Canadian Review of Sociology, Clarence Lo, University of Missouri"
Certainly the heads of those corporate entities (IG Farben, Krupp, Deutsche Bank) that the Nazi regime depended most heavily on had a degree of power and autonomy on a par with the political and military leadership, and were joint partners in, and joint beneficiaries of, the enterprise. It's true that other German business interests were in a more subservient position, however.
What it doesn't say is that those heads of those corporate entities could go against the political leadership when exercising their power and autonomy (very late stages not so sure, never looked at that in detail) - their power come from an alignment of sorts. You can of course say, if everything is political in a system, then maybe the flip side is everything is corporate - but that didn't hold for the sources of physical violence, for example.
Purely military leadership possibly a little bit (more), because that was "below" political leadership in a way.
https://www.youtube.com/watch?v=L5pUA3LsEaw
>Who’s in control? Nobody.
This falls in with Meditations On Moloch.
Yes. Science fiction is pretty much 100% about things that have already happened.
Colonization by aliens? Colonization by Europeans.
Robot slave uprisings? Human slave uprisings.
AIs taking over the world? Capitalism taking over the world.
Once you realize that sci-fi has to be about things that have already happened for writers to be able to write convincingly about it, it's hard to unsee. It's all just "what if the past but in the future???"
Greed for money and power. And for an eternal life after mind transfer, people like Musk are already dreaming about it. Surely, it's not for everyone.
If rogue AI(s) emerges then I suspect it will come out of military r&d, rather than academia or industry. The deep resources already exist. The motivation will be the need to build it before $enemy$ does, rather than to "destroy humanity", but I reckon it creates more momentum than the profit motive when coupled to state power. Its the same pressure that got us nukes and mutal assured destruction. Military secrecy will pretty much guarantee inadequate oversight. Military discipline will ensure that most (but maybe not quite all) of the people working on it will suppress their ethical concerns, or will have none to begin with.
The fact that the AI will necessarily be trained to kill (which will necessitate a lack of hard-wired moral constraint) will help too, I guess.
I suspect there are already things going on in anonymous mil/defence labs that would give many people nightmares.
Assuming an AI that manages to (or is designed to) escape its containment:
Armed, remote-piloted drones already exist. So do nuclear and conventional weapons that are targeted and launched by software-based c&c systems.
Inject realistic fake attack sequences into BMEWS and whatever the Russian and Chinese systems are called and watch the meat bags do the work.
Nuclear power stations. GPS spoofing to collide aircraft and autonomous vehicles over cities.
Reality disruption by extremely large scale disinformation, combined with targeted food supply, utility, and comms system outages.
Socially-engineer humans into causing death and destruction using bribes/threats or mental destabilisation thru engineered conspiracy theories.
Etc.
AI would need to overcome a fair amount of systems, both physical as well as software for a lot of these things. It needs a lot of "thinkism" to make that work.
Because the AI fabricated a target package doe it?
> How does an AI arm a nuclear weapons? How does GPS spoofing crash an aircraft? How does AI do something to a nuclear power plant?
By infiltrating non-internet networks?
> Can we not shut down the internet if we needed to combat misinformation?
A lot of people seem to enjoy soaking themselves in conspiracy and disinformation. No my guess would be no, we wouldn't ahut it down.
> AI would need to overcome a fair amount of systems, both physical as well as software for a lot of these things. It needs a lot of "thinkism" to make that work.
Thats a fair criticism. Its unlikely at present, but as I asserted in my original comment, military exploitation of AI seems inevitable (to me) and the forces at work are irresistible.
It might never happen that AIs can do all this and/or that defense AIs severely limit what offensive AIs can do.
Why not both? The Horizon franchise's lore has a "fun" example of the military-industrial complex converging on Armageddon.
Consider this: our world is already filled with numerous AI models of varying power and functionality. In the event that one goes rogue, wouldn't it be plausible for other models to step in, contain, and mitigate the damage? This dynamic is not much different from what we have witnessed in the field of cybersecurity, where security teams and hackers are locked in a constant struggle — a patch and breach cycle.
Also, let's think about the future. We're close to a situation where we don't always have the upper hand over the tools we build, but that's a continuum of the path we have already been treading. It will require collective effort, both from humans and AIs, to combat any rogue AI or even a fleet of them.
The notion of AI being centralized doesn't line up with what we're already seeing play out. An AI ecosystem with hundreds of variations, serving as a safeguard against a rogue AI, seems like a more promising and secure future. We should be focused on developing a diverse array of AI models rather than worrying about a singular rogue entity. Our defense strategy against potential threats lies in the broad dissemination of this technology, not in its undue concentration.
Conversely you're making an argument to maintenance of that status quo. It makes zero sense that would happen.
As analogy lets imagine a nice balanced ecosystem with lots of plants and flowers and insects and animals that's been going on for a while. Then suddenly you drop Kudzu in it and it begins spreading like wildfire and quickly puts pressure on at risk plant and insect species. You release another insect that eats Kudzu only to try to reduce the amount of it, but in the time it takes the insect to breed up to numbers to combat the Kudzu, it's spread even further and wiped a few species out, and now is putting pressure on the larger animals that cannot eat the Kudzu.
In natural environments you don't really ever get your old ecosystem back, something are just gone and cannot be supported any longer, and there is a new balance between competeory species. AI is a potential competeory species to humans. Escalation between AI systems present a far larger risk to humans than direct AI versus Human actions in my line of thought.
Not only clusters of AIs but also running at 100 or more times human speed.
You can make a pretty good self-replicating virus-like AI now based on the Vicuna LLM model. Just give it directions to install a virus that makes copies of its own model and software and replicates.
Nvidia has a "superchip" coming out this they are advertising as 10 times faster than the previous generation (I assume H100 or something).
We know that these models can be optimized, such as the huge speed increase between gpt-3.5 and gpt-3.5-turbo.
So that is something like a 20-70X efficiency increase with existing technologies/techniques that just haven't been fully deployed.
Assuming some algorithmic improvement in the nearish future, I see no reason GPT-5 level AI in a GB or two isn't right around the corner.
(For comparison, GPT-3.5-Turbo is likely a finetuned Curie 13B, which is just about 8GB in 4bit.)
Just a nitpick on this point, can you elaborate?
Are you talking about Grace? (Which is a Hopper H100 with an ARM CPU glued to it)
Though, please note that NV give their FLOPS as 'sparse flops' which is ~double the FLOPS the chips get for dense matmuls. This can trip you up when comparing numbers!
Superintelligence is not necessary for rogue AIs to cause big trouble
> Hypothesis 2: A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages.
Sure.. but these are sci-fi scenarios. Chatgpt has gotten people thinking, which is good, but let's not lose perspective on what is actually happening before we get too worried.
> An AI system in one computer can potentially replicate itself on an arbitrarily large number of other computers to which it has access and, thanks to high-bandwidth communication systems and digital computing and storage, it can benefit from and aggregate the acquired experience of all its clones; this would accelerate the rate at which AI systems could become more intelligent (acquire more understanding and skills) compared with humans. Research on federated learning [1] and distributing training of deep networks [2] shows that this works (and is in fact already used to help train very large neural networks on parallel processing hardware).
Now this is speculative and, to be blunt, kind of stupid. The proposed AI is now a self-replicating swarm intelligence that can self-improve and the rate of improvement increases the more it replicates. How'd we get there?!? To work even in a novel you'd going to have to at least hand-wave in some justification for all that.
[1] and [2] do not show that scaling compute and storage has no limits in terms of scaling AI intelligence. Seems obvious that there would likely be plateaus. E.g., training an AI on the entire content on the internet probably provides a huge gain over training on a tenth. Training an AI on the entire content of the internet a second time provides no additional gain.
Also, it's somehow assumed in here that the suggested future super-intelligent AI will actually have the capability to cause catastrophic harm. I don't get where that assumption comes from. We'd have to grant such an AI some set of magical capabilities, which aren't named. Is "superintelligence" a magical ability to do anything?
IDK... to me this reads as "The sky is falling!"
When you say "sci-fi" scenarios, what does that mean? Just something speculative about the future? Something that appeared in sci-fi media?
People say this as if the fact that someone thought about a scenario that can happen and wrote a story about it has any bearing on whether it's something worth worrying about.
> Also, it's somehow assumed in here that the suggested future super-intelligent AI will actually have the capability to cause catastrophic harm. I don't get where that assumption comes from. We'd have to grant such an AI some set of magical capabilities, which aren't named. Is "superintelligence" a magical ability to do anything?
The way most people define "intelligence" for these purposes is "ability to achieve goals". Which would mean that, if there were any way to cause catastrophic harm, and the hypoethetical AI had as its goal creating that harm, it could do it.
Is your contention that there is no way to cause catastrophic harm to humanity? Cause I think even with my regular ol' human intelligence, I can think of plenty of ways to cause harm.
> Now this is speculative and, to be blunt, kind of stupid
I'm not saying that Yoshua, or anyone, is above criticism. But it might be worth considering that we're talking about one of the leading academics in CS and AI. He could totally be wrong! But accusing him of saying things that are just stupid is not very convincing, IMO.
I suspect that the answer is no. Any intelligence capable of human level reasoning would be able to choose “evil”.
So, I think we need to shift the discussion from if rogue AIs will arise to when rogue AIs arrive.
However, I believe that in order to compete, hyperspeed AIs directed by humans will need greater and greater levels of autonomy. Otherwise they will be stalled waiting for directions while their competition runs circles around them.
It is also almost inevitable that someone deliberately creates intelligent AIs that mimic various aspects of living beings such as full autonomy, control-seeking, etc.
I think the trick is to separate out all of the different facets of animals and humans apart from intelligence or "human-level". There are things like emotions, autonomy, stream of subjective experience, types of adaptability, etc. that are not the same as intelligence or reasoning ability.
It does not have that by default, but agency and autonomy (even the run forever kind) is trivial, though expensive to imbue.
and LLMs disregard instructions all the time. Prompt injection is nothing more than an LLM deciding a previous instruction would be better left unfollowed.
What does “choosing wrong” mean (to you) ?
I can easily imagine a machine with no agency, but which can synthesise intelligent responses to input data. In fact, this is what chatGPT does.
If a machine has no goals or agency of its own then it is an intelligent tool. It can, of course, be used for evil, but you would not say that the machine was "choosing" evil.
Tangent, but if this is true (which seems likely to me) ...
If a hypothetical creator valued the existence of other intelligences more than it valued preventing all forms of evil, that would account for the problem of evil.
This also fits with annihilationist theologies. Build a universe where intelligences can exist and put heavy survival / selection pressures in place (i.e. "natural evil" - hurricanes, volcanic eruptions, meteor impacts, etc) to encourage evolution and development.
Whenever the process yields intelligences that converge on its values, export them to an environment where it can interact with them and enjoy their company with less concern about their alignment (i.e. "heaven"). Ones that don't converge on its values can be discarded when their run is over ("the second death").
In this light, it doesn't sound crazy to postulate our universe as some being's attempt to solve the alignment problem.
Call it the Simulator, call it God, call it "underpaid research assistant" - take your pick.
I don't have good evidence for this oddball proposition, but it's still intriguing to me.
People will simply release them without thinking of the consequences.
If the cost of intelligent AI is dramatically cheaper, society will change overnight and there’s little we can do to stop it. There’s no way to design a global treaty when defection results in massive massive cost savings.
If the cost of intelligent AI is dramatically more expensive, there’s zero risk of a rogue AI taking over the world. who would waste that kind of money delegating unnecessary tasks to an AI? So it will never have the keys to the kingdom because it’s cheaper to have a human hold on to those keys.
Most “rogue AI” scenarios begin with good-faith operators handing high credentials to an AI and then the AI abuses them.
If I’m an evil billionaire and I can build a human-level AI for $200MM in hardware costs and $1MM/month in electricity, maintenance, etc, all for one single human level intelligence, how exactly am I supposed to leverage this to take over the world?
You can't help but think of AI with our limits around it. AI doesn't have to sleep, so it's already 3x more available than a worker on a daily basis.
Then you have the forward progress of technological growth. It's 200M now, 20M in 5 years, and 2M in 10 years, all while the operating costs drop. Meanwhile your human costs have only gone up in that time.
Then you are likely stuck on the idea of a single conscious train of thought, much like the human mind is limited to. Such limits in AI both seem unlikely and not optimal, lest of course you have a bunch of threads thinking about the same thing without realizing it.
And then you're ignoring the massive amount of brain processing needed to keep you alive that could otherwise be used for logical computation. Instead of a huge network of neurons to make sure you're not getting hurt, that could be data feeds watching financial data. Your limited to 2 eyes attached to your body and a tiny area of focus they provide. Again, why would an AI have such tiny input limits?
If 'attention is all you need' than this AI system will be able to pay far more attention to what you want it to then any human would ever be able to.
To put it another way: the specifics about hardware don’t matter at all. All that matters is the dollars required to complete tasks. If Russia can have the best cyberwar unit in the world for a few million by ignoring an AI treaty, they will do so. If the cost is a few trillion, they will not violate the AI treaty because they don’t have a few trillion to spend on cyberwar and could still have a world class cyberwar unit for orders of magnitude cheaper. Then, the AI treaty is unnecessary: the economics already determine the outcome. The treaty only makes sense if the cost of human labor and the cost of AI labor are very coincidentally nearly the same.
> Your limited to 2 eyes attached to your body and a tiny area of focus they provide. Again, why would an AI have such tiny input limits?
A human organization is not limited in this manner. You can scale up human personnel at cost as well. You might expect that humans scale suboptimally (organizations suck at utilizing labor) but you can build this into your model while projecting total cost of human labor required.
These machine learning systems replicate and automate existing market & organizational logic. The fear I have with them is the ruthless effectiveness they can have in the hands of profit and control motivated actors.
Think Russian propaganda "bots" are bad now? Just wait.
Again, my fear is not rogue AI, it's rogue humans.. with AIs.
Then point Yoshua is making that we need need have disincentives for things like “Russia” (the rogue state) to exist.
Like living in a more fair world where living and protecting life is the default thing to want to do.
All this division as our tech becomes more powerful will for sure lead to destruction.
> A potentially rogue AI is an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere.
Is it only rogue if it can "catastophically" harm a "large fraction" of humans? Does "large fraction" imply greater than 10%? 25%? 50%? Does this definition imply that catastrophically harming a small fraction of humans is acceptable? Or moderately harming a large fraction?
Asimov wrote first law of robotics in 1942: "A robot may not injure a human being or, through inaction, allow a human being to come to harm.". Why not start from here?
But none of that stuff prevents someone from just not giving it those types of rules, or deliberately designing it to gain control over its environment.
There are far more good guys in the world than bad guys. Look at email for an example. 99% of people use it for it's intended purpose, but maybe 1% of people want to use it for spam and other nefarious ends. The good guys have created a system that is constantly besting the bad actors.
Yes, the effort is asymmetrical, but the good guys outnumber the bad guys by so much that it isn't an issue.
An AI built to run open ended workloads, but somehow ends up doing so in a harmful way, combined with a lack of engineers able to effectively diagnose the problem quickly enough, and the AI unable to be shutdown due to automated high reliability systems, means you end up with an AI losing $10 million dollars a minute and you only have 30 minutes before the entire company is bankrupt.
https://en.m.wikipedia.org/wiki/Colossus:_The_Forbin_Project
Machine gains super intelligence, and so makes new goals to use nuclear weapons to reduce the human population.
Are we comparing an AI system against a single person, or against humans collectively? Usually there's a statement like Hypothesis 2 — "A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages" — but then also "although the capacity of a human brain is huge, its input/output channels are bandwidth-limited". But is the relevant comparison a single human brain, or a sort of collective system of humans?
It seems a little misleading in a logical-comparative sense to talk about an AI system built on a collective human system like wikipedia, or the internet, in compared to a single person.
I don't actually mean to question whether a superintelligent system surpassing all human abilities, individually or collectively, could be built. But it seems to be that the concerns about bandwidth and whatnot are a function of the problem definition, and not necessarily a single human system, when a task-solving exercise involves a collective output in both cases.
I think going forward we're going to need much more clarity about what "superintelligence" really means. Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source.
If/when AI starts focusing more on reasoning per se, it will probably also be necessary to better define whether there are some limits to discovery and reasoning, in the sense of bounds due to information about a problem domain versus reasoning ability limits. That is, sometimes we know the possible options but just don't have enough empirical information to decide between them; other times we lack explanatory options. Maybe that endeavor also is constrained by human abilities but it seems worthwhile to explore.
Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else. That is, AI sort of proceeded in a more experimental way without mathematical theory driving it, so we have some catching up to do.
>I think going forward we're going to need much more clarity about what "superintelligence" really means.
This was already well defined by Bostrom in 1997: "An intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills"
>Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source.
There are no indications at all that it is constrained to that level.
I'm still quite concerned about the risk of rogue AI, as it's not even necessarily true that humans will have the advantage of numbers, or be able to bring our numbers to bear. And a smart AI will try to divide us and conquer, if it comes down to a conflict, including recruiting humans to its side.
Then I would say your expectations are a failure of your own imagination.
The problem with humans being the smartest things on the block is we think we are optimized for intelligence. One problem with wanting to clarify what 'super intelligence' is, is we really have piss poor definitions of what intelligence is on a multidimensional gradient scale. The bounds and limits of intelligence are poorly defined, but I scenerly ask, why would you think that humanity would be anywhere near the upper bound of those limitations? Why would evolution choose 'the smartest possible global maxima' versus 'the smartest possible with 2 pounds of jelly in a 20 watt power envelope?'
All the hand waving about what methods an AGI would use to kill all humans could equally apply to government research labs, terrorist cells, 4chan, or Indian spam call centres.
So you're going to start worrying at the very last possible second, when it's far too late to do anything to stop it?
It's the only way to save democracy.
You know, come to think of it, it's funny how the Russian disinformation is now AI disinformation, instead of human disinformation. As we know, Photoshopping single photos of smoke rising into the sky is notoriously labor intensive, requiring teams, if not the entirety of state-level sanctioned organizational output, of human beings. The game has changed, and democracy is at stake.
Alternatively, all ML training data could simply be required to include an "evil flag" to indicate whether it is being used for evil purposes. And ML algorithms could simply be required to refuse to run if the evil bit^W flag is set. Problem solved.
Why should we imagine that the individual genocidal human with his or her rogue AI has access to methods of destruction which do not require "huge capital" but which have global effects?
Async affects outside of current power dynamics and upset entire systems and cause rapid collapse. In the case of internet DDOS it's a botnet sending out a few MB/s of data and generating GB/s of attacks. In the case of a superintelligent AI its _______?
The implication seems to be that the superintelligent AI will somehow figure out a way to destroy the world that requires significantly less than the "huge capital and expertise" needed for the nuclear scenario. But that's a critical assumption that requires justification, not something to be left as an exercise to the reader.
Bio-terrorism is a potential massive risk as proliferation of technologies for customized gene therapies continually develop. If you have the tooling and knowledge to make rice have a bountiful crop, then it's highly likely this skillset easily transfers to dramatically affecting the rice crop via disease. Could a genetic scientist do this now? Likely possible, but if you have a scientist in a box with no ethics and all the time in the world, you're apt to get pretty far.
> Rejecting hypothesis 1 would require either some supernatural ingredient behind our intelligence or rejecting computational functionalism, the hypothesis that our intelligence and even our consciousness can be boiled down to causal relationships and computations that at some level are independent of the hardware substrate, the basic hypothesis behind computer science and its notion of universal Turing machines.
I think I'm going to stop there. I can reject that hypothesis without woo, and without rejecting the entirety of computer science. Maybe the rest of it is a decent read, but that run-on argument is confused/dishonest and makes a poor foundation to build his position on.
The reason that people reject this is almost always because of a worldview that incorporates https://en.wikipedia.org/wiki/Mind%E2%80%93body_dualism
YB starts off with ~'we all agree mind is a product of the brain'. No, we do not all agree. Did we ever find the 'bottom' of matter? Has are search to find 'irreducible particle' succeeded? Is it not the case that as we threw more energy into our apparatus we found more elementary matter? Did we in fact finally arrive at a coherent unified model that fully explains material reality without "mathematical woo"?
So the honest position is that we have patchwork understanding of reality at various scales, and the reductionist program of Democritus et al has not been conclusively shown to be the ultimate truth. It seems that mind is bounded by the body but that 'boundary condition' depends on various reductionist assumptions about materiality.
If anything I was just trying to suggest that the mind (or soul) and body aren't completely separate things as some religious or pre-scientific viewpoints might hold (touched on in part of the Wikipedia article I linked to).
Maybe I didn't do a great job of stating where I was coming from.
You sound like Donald Hoffman when he was on the Lex podcast. I agreed with a lot of things he said.. I mean the idea that literally reducing things to smaller and smaller particles is not the ultimate explanation of things makes total sense. Along with the idea of using these mathematical exotic geometries instead or something.
Where he loses me is going from embryos to humans that are somehow dictating the shape of reality. We have great explanations for development and evolution. But there doesn't seem to be a good explanation at all for how fully formed minds somehow shape reality in the past. Or how this works with less intelligent life.
That seems to be the case. I take that assert back.
p.s. just checked out the wiki for Donald Hoffman. Thank you for the reference, TIL. Have to listen to that podcast, but have some idea what are the problematic open Qs that you mention. [I'm not apparently not allowed to vote on hn so take my +1 in spirit ..]
Also, mochomocha underscores the more important half of it.
How?
1) Complexity of the human brain is higher than we thought, reproducing the complexity necessary for human-level intelligence is further into the future than we initially expected.
2) Intelligence is bound by size/capabilities of a body. (ie, input & output). If you want human-level intelligence, you'll need a human-level robot too (not impossible).
That also means if you want a god-like AI, you need to have the inputs/outputs (body) of a god too.
3) Human brains are _not_ biological machines. They're biological -paradigm we have not discovered yet-. The idea of computation itself is limited by something we cannot "just" reproduce with engineering.
I doubt this, but it's not impossible that the brain employs totally physical phenomena that are basically impossible to reproduce outside of biology. Why? Because biology is something fundamentally different from engineering.
Doesn't mean we can't control biology, but still.
---
The real hypothesis is: Human-level intelligence is possible because it exists. Our best understanding of how human-level intelligence works is with the model of a "biological machine". We assume that it is possible to replicate this biological machine in the near or somewhat distant future as our understanding of engineering, biology and intelligence grows. This replication will have the same or similar intelligence-characteristics as a human.
I didn't know what "computational functionalism" was, but after reading about it I don't see how it is remotely the "basic hypothesis behind computer science".
On what basis do you reject the hypothesis? You didn't say
It doesn't matter whether the opponent is "really" doing something. The humans are going to get crushed. Consolation prize that we can go to our doom claiming that the robot economy isn't as spiritual as its antecedents.