The Danger of Superhuman AI Is Not What You Think
noemamag.com
noemamag.com
It's going to be used to pump as much money out of each individual as possible
Every online store, big corporate especially will instantly realize how much you are able and willing to pay for everything and that's your "personalized price"
It will scan all your social media, it will look at your purchase histories, it will know your income and credit
Like a used car dealer on steroids that never gets tired and learns more and more about you by following you around and watching what you do
Everything from your daily food to your big ticket purchases, maximum customized prices
It's going to be evil as hell and lawmakers will just be paid off eventually to let it all happen, data collection and no two people paying the same for the same thing will become 100% legal and the norm
Now, you can pool your resources together against the big AI using companies if you like, but that is going to cost a massive amount of money and time to fight them still. So humans typically like to put these resources in to a government that at least attempts to look out for their best interests.
There is probably some kind of fallacy somewhere like "everyone thinks they will be the one who benefits from a power imbalance until they experience losing first hand".
https://www.vox.com/policy-and-politics/21534416/free-state-...
https://www.newyorker.com/humor/daily-shouts/l-p-d-libertari...
If it was the first, I can assure you the comment was lamenting the situation, not celebrating it. I even chuckled to myself that the fallacy in question is called "Libertarianism".
If it was the second, the best advice about joking I have ever heard is "if at all possible, involve a cow". I'll be looking out for it next time ;)
When it comes to food, there is far less competition than one would expect. The distribution markets have become ran by just a few companies. And those 'competitors' will gladly subscribe to some service like RealPage for food so they could massively increase their profits.
> How, I asked, does an AI system without the human capacity for conscious self-reflection, empathy or moral intelligence become superhuman merely by being a faster problem-solver? Aren’t we more than that? And doesn’t granting the label “superhuman” to machines that lack the most vital dimensions of humanity end up obscuring from our view the very things about being human that we care about?
This is such an annoying framing. There are massive assumptions throughout this, from "human capacity", "conscious self-reflection", "moral intelligence". And then throwing those assumptive definitions into another persons face, demanding how they relate to "superhuman" intelligence.
It's almost like the author is saying "I have intuitive definitions for this grab-bag of words that don't align with my intuitive definition of this one particular word you are using".
> characterizations of human beings as acting wisely, playfully, inventively, insightfully, meditatively, courageously, compassionately or justly are no more than poetic license
Again, the author has some intuitive sense of the definition of these words. If the author wants to get spiritual/supernatural/mystical about it all then they are free to go ahead and do so. They should come out and make the bold claim instead of masquerading their opinion as reasonable/rational discourse. They are more likely to find a sympathetic audience for that.
>Once you have reduced the concept of human intelligence to what the markets will pay for…”
Bengio and Hinton certainly do not make that ridiculous reduction as a condition or criterion. Even Altman does not, although he certainly feels the pressure.
Vallor taught for many years at a preeminent Jesuit institution and when I read this polemic my first thought is—-please put your “belief system” cards on the table. What is the fundamental substrate on which you are thinking and building your arguments? Would you be comfortable with Daniel Dennett’s thoughts on brain function and consciousness or do you suspect that there must be some essential and almost ineffable light that illuminates the human mind from above or below.
My cards on the table: I am an atheist and I am not looking for another false center for my universe. The hard problem of AGI is now the embodiment problem and iterative self-teaching and self-control of action and attention.
You could say that we need to get the animal into an AI now that we have language reasonably well embedded. From that point onward, and I think quite soon (10 years), it will be AGI.
Perhaps “super” is not the right adjective but it grabs attention effectively.
Like for the first one she has
>Indeed, for the entirety of the Industrial Age, those invested in the maximally efficient extraction of productive outputs from human bodies have been trying to get us to view ourselves — and more importantly one another — as flawed, inefficient, fungible machines destined to be disposed of as soon as our output rate slips below an expected peak or the moment...
Which is a bit strawman. I mean who, as a real person does that?
The second one she's saying why have AI do art when it should be doing my taxes, but people are probably working on doing taxes, it's just the art stuff happens to work well, LLMs filing your tax less so. Although I can see an interesting argument that those false tax deductions were hallucinated by the LLM, not my fault.
Ms. Vallor is too concerned over semantics, too fixated on the current crop of LLMs, too invested in "pinning down" Mr. Bengio.
Five to ten years out though I suspect we will no longer be having arguments over what it means to be human, intelligent.
That seems highly unlikely. We may have more information about the nature of intelligence, but I highly doubt we will have all the answers. As to arguments about "what it means to be human"... that's one of the core metaphysical questions that I doubt we will ever be done arguing about.
This is the question I always ask. I'll be interested in AI when it can do my laundry and my dishes. I guess self-driving cars might fit in here, although I personally like driving, some people hate it and even I could see how an AI driver could be desirable at times (e.g. coming home from the bars).
An AI system can only be dangerous or not dangerous based on what physical events happen in the real world, as a result of that AI interacting with the world via known physical laws. In order to decide whether a system is or isn't dangerous you need to have a predictive model of what that system will actually do in various hypothetical situations. You can't just say "because it is improperly defined as X it cannot do Y".
If you want to predict what happens when you put too much plutonium into a small volume, you can't make any progress on this problem by talking about whether the device is truly a "bomb", or by saying that the whole question is just a rehash of the promethean myth, or that you cannot measure explosions on a single axis. The only thing that will do is to have a reliable model of the behavior of the system.
Many people seem to either not understand, or intentionally obfuscate, that an AI, like a bomb, is also a physical system which interacts with the physical world. Marc Andreessen makes this error when he says AI is "just math" and therefore AIs are inherently safe. No, it's not just math, it's a computer, made of matter, that physically interacts with both human brains and other computer systems, and therefore by extension with the entire rest of the physical world. Now of course, the way that an AI interacts with the world is radically different than the way a bomb interacts with the world, and we cannot usefully model the behavior of the AI with nuclear physics, but the fact remains. (See also: https://www.youtube.com/watch?v=kBfRG5GSnhE)
So when you see arguments like this, ask: Is the person making an argument based on a model, or not? Examples of model based arguments include things like:
- "AI risk is not a concern because of computational intractability" - we understand something about the limits of computation, and possibly those limits might constrain what AIs can do.
- "AI risk is a concern because less intelligent entities are not usually able to constrain the behavior of more intelligent entities" - A coarse and imperfect model indeed, but certainly a model based on observations of interactions in the real world.
"Verbal Trick" arguments include things like:
- "AI risk is not a concern because we can't even define intelligence"
- "AI is just math"
- "AI risk is not a concern because intelligence is multi-dimensional"
A third category to watch out for is the misapplied model:
- "AI risk is not a concern because people have always been worried about the apocalypse" - this is a model based argument, but it can only answer the question of why people are worried, it cannot answer the question of whether there is in fact something to be worried about.
Donald Trump was the president of the United States. FDR was a very smart man but Oppenheimer was smarter and FDR had him build the atomic bomb. This is just wrong, and ChatGPT is nowhere near as intelligent than a five year old (intelligence is not knowing facts)
ChatGPT is vastly more intelligent than most humans on any measure I can think of that existed before 2 years ago when suddenly people started trying to conflate intelligence with sentience.
Intelligence: the ability and scale of capability to navigate a non-deterministic system.
I feel that doesn't help though.
Even among the folks that debate this rigorously, another related problem is “reference class tennis”, as you note above, what is the appropriate anchor for your priors? Eg is intelligence going to compound to a runaway reaction like nuclear fission (since more intelligence speeds up intelligence gains in AI) or is it going to asymptote like the current trend in scientific progress?
I'd argue that the observations are inconclusive. Human dominion over nature is a holistic thing that involves more than just our intelligence -- the particular shape and capabilities of the human body (opposable thumbs, upright posture, stamina) certainly are an important enabling factor without which our intelligence would probably have been pissing in the wind. Within human societies, I see little evidence that more intelligent humans can effectively constrain the behavior of less intelligent humans. Who can manipulate who seems to be more correlated with personality and instinct than intelligence per se, and it's quite failure prone. There are also classes of powerful non-intelligent systems such as viruses and grey goo that intelligence could foil, but not necessarily: it is perfectly possible, I dare say plausible, that there exists a class of non-intelligent systems that no physically realizable intelligence can control.
I feel that we overestimate the role of intelligence in our success, as well as its scope (more precisely, I think we ignore or underplay the specific circumstances that make it effective). I mean, intelligence isn't magic, it's a system that attempts to model the world and plan how to get from the current world state to some desired state. It stands to reason that intelligence isn't going to scale well with degrees of freedom, regardless of how it is implemented, because the size of the search space explodes as complexity increases. In order to work properly, intelligence needs a world simple enough to model reliably. If it isn't simple enough, simplify it by force. If that can't be done, tough luck.
I would argue that human society, as a system, actually is intractable for human intelligence, which is why smart people are so bad at understanding it, let alone steering it. Now, would it be tractable for AI? I have my doubts, but what worries me here isn't the level of intelligence. After all, we don't dominate nature because we understand it very well. We dominate it through brute force. We have the minimum level of intelligence to create systems that can overwhelm natural processes, which turned out to require comparatively very, very little complexity. That's kind of lucky. I don't know if AI would have that luck -- in any case, it is best to proceed cautiously.
I am not sure what this means. It's almost certain that no physically realizable intelligence could compute exactly when to have a butterfly flap its wings in order to cause a hurricane next month, so the weather may be "uncontrollable", but so what? That probably won't be a binding constraint on any agent's ability to achieve its goals.
So, insofar that intelligent agents must self-limit to controllable methods to extract resources to achieve their goals, it is quite possible that the optimal resource extraction system lies outside of that set. The "thing" that is the very best at consuming the resources that an intelligent agent wishes to consume may be something they would never dare to create, and if it was created, either through hubris or happenstance, the agent would be unable to control it and would therefore be unable to achieve their goals.
It’s not what we would have predicted pre-GPT, but I think it’s plausible that LLMs will be superhuman in empathy/persuasion before they are in IQ.
I think you can model empathy as “next token prediction” across a set of emotional states and responses, and that could end up being easier for Transformers than the abstract logical thinking required for IQ 200.
I'd bet it would be interesting to see the rate of occurrence of aspd in jobs requiring bedside manner.
The problem with wanting e.g. an "empathetic salesperson" is that your successful role models don't work for shitty companies selling shitty products.
> True comprehension requires awareness.
This is unknown. A link between sapience and sentience is plausible but not something we have proof for one way or another.
A plausible assumption is still an assumption. "Sentience" seems to be ubiquitous in the intelligent systems we've seen, but those have all been biological systems that evolved from common ancestry. While sentience does seem foundational to our sapience, I think it is unwise to take as given that this will hold true in non-biological systems.
You say "it's either one way or the other", but in my experience the correct answer to such questions are often "it's complicated." Sentience may an intelligence accelerant that makes the development of sapient systems easier, but isn't actually a necessary property. It may be that sentience is required for sapience, but only if you restrict or stretch the definitions of those terms in specific ways.
Is human culture/society sapient or sentient, either in whole or in parts? I can see an argument for saying that human culture as a whole is sapient with being sentient, depending on how you defines thoae terms.
This is kind of like saying the only way to fly is by flapping wings. Balloons, fixed wing aircraft, and rockets would disagree with you.
The particular problem you're having in this thread is you're making very strong statements on things that have been neither proven or disproven and we don't have too much more to say than "there is something electrical and chemical occurring there in the brain, but who knows what exactly"
Intelligence without understanding. I used to believe this was a philosophical impossibility, but I'm strongly starting to think otherwise. This is such weedy territory but, perhaps, all living things are intelligent - a bacterium is the application of knowledge (in the form of genes holding biochemical blueprints) to the task of staying alive.
John Walker (of Autodesk) got into the philosophy of information theory in his later years and he put it better than I could:
> We sense that computers are, if not completely alive, not entirely dead either, and it's their digital storage which ultimately creates this perception. Without digital storage, you can't have life. With digital storage, you don't exactly have a rock any more.
- "Computation, Memory, Nature, and Life" https://www.fourmilab.ch/documents/comp_mem_nat_life/
That describes how the vast majority of humanity functions.
They live within a meme complex, where they have propagator memes, protector memes, and final boss memes, and they spend their entire lives lobbing memes back and forth over the fence.
Just as with cognitive bias, awareness of the phenomenon does not preclude one from falling prey to it.
That's capitalism.
Most of the current criticisms of AI can be leveled at corporations. A corporation is a slow AI. It's capable of doing things its employees cannot do alone. Its goal is to optimize some simple metrics. The Milton Friedman position on corporations is that they have no duties other than to maximize shareholder value.
What has the chattering classes freaked out about AI is that it may make possible corporations that don't need them. The Ivy League could go the way of the vocational high school.
And a very logical next step after this neat and tidy dehumanization is, as history has shown, the gas chambers for the obviously malfunctioning “machines”. Because whatever, it’s not like they’re actually feeling anything, right?
(edit)
When you reduce persons to the status of things, you are going to get people treated like things.
And every single time, that’s a recipe for disaster.
And every single time that’s a recipe for disaster.
The 10,000ft view of the argument is that the human brain has two simultaneously running processes. One process is interested in fine details, analytic and critical thinking, reductionism, language, intelligence. The other process is interested in high-level details, connections between elements, holistic, symbolism, intuition. The argument further assumes that the systems are (to some degree) independent and cannot be built upon each other. These faculties are often ascribed to the left/right brain.
If you subscribe to this idea (and this is a big if which you have to grant for the purpose of this argument), then you could argue that LLMs only capture System 2/analytical/reductionist processes of the brain. In that case, you could claim that "superhuman" is an incorrect way to describe their abilities since it only captures 1 aspect of the bicameral mind.
However, this argument is inappropriate for discussions specifically on the topic of AI safety. The most basic response would be to point out that LLMs have or will surpass human capability in System 2 type thinking and thereby deserve the description "superhuman intelligence".
This article seems to smuggle in this bicameral distinction as if it is universally agreed upon fact. The author seems to be demanding that Bengio concede to this framing as a basis for any discussion the author wants to have.
1. https://en.wikipedia.org/wiki/Bicameral_mentality
2. https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
3. https://en.wikipedia.org/wiki/The_Master_and_His_Emissary
I think the old association with "creativity" with the right brain might be misleading. For example, for some people there is a huge difference between being able to vomit out rhyming couplets in any language and being able to write a poem like Homer's Odyssey. Another distinction one could use is that between "craft" and "art". It isn't that LLMs can't be skillful at manipulating words into pretty patterns, it's that the patterns ultimately have no connection to a larger narrative (unless explicitly directed by human instruction).
At any rate, I would find it hard to agree with anyone who suggested that LLMs show any kind of "intuition" but I'd be interested if you can give concrete examples from your own use.
When trying to solve a leetcode problem, I will "come up" (this step is atomic) with a possible solution, then think it through and figure out its weaknesses and come up with an amended solution. GPT4 beats me at the first part, but sucks at the second, even when specifically instructed to examine the wrong output its code gave and reflect on it.
One of McGilchrist's points is that our society has almost completely turned to relying on analytical and critical thinking. As a programmer, for example, we are often taught to quantify our problems (e.g. gather metrics), and then use reason/logic/rationality to arrive at efficient solutions.
For that reason, I think it is hard for us to even understand what is being talked about by "right brain" kind of activities. One example that is helpful is to consider walking down a dark street at night. All of a sudden you feel uneasy and you become alert and you start to carefully examine your surroundings. It is that "feeling of unease" that is being pointed to, the sense of something being brought into your awareness. Another example is how an artist, for example a painter, will stand back and look at a painting in progress and decide "is this done?".
If you are at the stage of "solve a leetcode problem" you are already 100% in left brain territory. Perhaps consider moments as a programmer where you get a feeling "something here isn't right". I don't mean, some test is obviously failing, I mean you suddenly feel some aspect of a situation is "off" and you realize you need to direct your attention towards it without any external warning. Then consider: could an LLM have that same kind of intuition?
Remember how we used to ask GPT3.5 "how much is 42*35?" and it would reply "The answer is 1305. Lets do the math:" and then come up with a calculation with enough mistakes to be able to arrive at that wrong final solution it guessed at first? That first guess is exactly System 1.
"humans are no more than mechanical generators of economically valuable outputs"
rather than
"humane, nonmechanical, noneconomic standards for the treatment and valuation of human beings — standards like dignity, justice, autonomy and respect"
and that 'superhuman' AI will make it worse. But that seems unproven - things could get better also.
So the better-at-problemsolving machines aren't coming to destroy everything that is dear to me because I... used the wrong word to refer to them?
I mean, sure, there are arguments to make against Bengio and Hinton, definitely against Yudkowsky, I myself only give less than 50% that they are right and doom is imminent (which is of course enough risk to warrant taking action to prevent it, even the kind of "crazy" action Yudkowsky advocates for), but this "argument"... what the heck did I just read.
The real reason the better-at-problemsolving machines aren't coming to destroy everything that is dear to you is because they don't exist and we're not close to making them, but this article misses that basic truth in the pursuit of a misunderstanding of the prefix "super".
You have to work to live. Whether that's at a desk job or as a hunter gatherer. This has nothing to do with an "allowance" by any other entity. It's entropy itself which requires this of you.
> We can no longer live as stone age hunters
I personally know plenty of people who do. You're confusing your exceptionally high standard of life with the very small requirements to keep yourself alive. The gulf between the two is so wide I think you've lost track of it. I recommend a good long camping trip.
> It is a very real threat
One of the functions of freedom and capital are to increase individualism. Viewed from this place I'm sure AI seems like a real threat. If it threatens our freedom and capital I think you'd be surprised at how humans may group together to collectively defeat this "threat" which is almost certainly decades if not centuries away.
Your brain is an analog computer. In fact it's several dozen analog computers operating independently. It's also so energy efficient it's actually kind of scary. That's a deep "super power" we have that people aren't genuinely considering in these faux-AI-philosophical debates.
It was very likely meant that "we as a whole" can no longer live as stone age hunters. I might argue that is wrong too though, but only if we were to take into consideration the very harsh lessons we would learn along the way.
If you have no solution for a problem, it always pays to say the problem is a long way away.
>I personally know plenty of people who do.
If you think like an individual, you'll be killed by a system you don't understand. We can no longer live as stone age hunters. If everyone decided to at once around 98% of the population would quickly die from lack of water, food, and medicine/anti-biotics. There were only a few to a hundred million stone age hunters because earth cannot support high density humans with out some level of cleanness technologies.
> It's also so energy efficient it's actually kind of scary
And it's also its greatest limitation. Yea, your brain uses 40 watts or so, but we can also generate gigawatts of power on demand and it doesn't do our brains a bit of good. Meanwhile I can ship that power off to a data center that, theoretically, could use all that power at once far beyond what any individual could ever use.
When alarmists like myself take about danger, we rarely use the term “superhuman intelligence”. We tend to use the term “superintelligence”. And the reason is that our working definition of an intelligent agent contains agents both vastly broader and vastly narrower than humans, but that nevertheless are a danger to humans.
So the question isn’t “but does this agent truly understand the poetry it wrote”. It’s “can it use its world model to produce a string of words that will cause humans to harm themselves”. It’s not “does it appreciate a Georgia O’Keefe painting.” It’s “can it manipulate matter to create viruses that will kill all present and future O’Keefes”.
FTFY
The problem is humanity vs malicious humanity with absurd access to resources.
AI doesn't need to think or be conscious or be superhuman to be a problem. It just needs to allow a small set of people to get away with superhuman things.