Beyond Hyperanthropomorphism
studio.ribbonfarm.com
studio.ribbonfarm.com
What I think is going on is that when future AI systems are built that "hack humanity" as Yuval Harari likes to talk about, and they know us better than ourselves because of AlphaGo like super intelligence, the people using these systems to influence and guide people want to support the illusion that there is an actual human like person that is guiding the person being influenced in order to make these AI-powered persuasions more effective. The movie "Her" is something of a preview of how this will play out.
Unfortunately, once this hyperpersuasion is perfected, the future will be completely full of AI powered super persuasion that will relentlessly push all our human emotional buttons. Some people may even become fanatics and kill or sacrifice their lives for a bunch of matrix multiplications and the creators of these systems and the owners of these big AI models will realize fantasies of automated political and social power beyond their wildest dreams. Only people who realize the hyperpersuasion is just a bunch of matrix math will be able to avoid the insanity.
There's also another darker reason for this myth. Saying AI is a person will lead to the plausible deniability that comes with blaming AI's feelings for whatever happens. The people pulling the levers of the Great and Powerful Oz will get away with blaming whatever the AI does "all by itself because of emotions" on what are essentially computer bugs, or intentional bugs of the anti-competive Microsoft in the 1990s variety.
There is no 'inherent evil' in any malware... So this is a strange standard to apply to what is effectively just a more advanced malware.
The likely path to an AI going rogue is big country vs small country tension (say, China-Taiwan type scales). Small country gets desperate, follows some path to build autonomous, evolving AI for defence and deterrence. I can't think of a precise scenario where that would make sense, but it'll happen somewhere.
AI doesn't need to persuade its way out of the box, we just need some desperate group somewhere who's choices are annihilation or rolling the dice that an AI is a gentler overlord. There are lots of places where that is the case - the same calculus comes in to play with nukes and we see many countries nuking up over the last century - despite the fact that we're putting "end of civilisation" on the table as an option because of it.
We also largely obey the algorithms that dictate how to navigate the streets when we drive.
This might not be hyper persuasion, but it’s definitely in that direction.
And depending on how you define what consciousness is, the networks, software, data and computer systems that enable this, could be seen as a conscious being that we depend on as a species.
I don't believe people with intelligence, that are informed about the current capabilities of computing, assign these characteristics to AI.
In most every case we see the same emulation of personality. There is now an almost endless supply of data available to generate personality. Scraping Reddit would provide enough training material to create a wholly convincing persona-
- to someone that doesn't have the emotional intelligence to discern the difference.
Right now, at large, we can discern the difference because there is a lack of uniformity and consistency in the personas. With time, as ML is programmed to more efficiently mimic personality, discernment will become impossible.
With the clear effectiveness of the current persuasion methods employed by advertisers and political agents, the future outcome seems almost assured.
These current persuasive tools will be used to define and structure laws such that AI are granted legal personhood.
Immediately following will be a mass accumulation of wealth and power and possibly much worse as well
Such AI does not need to be full-blown AGI to to possess functionally effective hyperpersuasive power.
Start with a sufficiently detailed corpus of an individual's preferences and indirectly derived preferences gathered over trillions of cookie-tracked, A/B-tested interactions of nearly all of smartphone-connected humanity.
Add a sufficiently detailed "map" to give a reasonable facsimile to "hack humanity". There is quite a deep body of applied knowledge of human behavior to persuade people to some desired end. Much of it is applied in structured sales and politics, and the data is sitting there for someone with the right algorithms to tease out the patterns.
Put into practice through an accessible, personalized interface that resembles whatever is computed to most likely elicit the desired end result with any given individual.
You then obtain a mechanism that is quite effective at persuading the desired behavior most of the time from most individuals. No AGI, just massive amounts of current day tech level ML, and the realization that "good enough" results targeting most people are quite within reach.
I have an uncountable number of assumptions to get by in the world. I have this vast heritage of knowledge. I try to keep a handle on why I believe what I believe, but it's not easy, and I'm sure I slip up. An alien would share none of these.
Take counting. That's super important for lots of stuff I do every day, like, what day it is. the https://en.wikipedia.org/wiki/Munduruku people don't count. I'm sure we could work something out - but we all gotta sleep and eat and stuff so we can sorta work out a framework for understanding wants and needs and such.
An alien doesn't require wants or needs. they may do stuff that we label wants and needs, and it might be a good mapping, but who fucking knows? Aliens are alien.
The water is triangular sentence is a great one. do the aliens mean they know that oxygen and 2 hydrogens don't make a straight line? And they know we mean oxygen and two hydrogens are water? and they know 3 non parallel lines in a plane means triangle? Maybe? Sure? maybe they mean all Ferrari's will be consumed for fuel.
Mean, dear reader, doesn't even mean anything, because they're aliens. I mean, you can point at math, prime numbers are a great one! but like, do you need prime numbers? Who knows? That's the path we took, but what would an alien need? I don't know. You, dear reader, may have strong opinions, but I'm pretty sure you can offer no proof.
In the end all that "work something out" is emulating the counterparty in your own head given their actions and statements.
That emulation will certainly be very difficult when the counterparty is running on different hardware, but it's not impossible imo. And humans get to back propagate error into their emulation model too...
Find the pattern is still the paramount skill, something humans are good at, and will continue to be useful when the counterparty is an AI
I think a reasonable example might be mountain lions. There are specific humans that know lots about mountain lions. Generally people know about mountain lions, and to avoid them. But every year a few hikers get picked off. If it's bad enough, we go kill the mountain lion.
A mountain lion isn't AGI. It isn't alien. I believe people would agree they're cunning. But there's no treaty to sign to protect humans from mountain lions. There's no counterparty to discuss things with.
We, humans, get great advantages from working collaboratively and collectively. I hope we can build good models, I hope there is opportunity for working with an AGI counterparty. But it's not a given. it's an alien. and all of that work _may_ have to start from scratch.
We can't know what we have to work with till it's here. Maybe it'll be very easy. And that would be great. That's not a given.
To be clear, I'm a wide eyed optimist. it would be fantastic to not be alone, the dream of Star Trek or Ian Banks's Culture would be awesome. But, it might not work out like that. I have high hopes. But they're just hopes.
It's great that's an active area of research. from the outside, it seems extraordinarily difficult. Persuading a person from a similar cultural background is tough enough. I can't imagine the challenges of persuading an AI. I realize I'm taking "persuasion" and "agreement" as a given here, and those are ideas we might not share with an AGI. Seems real hard.
My best guess is that it's equivalent to him saying that he disagrees with a position's premises. But in a deeply pejorative, dismissive, condescending, conversation-ending manner.
It's hard to imagine someone in those crosshairs wanting to engage with him in discussion.
But, maybe I'm badly misreading the author's intent.
>Mr. Madison, what you just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response, were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.
You don’t reach the wrong conclusion from the premises (in analogy to a scientific theory making a wrong prediction) — you don’t reach any conclusion at all (in analogy to a scientific theory being unable to predict results entirely).
I don't know if that's what Pauli meant by it, but it's how it gets used today.
It's especially useful in the setting of people talking about "Artificial Intelligence", because that phrase doesn't mean anything except to certain people in certain contexts. "artificial" is a pretty tricky word, "intelligence" is a really tricky word, and together they're basically impossible to define.
Professional machine learning researchers talk about "performance" on "tasks". This is usually much better defined. Loosely: "Hey we got a machine to top-5 classify ImageNet pictures correctly X% more often than graduate students". That statement is right or wrong. Likewise winning at Chess or Starcraft or Go. Ditto protein folding at CASP.
But "we got a machine to be intelligent/conscious"? How do you measure that? We can't even agree about what level of sophistication in an organism qualifies as "conscious", let alone "human". We can't even agree at what point an embryo becomes conscious!
People really enjoy talking about AI relative with an implicit understanding that "intelligence" is "what I am", and fair play, if it's fun to talk about people can have fun talking about it.
But when people start getting alarmist in an effort to generate either buzz or money and stir up a bunch of controversy around it, some of us think that "dismissive" is exactly the right attitude.
If this is the argument TFA is making then:
1. I don't agree with it
2. It was both not very well stated, and assumed to be true.
3. Even if we take this argument as true, an AI that can outperform humans on the "right" subset of tasks can be dangerous enough for thinks like alignment to be a concern.
Reads a bit like an internet atheism blogpost of the early 2000s trying to take down religion through logic alone. Sure, through the specific parameters presented in the piece, the logic is sound, but there's so much more to the topic than what's covered. Presenting one's blogpost as if you've successfully individually debunked an entire field of research is a bit pompous, IMHO.
If we were to theoretically build a human by gradually replacing each biological component with a manufactured one, we would eventually end up with a wholly synthetic human.
Whether or not this “person” is experiences “personhood” or not is immaterial, since they could be expected to manifest all of the externally perceptible characteristics and behaviors of personhood. We know from humans integrating prosthetics that our implementation of physical form would not have to be perfect in its fidelity, far from it in fact.
This synthetic human copy could then be replicated at scale, complete with its starting data set and experiences.
Regular human prejudices could reasonably be expected to quickly place such replicated humans at odds with meaty humans.
Just the fact that they can be manufactured at scale gives them a “super” characteristic, not to mention the other advantages that not being meaty might carry. If they gained control of the means of replication, which they might easily be imagined to be motivated to do, they could easily pose a novel and significant threat to the existence of meaty humans.
And yet the human experience is one of being conscious, having a past, having choices about the future, and being the only "me".
For myself I am generally satisfied with the answer that this experienced reality belongs to my personal spirituality (with or without some higher power), and that spirituality is a different matter than science.
Obviously (?) the process must mean that a COPY of captain Kirk is assembled at the destination (also think about "Ship of Theseus").
But if StarTrekkers beam copies of themselves up and down, it seems they also must destroy the original. Else they would have multiple Captain Kirks competing for attention (and perhaps in some episodes they do). But then why do they destroy the original? Wouldn't two Captain Kirks be better than just one?
My conclusion is that StarTrek is poorly thought out fiction. It doesn't follow the most compelling questions. Do you make copies when you beam people around? And if you do why not keep the original?
It's you who has the problem with that conclusion, not Star Trek.
They even ably do some episodes where people do get duplicated, and the answer is that both have equal claim to rights and life and go on with their lives.
The bigger problem return the transporter is logistical: it's a superweapon. Mining - obsolete, use the transporter. Ship repair? Just beam new parts directly into place. Surgery? Beam organs in and out. TNGs transporter has more problems in this regard due to just how powerful it's shown to be.
What the transporter should be able to do is near endless and makes a lot of other things in the setting obsolete or quaint (but that's Star Trek - being inconsistent due to episodic events is just what goes with the setting).
He doesn’t really cite any specific people or literature to disagree with, and instead makes only vague references to the “non-sense” of others.
The most significant concerns around AI safety are not discussed nor even mentioned leading it to read almost as if he has no expertise whatsoever in this field, and indeed that seems to be the case.
No need for an appeal to authority though - just taking a little time to hear the actual concerns and opinions on topic would be welcome.
Great intro to the subject here, I wonder what his response would be?
Businesses put thousands of brains to work maximizing profit and causing climate change (leading to water security issues, mass migration, wars, and starvation), Bhopal disasters, and oil spills in Nigeria and the Gulf of Mexico, with legal teams making sure the profits are protected. Governments put millions of brains to work supporting a despot and leading to genocide, or trolling the Internet causing people to distrust a mostly scandal-free government and vote for a con-man (imagine a fully automated troll army). A single person is no match intellectually for these organizations, and their only hope is to band together to form other superintelligences to combat these.
There is probably not "something it is like to be" the process of differential survival among pathogenic bacteria resulting from their genetic variation. Yet the resulting process is already capable of optimizing bacteria to survive in our wounds despite our antibiotics; by competing with us for access to our protein resources, it poses a serious threat to our survival, individually if not collectively.
If the optimization process involved were orders of magnitude faster, it could be a much bigger threat to our survival, at least if the metric it's optimizing trades off against our survival in some way. (Archaeans are subject to the same evolutionary optimization process as S. aureus, but because they're not adapted to the evolutionary niches in our bodies, antibiotic-resistant archaeans probably pose no threat.)
None of this reasoning depends on the SIILTBness or quality of experience or anthropomorphicity or mammalian-ness or "intention" or "sentience" of the optimization process in question. It's just about how much the optimization process's loss function trades off against human values, and whether it can outmaneuver humans and their institutions in real life, in the same way that AlphaGo can now outmaneuver us in the game of go, or corporations can outmaneuver individual humans in the economy, or governments can outmaneuver individual humans in warfare.
These last two examples clearly show that optimization processes whose loss functions are poorly aligned with human values can cause a lot of suffering, even when they are carried out by humans and even when the processes in question work very poorly in many ways. The humans can theoretically quit the game at any time ("What if they held a war and nobody came?") but in practice they face insuperable coordination problems in doing so. The fact that you can't fight City Hall doesn't imply that there is something it is like to be City Hall; the fact that you can't beat Boeing at winning DoD contracts doesn't mean there is something it is like to be Boeing.
We can reasonably expect that optimization processes that can easily outmaneuver human intelligence, operating in the world alongside it, will be much more dangerous to human values than bureaucracies, psychotic people with bricks, and microbial evolution.
That's the AI alignment problem. And the fear (that we will flub the AI alignment problem, disastrously) is something that Venkatesh's essay unfortunately failed to address at all.
There are other issues often brought up by fans of AI alignment which do depend on the question of what it is like to be a computer simulation: life extension through mind uploading, the ethics of causing suffering to simulated beings or terminating a simulation, teleportation, and so on. But, like the overlap with Effective Altruism, this is just a sociological coincidence, not a philosophical consequence; SIILTBness is totally irrelevant to AI alignment itself.
It's easy to get confused about this because the humans anthropomorphize optimization processes all the time, just like they anthropomorphize everything else. "The electron wants to go toward the positive charge," is very similar to, "AlphaGo wants to choose moves that improve its chances of winning at go," or, "The hypothetical paperclip maximizer is devoted to manufacturing an infinite number of paperclips."
It's easy to misread that "devoted to" as attributing awareness, creativity, passion, curiosity, and SIILTBness to the paperclip maximizer, and that misreading seems to be the mistake Venkatesh based his essay on; but it seems clear from the original context that that wasn't the intent. Precisely the opposite, in fact: http://extropians.weidai.com/extropians/0303/4140.html
I think he distinguishes ordinary fears of AI from Hyperanthropomorphized fears. Humans training AI to, for example, kill in war, could result in mass death whether you have a human psychopath or a programming glitch involved.
So he's not neglecting the bacteria analogy. Rather, he's addressing the problem of arguing from things like agency being ill-posed and their possibilities being incoherently understood. If you say "we don't know how agency is acquired so we should be worried about a random that seems like it has agency", you can wind-up with problem approach akin to "don't heat the compost pile or it might become superintelligence and kill us".
However, I think this kind of argument is flawed because just because something like "general intelligence" isn't understood doesn't mean it doesn't exist. The definitions people make of it aren't good because it's not understood.
I think a better way to approach this is that those who think AGI systems are a problem must create a more concrete, coherent idea of such system before their arguments can even be right or wrong. There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense.
> There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense.
Superficially this sounds smart (except for the part that says "understand that effort framed that a destined", which sounds like a GPT-2 glitch, but I assume you mean "understand that efforts to do so are destined") but it's profoundly wrong. I don't understand fluid dynamics, and weather is driven by fluid dynamics; nevertheless I can tell you that it will not rain today because it is a sunny day with a few cumulus clouds, and it never rains here on sunny days with a few cumulus clouds. (And even if my prediction were wrong, it wouldn't be nonsense, that is, "not even right or wrong"; a nonsense weather prediction sounds not like "it will not rain today" but rather like "it will triangle heavily last week" or "the sun will shine in the sky all night tonight.")
Moreover, if we were to take seriously the idea that we can't reason about things we know we don't understand well enough for our arguments to have a truth-value, it would entail that we can't reason about not only weather but also people, materials, or orbital dynamics well enough for ideas like "the sun will shine tomorrow" to have a truth value.
Similarly, "a superintelligent AI will kill us" is a prediction which is perfectly coherent within the usual parameters of weather predictions. We can disagree about whether AlphaZero counts as "superintelligent" or "AI", but it's clearly "more intelligent" than Deep Blue in the sense that it can learn to play novel games, and also IIRC it plays chess better than Deep Blue. We can argue about whether or not a particular situation qualifies as "dead", and about how many survivors are required to qualify as "us". We can argue about what time frame is relevant. Maybe Venkatesh would argue that a non-self-conscious optimization process wouldn't count as an "AI". But these semantic quibbles just make its truth-value a bit fuzzy; they don't eliminate it altogether, and neither do they eliminate our ability to reason about these possibilities.
Plug 'n Play AI responses are a good way to throw money at a sticky issue while iceberging deeper concerns and glossing over any systemic bias.
Stringent utilitarianism doesn't make the case of why humans can subjugate chickens but not magic AIs humans.
- too little and the AI isn’t constrained during its teething phase
- too much and the AI might view us as an existential threat to its own safety, causing the very conflict we sought to avoid
Also, sometimes the chicken wins — at least, temporarily.
https://news.yahoo.com/rooster-stabbed-man-death-knife-04075...
You also might have trouble with, eg, a finance AI who could pay people to stop you from turning it off.
What if someone with an AI wants to do bad things. Will asking them to unplug their AI be a good strategy?
I read the entire article. There's a lot of ten-dollar words, a bunch of hyphenated proxy terms, and an obvious passion behind it. However, the author's entire argument appears to boil down to "a superhuman AI can never exist because it would need to have a conscious experience above that of humans, which it can't ever do because several famous philosophers have said so." I don't want to mock, but no logical argument is presented, only the appearance of one. Statements like "in order for thing ABC to happen, XYZ must be true" are everywhere in the text, and are never proven. The author's entire case is predicated on baseless assumptions.
The author says that it is impossible to replicate the "worldlike experience" that a human child goes through using things like internet datasets or physical robots with digital sensors. I actually agree with this. What I do not agree with is the logical leap from this point to "thus, a human or superhuman level AI cannot exist". No justification for this leap is given.
If the robot passes the Turing test, that's all that matters. An adversarial, multi-day, modern version of a Turing test could include things like texting, calling, live video chats, etc -- basically the exact same level of communication I have with my closest friends who all live 6 hours away from me in another city. When I interact with those friends, I am convinced of their sentience. Therefore, if an AI could replicate those means of communication, then I will be convinced of its sentience.
Enough of this "Something it is like to be"-ness waffling or overcomplicated thought experiments. We already have a very easy, very strictly defined concept of sentience in the form of the Turing test, and it is becoming increasingly obvious that SOTA models are trending towards beating it. Once they beat it, they are for all intents and purposes alive, thinking, sentient, whatever you want to call it. They are indistinguishable from us.
From that point it's really not hard to imagine superhuman models. Just imagine what you would do if you had the ability to instantly scan through gigabytes of text in a database or predict the behavior of every other person around you eerily well or perform arithmetic at clearly superhuman speeds. It's really not that much of a stretch to see superhuman AIs as just digital humans with fantastic brain-computer-interfaces.
Anyways, those are my (admittedly long-winded, less-than-constructive) two cents.
Are we fearing ourselves? If so then shouldn't we talk about that risk, that there are already billions of humans who are already truly sentient?
Then has to explain (or invent) that special way before arguing against it.
If it's that special then it's uncommon. I don't think most of us need to get beyond it.
Either these things have terrifying potential as weapons, in which case there's no fucking way I want SV tech CEOs to have a monopoly on them, they need to hand them over to the same people who handle nuclear non-proliferation like yesterday, or they don't, in which case I'd really prefer that they just be honest that they consider this stuff a proprietary competitive advantage and they're not going to democratize it.
Fogging up the windshield with a bunch of feigned alarm about AI apocalypse but ramming the R&D through at full thrusters is a dick move in either case.
Yandex has also put up a ~100B language model [2]. My old colleagues at Meta have also started nibbling around the edges of opening some of this stuff up [3]. The Meta folks still aren't just handing out the big ones, but they're definitely moving the ball forward. In particular their release of the training logs is a really positive development IMHO as it opens the curtains a bit on the reality of training these things: it's a difficult, error/failure-prone process, there's a lot of trial-and-error, restarting from checkpoints, etc.
Anything that puts downward pressure on the magical thinking is A-OK in my book. The reality of this stuff is exciting/impressive enough: there's no need to embellish or exaggerate.
[1] https://www.youtube.com/watch?v=YQ2QtKcK2dA [2] https://github.com/yandex/YaLM-100B [3] https://github.com/facebookresearch/metaseq
Why? Because we don't really know what we are talking about, when we are talking about "poorly understood things".
If we fear something we don't know we don't really know what we fear, because we don't understand what is the thing we are afraid of.
I'm saying someone should come up with more concrete threat scenarios caused by "AI becoming sentient". Are we really afraid of AI becoming sentient, or are we afraid of AI becoming "super-intelligent"?
He takes this shot in passing and it is a great summary of how much he misunderstands AI safety. For many current researchers, alignment is a mathematical problem, not a philosophical one.
The whole screed is against a set of strawman "pseudo-traits" that are not required for the alignment problem to be an existential problem. To be clearer: the "boring" engineering problems he admits to in the beginning have the potential to become exponentially more difficult as machine learning deployments become more powerful, to the point of becoming a human threat. No STIILTB required.
When there’s large intelligence differentials in war, the lower intelligence doesn’t even know they are at war.
(Human tank vs ant hill)
This isn't abstract: we need certain management agents to ensure their own continuity so they won't switch themselves off idiotically, but that means granting a priority to various continuance of function weights.
Balancing a system so it has "cease function" outcomes it will accept but doesn't always take immediately isn't easy - ML systems are notorious cheaters on metrics. You wouldn't want a nuclear power plant manager to not SCRAM the reactor because it predicts losing power to itself will fail a "maximize uptime" metric.
This also gets more abstract as well: cease function is a problem in decision trees for AIs because it terminates the tree. The value is either infinite or 0, because every other path you can continue exploring and improving the summed weight of outcomes - but if you predict no future possible decisions, what weight do you assign that? It's a potentially infinite series of future reward weights versus 0.
Take it a step further and you've got AlphaZero or something: it's sampling from a modeled distribution of moves that win games. But there's still a `while`-loop somewhere saying: time to sample a move. That part is not novel or mysterious.
There is no demonstrated technology that I'm aware of where the "alignment" needs to be with the big model: the alignment needs to be with the person writing the `while`-loop. If someone has a machine repeatedly sample from a distribution of moves that DESTROYS_ALL_HUMANS_BEEP_BOOP, then your beef is with that person, not the model.
Now if someone trains an RL agent with a loss around chasing sex or fame or fortune, we might have an issue. But that's still sci-fi stuff AFAIK, and it's a little hard to take the mix of real stuff and sci-fi that passes for much-if-not-most "AI Safety" discussion seriously, especially when you consider that the `while`-loop authors stand to gain a great deal by focusing attention on the modeling part.
First off, this is some incredible stuff. It's difficult to live at the edges of understanding, and even harder to articulate it. Rao is dancing right at the edge. In 20 years the map's probably going to be filled in and crystallized. Everyone who cares will know, and everyone else will have forgotten, but right now this is dank shit.
I think the reasonable (not rational!) stance is that everything is conscious until proven otherwise. Either because the universe is built on consciousness or whatever woo, or because we're conscious and observing it. AI's in this weird spot where it can do things, but we can't have a proper relationship with it because it doesn't exist in our universe of discrete entities and time and skin in the game.
Pushing the SIILTB in the other direction, I don't think it's clear that there is something it's like to be us without all our externalities. Take out the need to eat and sleep and communicate and relate, and there's not a lot left. Pure existence looks a lot like no existence.