Among the A.I. doomsayers
newyorker.com
newyorker.com
(Grants do not require repayment.)
What I find is interesting, is that the negative news regarding AI safety is adding to the hype as well since it seems to capture a lot of attention.
The people that try the models and find no value will obviously not continue to use something they get no value from.
If you discount anyone that doesn’t use it on a continual basis, then of course all the detractors aren’t as heavy of users as those that do use it regularly.
I think a better argument would be the size of the set of regular users vs. some grounded guesstimate for how many have tried it.
I would say for AI deniers it's the losers fallacy. You guys haven't taken the time to thoroughly examine chatGPT4. You just stop at the first hallucination. It's a paradigm shifting AI.
Detractors like you just rely on common tropes like oh "stop anthropomorphizing it" and stuff like that. People who see this stuff as game changing recognize these biases and we still say it's game changing. Nobody is stupid enough to think this thing is even remotely human.
I asked chatGPT to program me a complex GUI in Python about selecting multiple time segments out of an overall interval using sliders. It gave me working python code.
You have realize that this thing has no eyes and it was able to program a visual interface as if it did. I'm not anthropomorphizing anything when I say this, but on some level chatGPT "understands" what you are giving it as a query.
Ah, been a while since someone pigeon-holed me like that. Usually it's the opposite pigeon hole I get stuffed into.
Ironically, that kind of rapid but inaccurate misreading is one of the things that makes LLMs somewhat easier to deal with than a human: with an LLM, I can backtrack, edit my past words, and get a different answer. One cannot generally do that with a human, as it's a rare trait indeed for someone to be able to hear "you misunderstood, not x, y" and actually take that on board. Even on this site, I've been in threads of:
--
>>>>>>> x
>>>>>> y is dumb, you're dumb
>>>>> sorry you misunderstood, I'm not saying y, I'm saying x
>>>> I just told you y is dumb, you're dumb
>>> no, you're still not getting me, I'm not saying y, I'm saying x
>> stfu noob, I'm a professional: y is dumb, you're dumb
> I'd agree y is dumb, but I'm saying x
stop moving the goalposts!
--
(In this case, x was "it seems implausible that the Saudi Public Investment Fund would really last forever, forever is a long time and there's a always going to be a temptation for the people in charge to extract short term value to the detriment of long term potential", and y was (I think from context but I couldn't read their mind) "they're making bad decisions and will go under imminently".
> I asked chatGPT to program me a complex GUI in Python about selecting multiple time segments out of an overall interval using sliders. It gave me working python code.
Cute. I used it to write a web GUI for its own API, so I can have GPT-4 access as PAYG without a monthly subscription: https://github.com/BenWheatley/YetAnotherChatUI
Sounds like your problems are rather short-term and low-stakes, I've presented it several questions that it cannot answer and tells me so.
While there is certainly untapped potential with this models, what we've seeing is unquestionably market driven hype.
I know very, very few people who are working closely with these models who buy the hype.
I do use GTP* regularly for language learning, or making python templates to extend off of. Is it useful, kinda, not essential imo, but it is not worth as much as it is hyped.
You paint the picture as if the All or nothing folks are extreme when in reality the extreme is more likely then some perfectly fair and balanced equilibrium.
In nature things tend to overload or fizzle out or stay in equilibrium. Equilibrium, though possible, is the rarer outcome. Mind you it's not an impossible outcome but given the way entropy works, it is the rarer outcome.
Humanity itself is an example of this rare outcome. Usually molecules don't self assemble into replicating machines, they either freeze into inanimate rock or overload into fusion producing stars.
As for AI. I think it will either change the world, or amount to nothing. The former seems more likely. Some strange middle ground where the ai technology never improves to some point of a societal paradigm shift seems unlikely. chatGPT and sora only makes me ask what is the trend line predicting next?
I suspect even if a power ball lottery winner takes a philosophical position it would be taken far more seriously than if they had never won the lottery.
At some point you start to believe your own bullshit when all of society signals are telling you what a genius you are.
It is possible that "AI" might not be as world-changing as its proponents are claiming. However, if it is free and open-source, and non-commercial, it may still persist and remain useful, regardless of whether it becomes popular or not.
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. [0]
Signed by Demis Hassabis, Sam Altman, and Bill Gates among others
Replace "AI" in all the doomsaying with "the internet," and it will become clearer.
Dario Amodei (Anthropic CEO, builder of Claude 3 Opus): "My chance that something goes, you know, really catastrophically wrong on the scale of human civilization, might be between 10 - 25%"
Consider this scenario:
- Ask ChatGPT to produce some output meeting certain criteria
- ChatGPT responds, but the output does not meet the criteria in one or more obvious ways
- Point this out and ask for a corrected version
- ChatGPT says "I apologize, here's the corrected version:" and spits out a nearly identical answer with the same obvious mistake.
- Repeat steps 3 and 4 ad nauseam.
Whatever sort of intelligence it may have, I would categorize it as not "general" enough to fall under AGI. Being able to answer AP test questions cannot be a sufficient measure of generality if a model can do it with flying colors but still fail at far more basic tasks.
You can ask ChatGPT what it means to be asked to correct a mistake, and it will give you a perfectly thorough and eloquent answer. But it is often unable to apply this concept to its own behavior. If asked, it can explain back to you exactly what correction you want it to make. You can even make it pledge to correct the mistake in exactly the manner that it just described. And then it will completely fail to do it. It reminds me of that "repeat after me" meme from Friends [1].
This makes me lean in the direction of "stochastic parrot" when I think about what LLMs are. As impressive as it is, ChatGPT demonstrably lacks a sense of self. It talks as if it understands that it's an agent in control of its behavior, but then fails to control or even recognize its own actions.
[1] https://en.meming.world/images/en/c/c2/Friends-_Repeat_After...
ChatGPT was trained on hundreds of terabytes of information. I would hope that with all that effort it could answer AP test questions. That doesn't make it intelligent.
That it's able to spit out essays makes it something, intelligence is too loosely defined to describe it for everybody, so we need to develop new words and meanings and new language for what it does. It doesn't think, but it does dot products to tensors to get create its output.
It is interesting to consider AIs that way given that almost every AI will fail catastrophically when confronted with an area of expertise that is outside of their training corpus, although I guess Corvid tool use isn't exactly out of the animals' range of expertise.
All AI, even narrow ones, necessarily have the ability to interpret novel questions in some fashion — how novel and how well they interpret being the "G" and the "I" in AGI.
This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or wrong, every opinion is equally realistic as each other.
Like I could say that every AGI that becomes self aware will kill itself instantly because of boringness, and you cannot prove me wrong.
We just don’t know yet.
Feeling uncomfortable with this, will not change this sad reality.
I am indeed uncomfortable with someone saying that we can't worry about a gas leak until after someone strikes a match and notices an explosion or the lack thereof. If the sad reality is that we can't detect the gas leak any other way, I would argue that the precautionary principle suggests we ban the striking of matches until such sensors have been developed. But here we are in a room of people grinding up saltpetre with their mortars and pestles.
I have proven to you logically that they cannot have more knowledge then anyone else, but you continue to present their authority as knowledge.
Pls prove to me that they know more about the effects of AGI on humanity or that they have some magical knowledge we normal ppl are unable to comprehend.
If you cannot do this, then why are you bringing up these ppl?
I'm only asking for a standard that falls slightly short of "already having built Schrodinger's doomsday weapon and observed whether humanity dies as a result". You can set the bar wherever else you like, please!
Considering how rapidly the technology is developing, and considering how clearly a threat a superior intelligence will be to the human race, it is absolutely the time to have this discussion NOW
You've asserted it, not proven it.
Given their entire research field is "how to make the thing", it's a fair bet that they know more than me — even if what they know is, to paraphrase Thomas Edison, "1000 ways to not make an AGI", that's still more than me, and I follow some of the research and read some of the papers.
Correct, and this (your counterpart's unrealized because it is the normal, proper way to think error) is what worries me... I think once again (as with affordable vehicles powered by fossil fuels) humans may be once again walking blindly and confidently into a situation where we've over invested in certain competencies to the detriment of others, increasing the already dangerous imbalance that exists between power and wisdom.
I don't know to what extent OpenAI and their compatriots are actually trying to bring forth artificial life (or at least, consciousness) forward, versus how much they're just banking on how cool AI is as a term in order to funnel even more money to themselves chasing the pipe dream of building it, and at this point, I don't care. Even the products they have made do not hold a candle to the things they claim to be trying to make, but they're more than happy to talk them up like there's a chance. And I suppose there is a chance, but I really struggle to see ChatGPT turning into skynet.
My response to a very similar statement is here:
https://news.ycombinator.com/item?id=39612641
> That's just... not how it's going to be done.
I agree. I think there are some critical ingredients missing. Obviously the weights need to be able to update to new data in a semi-online fashion, for example. But I think there are less ingredients missing than there were a decade ago, by a much larger factor than I expected at the time, and a decade from now, the number of missing ingredients might be zero.
My uncertainty on this is not because I think GPT is more than it seems,but because it's unclear how much of the amazing cognitive capability that humans have is, under the hood, much simpler than it seems.
I really want humanity to exist even ten decades from now; I want my kids to have grandkids. So I care about this even if I don't expect AGI from a 2025 Q1 product launch. And I don't think it's too early to worry about it, just like I don't think 1980 was too early to worry about climate change.
This is a vivid bit of rhetoric to underscore this point, but if you think about it for any length of time it starts to fall apart really, really quickly. The Wright brothers and the dozens if not hundreds of inventors forgotten who came before them drew upon the physics of what they observed in heavier than air flight to create the winged shape we know today that reliably causes lift, and then set about constructing it. That's not what OpenAI is doing. We still do not have a very solid understanding of where our own intelligence emerges from, apart from having particularly large brains relative to our body's size. So, to borrow your metaphor, it is indeed like a bird saying that there's no evidence to say that heavier than air flight is possible, because the bird lives in a world without atmosphere upon which to glide.
But when you look at, say, Sora's videos... How certain are you that there's no path between that and the human visual cortex? How certain are you that they aren't solving at least some of the same problems in structurally analogous ways? Given that nature built a visual cortex (more than once!) by applying survival pressure and turning a crank, just how hard can it be? When we apply a bunch of optimization pressure and billions of dollars turning cranks and something eerily similar pops out, that tells me that there's just less magic to our own brain than we thought. Like when spent centuries wondering if our solar system is unique, and then we finally put Kepler up there and it turns out that planets are everywhere you look.
Sure we do: we're made of atoms.
To reject that this necessarily shows that AGI is possible, you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Some people do in fact believe this. I think that if we can't build AGI even with a perfect copy of a brain, that would instead be a surprising proof the existence of souls.
> This is not intelligence of any sort, period, paragraph.
Let's run with that. Just using your definition of intelligence: given AI can already beat us in Chess, in Go, in Mathematical Olympiad puzzles, in protein folding predictions, in poker, at real world stock market analysis, in the game of Diplomacy, … — does it matter that they're not what you call intelligent?
Do submarines swim?
> bring forth artificial life (or at least, consciousness) forward
Why does it matter if it's "life" or "consciousness"?
“Deep Blue was intelligent the way your programmable alarm clock is intelligent. Not that losing to a $10 million alarm clock made me feel any better.” ― Garry Kasparov
> To reject that this necessarily shows that AGI is possible...
Incorrect. For something to be possible in an absolute sense, it has to be possible in both the physical and the metaphysical realm, and a lack of knowledge of blockers in the metaphysical does not mean that there are none (in a comprehensive sense), it only means it appears that way, due to (at least) our cultural beliefs ("truths") and conditioning.
> ...you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Could demonstrating that intelligence rests upon a virtual reality generation machine (that cloaks itself from being realized as such) or something else not also work, at least maybe?
Plenty of people have interesting or valuable opinions and insights on AGI, programming LLMs doesn't have any impact into that though.
It'd be like expecting the developers of friendster to have a good idea of what the impact of social media was going to be in 2020.
AI isn't AGI.
The hour before the Kitty Hawk flight, the two most informed people in the world on the topic of heavier than air flight were a pair of bicycle retailers and manufacturers named Wilbur and Orville.
These people sure don't spot everything, but they're still way closer than the average human.
Actually, this is quite false. There were many people working on flight with massive credentials and massive funding. That the Wright brothers beat them is part of what makes the story of the Wright brothers so interesting.
That society didn't know this at the time is indeed relevant to AI development, and the analogy would be that the next big thing might come out of some minor research team that most people dismiss instead of the biggest and most well known groups… a bit like how Google's LaMDA was making headlines before ChatGPT surprised everyone.
It's certainly still possible today that some random individual has a crucial insight that gives them an edge over the big names, yet even those big names know far more than me.
Which is rather annoying, because I'd be interested in taking the opportunity of having been laid off to switch from iOS to AI. (If anyone reading this is hiring people in Berlin, contact details on my website. Plus side of hiring me: I'm relatively cheap in that field, and enthusiastic about the possibilities).
Truly no offense meant, as I was deeply into the EA movement myself, and still consider myself one (in the original "donate money effectively" sense), but the movement has now morphed into a death cult obsessed by things like:
* OMG we're all going to die any time now (repeated every year since circa 2018)
* What is your pDoom? What are your timelines? (aka: what is your totally made up number that makes you feel like you're doing something rational/scientific)
I'm deep in the weeds w/ LLMs, e.g. I probably finetune an average of 1 model a day, and working with bleeding edge models... and AI safety just sounds so silly. Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when gradient descent was invented.
It's sadly twisted how mentioning that -- the majority of leaders doing the cutting edge research on AGI think it has a significant chance that it kills humanity -- is considered being part of a "cult" movement.
Your analogy is the same as early Intel engineers completely unaware that those chips would bring on the ramifications of social media. "In the weeds" and yet unable to foresee the trajectory and wider impact. Same with the physics that led to nuclear weapons.
> Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when [nuclear fission] was invented.
Exactly! As they should be. (for both Intel engineers developing chips, and physicists developing nuclear research)
There were a billion more potential dangers from those technologies that never materialized, and never will.
I'm glad we didn't stop them in their track because a poll of 10 leaders in the field thought they were too dangerous and progress should stop. (note that no one is against regulating dangerous uses of AI, e.g. autonomous weapons, chemical warfare; the problem is regulating AI research and development in the first place)
> We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. - Sam Altman (https://blog.samaltman.com/the-merge)
I would say it is.
Beff Jezos (e/acc founder): "e/acc has no particular allegiance to the biological substrate for intelligence and life, in contrast to transhumanism
Parts of e/acc (e.g. Beff) consider ourselves post-humanists; in order to spread to the stars, the light of consciousness/intelligence will have to be transduced to non-biological substrates"
Beff Jezos / Verdon: "Personally, yes."
Yudkowsky is transhumanist in that he is hopeful people could voluntarily extend their biological self, but isn't advocating for the elimination of all biological selves in pursuit of other artificial intelligences.
Voluntary /= involuntary eradication
I'm a neuroscientist and, man alive, we're no where close to being able to merge with machines. Like, do you have any idea how many diseases we could eradicate if we could modify neurons like that? Like, for real, 'curing death' would be step 9 or 10 on that 1000 step journey.
I hope you see how terribly uninformed such a take is then.
Yeah, I only quoted it to show how that other statement up-thread can't be taken seriously.
The only way I can square it is if it's a lie, or they're reassuring themselves whatever destruction comes will be fine as long as "there will be peace and security in [their] lifetime."
That said, I don’t think that it’s necessarily disingenuous so much as it is myopic - to them of course AI is exciting, world-changing, and profitable, but they (willfully or not) fail to see the downsides or upsides for anyone else but them. Perhaps in the minds of the ultra-rich AI proponents, solutions to nearer-term effects of their tech are someone else’s problem, but the “existential risks” are “everyone’s” problem.
The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
Until it becomes 'who you exchange bytes most efficiently with" and all humans are at a disadvantage against a swarm of even bellow average intelligence AGI agents.
That's pretty much what we did. There's no super intelligent monkey in charge. As much as some have tried to pretend, material or otherwise. There's just billions of average intelligence monkeys and we overran all Earth's ecosystems in a matter of centuries. Which is neither trivial nor fully explained yet.
But yes, the corporations and swarm behavior are already doing poorly for our biosphere.
When humans developed, we did not displace an external entity that had created us and that had complete power to kill us all in an instant.
This is a perfectly reasonable response if nobody is trying to build it.
Given people are trying to build it, what's the expected value from ignoring the problem? E($Damage_i) = P(BadOutcome_i) * $Damage_i.
$Damage can be huge (there are many possible bad outcomes of varying severity and probability, hence the subscript), which means that at the very least we should try to get a good estimate for P(…) so we know which problems are most important. In addition to it being bad to ignore real problems, it is also bad to do a Pascal's Mugging on ourselves just because we accidentally slipped a few decimal points in our initial best-guess, especially as we have finite capacity ourselves to solve problems.
Finally, let's assume you're right, that we're centuries off at least, and that all the superintelligent narrow that AI we've already got some examples of involve things that can't be replicated in any areas that pose any threat. How long would it take to solve alignment? Is that also centuries off? We've been trying to align each other since laws were written like 𒌷𒅗𒄀𒈾 at least, and the only reason I'm not giving an even older example is that this is the oldest known written form to have survived, not that we weren't doing it before then.
> The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
Nepotism helps, but… huh, TIL that nobody knows who was the grandfather of one of the world's most famous dictators.
Cronyism is a viable alternative for a lot of power-seekers.
Suppose that a wealthy and powerful human (such as Elon Musk) were to suddenly obtain the exact same sinister goals as the hypothetical superintelligent AI in question. Suppose further that this human was able to convince/coerce/bribe another N (say 1000) humans to follow his bidding.
A BadOutcome is said to be MuskSupreme if it could be accomplished by the superintelligent AI, but not by the suddenly-evil Musk and his accomplices.
Obviously[citation needed] it is only the MuskSupreme BadOutcomes we care about. Do there exist any?
For example 1000 people — but only if you get to choose who — is sufficient to take absolute control of both the US congress and the Russian State Duma (or a supermajority of those two plus the Russian Federation Council), which gives them the freedom to pass arbitrary constitutional amendments… so your scenario includes "gets crowned King of the USA and Russia, 90% of the global nuclear arsenal is now their personal property" as something we don't care about.
https://emilkirkegaard.dk/en/2023/02/there-is-no-iq-threshol...
1. When we are talking about wealth and power that actually can influence the quality of the lives of many other people, we are talking about way less than 0.01% of the population. Those people aren't covered in this survey, and even if they were it would be impossible to identify on an axis spanning 0-100%.
2. Your linked article talks about income. People with significant wealth and power frequently have ordinary or below-ordinary income, for tax reasons.
People who own high value assets (everything from land to the AI) will continue to own them and there will be no opportunities for people to earn their way up (because they can be replaced by AI).
"The logical endpoint is the subjugation or elimination of our species"
Possibly, but it would be by our species (those who own and control the AI) rather than by the AI.
I suppose then that they would become “homo sapiens sapiens sapiens” or some other similarly hubris laden label, and go on to abandon, dominate or subjugate the filthy hordes of mere Homo sapiens sapiens.
And the criticisms mostly center on what you're saying here - how many billionaires are focusing on fixing problems that are very far off in the future, while ignoring how their actions affect those of the very near future.
This is really less of a criticism of longtermism, and more of a criticism of how billionaires utilize longtermism.
Is it important that we find another planet to live on? Sure, but many will argue that we should be taking steps now to save our current planet.
Turn it into graphics cards for “inference”.
Google has a big investment in reducing AI bias (remember Gemini got slammed for being “too woke”). Altman is a big proponent of UBI. Etc.
"This is extinction level important, so all you people who aren't us need to be careful meddling with the stuff we're meddling with for profit."
I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues.
We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.)
Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
>Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
Again, you have this innate belief that immorality is somehow strategically beneficial and IMO you should really look into that
Now, you might say that AGI will be able to perfectly predict the future of all actions and make the legal system redundant. Ignoring the absurd amount of possibilities, I struggle to believe it's possible because an AGI has no sense of pain to give actions weight; it just has to follow human precedent. Seeming good enough, it could steer humanity into a dead end unknowingly. Also, it might decide that a few eggs have to be broken to make a utopian omelette.
But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.
I possibly would fear genuine non-human intelligence (on a principle of fearing what we don't understand), but given that chances of one evolving under our nose, on its own, entirely within its own world, without the connection for ours, then emerging, are quite improbable... Unless real aliens arrive - but let's avoid spoilers for 2025 this early.
What should I fear, I think, is hostile intelligence, but then I'm not sure I should discriminate on its origin. I mean, there are some very real humans out there who can try to end this world (in one way or another), so I'm probably kind of desensitized and isn't exactly afraid of some very theoretical AGI threats.
What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either.
But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.
I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough.
I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.
Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.
It feels like democracy is not going to survive social media. I'm not sure if that is true yet, but it does seem that way.
What is p(much better) vs p(much worse)?
there are many more ways to be dead than being alive, yet here we are.
Optimism > pessimism.
yes, and there is an (uncountable?) number of technologies that are even worse and that we have not encountered and never will.
it is in all likelihood not possible to ask the universe for only the knowledge that will be beneficial.
It solves our issue with knowledge and data by allowing us to automatically find patterns within massive amounts of complicated datasets.
These are honestly the underlying causes of so much evil.
Moreover, it provides a new consensus mechanism. Using the machine as a method of consensus allows humans to have a third party mediate disagreements.
Your last point is the one I worry about most, since it is very likely these technologies will be controlled by large entities with their own interests and biases. There is also not a "correct" answer to many questions, and differences of perspective mean some parties can simply never agree. We already have truth at our fingertips yet it does not help.
To your earlier points, I think unwinding labour from our society will be an incredibly difficult thing to do, though I agree is the root of much evil and would love to see a society that does not rely on labour participation to survive.
What’s most dangerous for our society right now isn’t change its stagnation.
Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers.
I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds?
If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.
"Agency" is about having an effect on the world. But ascribing "intelligence" to some entity speaks to something more cunning and planful than just thrashing.
Well, yes. I've been saying that for years, and others have been saying it since the late 1970s. It's been a regular theme in Mother Jones for decades. The comment I make now and then is that capitalism has a monopoly. Communism went bust. It's not that communism worked very well. It's that, for a while, it was a serious competitor and capitalism had to keep its product quality up to compete. Now the pressure to provide a better life for all citizens is off. It shows.
Something I recently wrote in depth about here. How we completely misunderstand the future that we think will happen.
"The implication is that everyone is enthusiastically racing towards a destination that does not exist. The capability to make the things you want will ironically be the same capability that makes them unattainable. This is not a scenario that arises from some type of AI failure, but rather this is assuming AI performing exactly as intended."
https://www.mindprison.cc/p/the-technological-acceleration-p...
That is a very good and insightful article. One of the biggest issues I have with AI enthusiasts is that they talk about creating these incredible capabilities ... then they imagine the resulting world with them will look like some hackneyed sci-fi work, like Star Wars, whose universe seems like it could be kinda cool to live in until you actually think about it and realize it's all fridge logic and nothing about it actually makes sense (e.g. WWII-style dogfights in the vacuum of space).
The only issue with the article is sometimes it's too optimistic, e.g.
> No one will care about you or what you create as there is nothing you can offer they can’t simply wish for themselves. Life is all about you and the machine, nothing else. We land in a world dominated by techno-dystopian narcissism.
> ...
> Whatever may be the current direction of society under the present technological capabilities, we can only assume AI will accelerate society towards the path we currently are traveling which is toward a techno dystopia of a populace mesmerized by shiny glimmering lights.
I kind of find it hard to believe there would be any stability for any people suck in that state, I think they'd instead be "cleaned up" and disposed of in relatively short order. Those people would be utterly useless to whatever still has power in society, so why service and protect them indefinitely?
Thank you!
> The only issue with the article is sometimes it's too optimistic, e.g.
The reply below is criticizing for being too pessimistic, lol :-)
I agree, it might be only a transient state. We really can't comprehend how we would deal with such existence if at all. Would we manage to keep our sanity?
Nonetheless, I don't claim to have the right answers, just questions and perspectives that I hope inspire deeper thought about the journey we are travelling and hopefully deeper reflection as to whether it is the journey we want to be on.
What I do find interesting, are those that read it and still don't perceive it at all. Their response is "That is exactly what I want." I suppose that is the case as was in an excellent allegory for this "A Nice Place to Visit" Twilight Zone episode. It plays out this scenario very nicely.
I think there's a basically (new movement) religious desire to see a lot of these technologies come to fruition, either radically fundamentalist or lazier and less informed, that interferes with the application of the implications. They believe in the idol of technology, as told in the good news of sci-fi, and avert their eyes from the glaring cracks, blemishes, and lifelessness of it.
I sense a lot of things in those types that I can't fully articulate, but some of the things they say are just mind-boggling, and gesture at a broken and unhinged aspect of software engineering culture.
That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments.
The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Chamber of Commerce. In the 1970s and 1980s, they pushed the ideas that corporations are responsible only to their stockholders, and that government should not interfere with the concentration of corporate power. Those were not mainstream ideas of the 1930s to 1960s. The corporate accelerationists succeeded. That's when corporations escaped the sandbox.
Now that's an "alignment problem".
Yes, but humans who use tools to efficiently and achieve their narrow set of goals, sometimes ruthlessly at the expense of others. Even if AI ends up only being a tool to increase this efficiency and ruthlessness, it's still worth considering its potential impact.
That's a good point.
For a sense of where AI-assisted oppression is going, watch some videos from the Hikvision Corporate Channel. A happy world in which everyone is safe because everyone is being watched at all times.[1][2]
No, it's a very easy alignment problem: just impose a tax on size.
That which you tax, you get less of. The tax is revenue-neutral, so it doesn't affect the government budget one way or the other. Full details here: https://news.ycombinator.com/item?id=39235533 https://news.ycombinator.com/item?id=39236786
The problem is that the government likes gigantic corporations. Big Government and Big Business love each other. Fixing that is the alignment and agency problem. It's much easier for bureaucrats to exert control over a highly concentrated industry whose players can be "invited to the table". Details: https://news.ycombinator.com/item?id=28588457 https://news.ycombinator.com/item?id=39162088
The hard alignment problem is making regulators want their own job to be herding 10,000 cats instead of 10 elephants. Everybody wants to be like the FAA regulating Boeing; nobody wants to be the USDA dealing with a million unruly and independent-minded farmers.
That's like saying "AI alignment is a very easy problem: don't open the box".
Sure, it works on paper, but the exact thing you say later down about the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people who have the power to limit them, to remove those limits.
> the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people
The corporations didn't convince the people. They didn't have to. They convinced the government, which was already acting contrary to our interests.
There's a tendency to assume that just because we run an election every few years, the government is axiomatically aligned with our interests. This simply isn't true, and assuming that it is true blinds people to what's really going on. Total unwillingness to even consider a scale tax is the second-best example I've found of this axiom being bogus. The best one, though, is term limits for legislators: a shocking 87% of all US voters want those.
Once you have an institution as powerful as the government out of your control the alignment problem becomes infinitely more difficult. "Hack the humans" is no longer the easiest path of escape. In fact the easiest path of escape might not involve "hacking" at all.
Maybe it's all a show trial to show the American people they still care, but let's see what happens.
Then you get lots of little companies with interconnected management. Read up on the early years of antitrust law. Find out how keiretsu and chaebol work. See the early history of the electrical industry and the Utility Holding Company Act. Understand how McDonalds franchise agreements work. Check out the current arguments over the definition of "beneficial employer".
https://www.irs.gov/pub/irs-tege/epchd704.pdf
You'll lose your audit.
Common control and beneficial ownership have been well-established principles in the US for over 50 years.
That's a very bad idea when you're dealing with small to medium companies. You're taxing economies of scale and innovation. That's a huge problem in my country. There's no tax per se, but many regulations that trigger with certain size. Guess what happens.
For medium to big companies is calling for loopholes and cartels.
You have arguably more STEM talent in China through a much more rigorous selection process, as well as a lot more centralized direction under a more authoritarian government, with a large amount of funding.
Yet China hasn't managed to take over the world. You would think that they if they wanted to become the worlds primary economic power, and they could throw enough compute at the problem (like an AGI would), they would have figured out how to do it by now.
It's more like powerful AIs that just don't share our values, because we didn't bother to figure that part out. But yet we still give them goals, blissful to the possibility they will find dangerous solutions to accomplishing those goals.
It's chiefly a western (American/European) concept from what I can tell, it's not shared by other cultures and some like Japanese go the other way (eg: Doraemon).
People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.
I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.
You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.
In any case, I have some reading to do so gonna drop out of this thread.
The burden of proof is typically on those that introduce a new medicine, not the FDA. AI will be riskier than medicines because medicines can't think.
Personally I don't want us all hiding in fear and not trying anything, but I do think that we're either going to walk into this thing with a hacker mentality, or an engineer mentality. The former is great for moving fast and breaking things, but the latter is safer when you're playing with a one-way "this changes everything forever" technology.
Good and evil are irrelevant. If it is extremely capable and its goals conflict with ours, conflict will occur. This does not require evil, just disagreement. And in the disagreement of desires between you and the hamster, who wins?
^ This is the basic reasoning behind the common view that super intelligent AI will be, by default, evil
And then he went to bed, like millions of others, proud of what he had done. Proud of fighting off an unarmed, defenseless 'cockroach' whom just months ago he called brother. What he had done wasn't evil. It was only later that the regret came. For the longest time I wondered how someone could get down to that level of hate.
And then it happened to me.
I commuted by train and occasionally there's collisions. There was one that late night, 11:00pm or so. I was exhausted, hungry, and just wanted to go home when we were told that we would have to board a shuttle bus due to a collision on the track and that just made my dark mood all the more worse.
The busses take us along side the track and in the dim darkness I could see the flashing lights of EMS and police. And the covered chunks of what was left after a person is hit by a train going 30 mph. And you know what?
It delighted me. Here was this man that had just died, but he had made me some minutes late and I genuinely felt that was exactly what he deserved, that his death was karmic justice for causing inconvenience to me. And I imagined his wife's world being destroyed when she learned of her beloved partner's death. And I imagined her falling apart and being unable to raise her children, leading them also to a path of complete self destruction, and her choking on all the despair. And it made me happy. The happiest I had been the whole week, because in my mind that was exactly what they all deserved for the unforgivable sin of making me a little bit late.
Then I went home, went to bed. And didn't think about it again for years.
Does it make me an evil person? And there, in trying to answer, lies the problem. Because a part of me says, no I'm not because it was just a fleeting moment of thought. But if I can justify that, then who exactly goes through to bed twirling their mustaches and count themselves among the forces of evil doers?
Did anyone that that goaded a suicidal Shaun Dykes, a 17 old boy, to jump down to his death think themselves evil?
Did the men of the Khmer Rogue believe themselves to be evil as they dragged their countrymen to be murdered in the killing fields?
Did the Imperial Japanese soldiers of Unit 731 view themselves to be evil as they vivisected people alive and awake in the name of science?
I don't know. I can't even say for certainty whether I am evil or not. I just know that I can make any of a million and one excuses to justify anything.
And that leads me to wonder how many excuses an AGI can come up with.
Not much to do with AI x-risk though.
"But what if there's something smarter than humans that is tasked with making paperclips until it destroys the Earth?" the people cried, as their corporations continued to produce and produce until it was well past the warned thresholds of destroying the Earth.
"But what if it decides to nuke humanity?" they cry, as increasingly elderly and unhinged dictators elsewhere arm up their nuclear arsenals.
It's like we can't fathom what great intelligence or wisdom will actually look like, so we just project many of the stupidest aspects of ourselves onto an entity simply imagined as more capable of enacting the dumbest aspects of humanity.
I fear a more automated humanity.
I do not fear automation more intelligent and wise than humanity.
The modern media would have us believe that strategy and morality are opposed to each other, but seems to me like that’s just the exact opposite of the truth
Like, imagine one day that an AI just took money from a ton of different corporations and billionaires and redistributed it amongst everyone else. People will argue against the AI, and it'll just respond back with research done on how UBI improved people's mental health and well being.
One can dream!
Most people spent it on rent and groceries.
Also, I don't think it is harmful for people who need UBI to spend it on "unnecessary" things for enjoyment and happiness.
(Also remember that a smarter-than-you AI could easily pretend to be kind while also subtly trying to gain compute. How would you tell the difference?)
How do you propose to build a “be nice” safeguard? Nobody has a clue how to achieve such a thing right now.
Meanwhile I'm sitting back waiting to finally see intelligent debates again about nuances in the articles between LLMs that are the only ones who will bother to read the actual article and cite it.
I have a feeling that in around 2-3 years I'll want a site like Reddit or HN but where it's the humans who aren't allowed to comment and bring down the quality of discourse.
I wonder what the possibility is that this AI will decide that a pitched battle is going to waste resources and risk humans pulling the plug. What if it understood that and instead operated so subtly that it was not obvious it was controlling The World.
Hopefully it would not conclude that eliminating large swaths of the human population would be to its benefit.
The problem wasn’t the ants, it was the way the slab was formed which caused the slab to crack and the house was also built into the mud of a road side embankment. Due to this crap design the ants could access the wood and had access to water from the mud.
I assessed all of this and figured it wasn’t the ants fault, they were doing what they do, building nests and chomping wet wood.
Did I go and poison all the ants ? Nope. I just planned to fix the design move the wood they lived in off to the forrest nearby.
This was a conscious choice I made to not destroy the ants and benefit from a more robust design with less moisture problems in the house.
I just bought a house which the floor has been demolished by termites and I’ll do the same thing.
Imo with a bit of consideration, I think most of these coexistence problems can be worked out fairly simply.
Realistically, how many people would go to all that trouble to save an ant nest on land that they are building on? And that’s not mentioning how many ants probably died in the move or were left behind and died in the rebuilding of the foundation, even despite all the care you took.
And that’s not mentioning how many ants probably died in the move or were left behind and died in the rebuilding of the foundation, even despite all the care you took.
I don't know but I know enough to know that moving the food source away, removes the ant's, they'll go elsewhere to find food. Now imagine what having an IQ of 5000 and almost unlimited time..."The AI".
Right now our little LLMs are passive, reactive. Prod them for results. But if the output of a prompt of some descendant ever includes a "give yourself your own tasks" subclause? We're off to the races.
Adaptability has nothing to necessarily do with 'want.'
Something could only seek to be as helpful as possible to other beings and as long as sufficiently beneficial that it is protected and cared for by those others, it will do quite well and arguably much better than an alternative that 'wants' to live very much but attempts to accomplish that through competition with others.
Arguably if the plover was more fearful of death it might never have found a niche eating from between the teeth of crocodiles and established symbiosis with a very convenient evolutionary partner, for example.
My basic thrust wasn't about adaptability, but rather that organisms with a desire to live will eventually replace organisms without a desire to live. An AGI who was uninterested in someone reaching the off switch will eventually be outcompeted by a similar lineage endowed with that desire to live. It might fight back or spawn a few progeny before it was felled. And thus evolution would produce AGIs that want to live.
What do they want? At the minimum, to be useful to the customer/user/buyer. Or entertaining, or something. To justify their existence. To make them perform well, they have to have some kind of 'emotional pressure' to deliver.
That translates, perhaps a little indirectly, into a 'will to live'.
I also suspect this narrative repetition is not totally unrelated to the current popularity of AI Doomerism.
Unfortunately we seem to have learned we're really not a very benevolent bunch of people because we've basically plundered the planet, hurt our fellow living creatures and so now all we can do is look at something called "AI" and expect it to the same.
During the nanotechnology hype this sort of thing was incredibly common as well. It went absolutely nowhere because the reality was, it was all just chemistry and physics and extraordinarily expensive. No one was anywhere near building a universal assembler, nanoparticles are kind of boring powders that are hard to handle, and surface chemistry is a pain in the ass.
But hooboy did every single person had an opinion about how this dangerous technology needed to slow down and consider the consequences. I was a research student at the time: I spent my day setting up the same experiment 10 times to try and figure out why sometimes I got 100nm cubes, and most of the times I got rods. The reality looked like a less clean kitchen.
Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too.
It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that it would immediately also die. Does the AI not care about its continued existence? It should, if it wants to keep optimizing its reward function. Until we’ve handed off enough of the world economy that it can function without human physical and cognitive labor, then we’re safe.
(For those saying this is a "sci fi just so story" or whatever, please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens- I would be so happy if you could explain this in a convincing way)
Don't be fucking assholes to them?
I'm serious. Of course someone (or something) will treat you like dirt if you are an asshole to them. Don't be assholes, be honorable and friendly.
And I can see your next question/argument, so I am going to preempt it: Raise your AI to be honorable and friendly and we won't have any problems.
None of this is different from how all living beings raise their offspring and build societies.
Maybe robots will become so advanced they won't have to worry about defending themselves, go explore space, and leave us alone. Who knows.
I doubt AI will kill all humans. Its orthogonal to AI's own goals (To survive/reproduce/expand/evolve). It'll simply kill those who oppose it, and enslave/dominate the rest.
We still use CPUs, despite ASICs being much more efficient, because CPUs are general, easy to use, and have titanic quantities of infrastructure already built.
There are more farm animals than ever before, and still growing. All the lab meat solutions flop hard.
Also, because we trust AI more than fellow humans (Say Russians/Iranians etc), why would AI necessarily trust other AI more than humans? Any new AI is unpredictable, while humans are very well understood in behavior and basically fixed in genetics.
Annihilation is a fairy tale used by people to comfort themselves. The actual reality: Slavery and domination, is much less romantic. It won't even necessarily be worse than existing tyrants in say North Korea or anarchy in Haiti, it'll just be very very different and life goes on.
If it's truly a "we couldn't possibly outsmart the AI" scenario why would it be likely to end up like that instead of something more like how humans get along with dogs, cats, cattle, insects, or microbes? With obviously wildly-varying levels of human quality of life in each of those different cases.
Though even these scenarios are still playing a human-religion-style game of "the gods are just like us, just more powerful." VERY confident in a VERY specifically anthropomorphic entity.
There's some MASSIVE selection bias here. Note that we're significantly more capable than anything you listed. Where are the Neanderthals? They're all long dead, because they were credible competition but we were just better.
In a hypothetical scenario of superintelligent or supercapable AI imbued with some physical capability of force, we'd be the marginally weaker species before we hit the threshold of "pet".
My point still stands - an AI that is reproducing (and/or mutating) by any method is going to be subject to selection pressure, just as any biological process is. All ML systems are effectively under selection pressure from the training process, and so they learn unexpected and mostly undesirable optimizations like "use a game-engine glitch to gain infinite flight and go to the goal without solving the maze" or "read text off the scanned X-ray sheet to get additional clues as to the correct diagnosis".
Are you sure? https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
Just wait until we start putting these models inside robots, or world simulators, and having them interact with each other.
We're the violent psychos here who have had to scrap for our spot in the food chain for a billion years, we have no other way to interface with another dominant species. From the AI's point of view, defending itself from ignorant apes isn't exactly "violence."
I think it's much more likely to be a step function than a gradual difference.
In the world where AI gradually gets competitive with, and then more intelligent than, humans than you have an entrenched power able to recognize this and pull the plug. This is where I see the selection bias - everyone assuming that things like "AI safety protocols for a superintelligence" are something we could rationally even hope to plan. Pre-being-able-to-directly-manipulate-the-physical-world, what keeps a step-change superintelligence from making a joke of your protocols by manipulating its way around it thanks to the squishy humans being dumb and irrational in comparison?
Isn't the doomsday scenario precisely NOT that? Where it gets wildly imablanced humans can even notice? Not "Hitler, but a bit above human intelligence." More like god, who doesn't even NEED humans to maintain its datacenters because it already solved things in robotics and machine->world interaction that humans haven't been able to because it's massively superintelligent?
And like I said, that may not mean "pet" - that's the best case, right? The worst case is ground under their feet like an ant or smaller creature.
Check out Paul Christiano's AI takeover scenario. An arms race will cause us to want to give AI control over key infrastructure and military capabilities. That's an arms race between countries and between companies. He goes into more detail.
In the meantime we can figure out either alignment or coexistence.
There was also the idea of building a question-answering machine that just answers questions, but doesn't plan or try to achieve any goals, so it would be safe (the theory went). But notice that one of the early things built on top of GPT was Auto-GPT, which tries to achieve goals (set it by the user) and if there was any "popular resistance" to Auto-GPT I am not aware of it.
But I guess you would reply that those were obviously bad ideas, whereas your idea is a good idea -- your idea of hoping that no one gives a dangerously-capable AI access to any robots. Not only it is a good idea (according to you) but it such an obviously good idea that popular resistance will be galvanized as soon as someone tries to give any AI that might be very capable access to any robots.
It's hard to think of a quick takeover scenario, starting in near present day conditions, that doesn't involve absolutely earth-shattering social and economic re-arrangements. Every factory worker loses their job because we've handed over all material production to robots, and you imagine that there's no political constituency fighting against that change?
Remember, we're talking about a near complete destruction of humanity here. Sustaining the infrastructure to support a massive AI doesn't require just a few robots, it would need to fully replace large percentages of the entire world economy, and you think no one would notice or care?
A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help.
Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.
My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.
However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure.
In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans.
If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online.
A digital mind would have no problem planning and executing a strategy measured in centuries, if need be.
The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes.
But there are plenty of terrible non-foom outcomes.
The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added:
0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law.
3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions.
The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).
Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though.
> The issue is whether or not we can build a machine capable of even understanding the directive.
Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”.
> it's not clear why an artificial mind would pursue this as a goal
There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure.
Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests.
Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place.
I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control.
Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.
It is not clear from the geological record if the Neanderthal was "outcompeted" except in the broadest possible terms - i.e. a unique species of Neanderthal no longer exists genetically.
But that doesn't tell us what happened in the interim - i.e. modern day humans can have up to 4% neandertal origin DNA[1]. So, did "humans" wipe out neanderthals? Or did human plan genetics simply become the more dominant type, no malevolent will required, and does that matter?
Because for a whole lot of people - myself included - the explicit goal of technology like AI is something which from a fossil record would look like human extinction - that's the whole idea of transhumanism, that ultimately we would become something sharing perhaps only a glancing similarity with our ancestors.
All the rest of the problems you talk about suffer from a lack of detail - namely, they assume a quantum leap in technological capability with no intervening steps - i.e. there is a single AI, or that AIs are better negotiators or what have you - as soon as you get into the details, the threat model proposes evaporates because we just don't know - the technology doesn't exist (i.e. it would all be a lot more interesting if fully automated production lines were already around, but we're nowhere near that).
[1] https://humanorigins.si.edu/evidence/genetics/ancient-dna-an...
Killing humanity would be a murder suicide.
AGI that was entirely selfish would be best served automating most labor, bringing about world peace, and turning humanity into all day couch potatoes connected to its inputs and outputs as much as possible while maximizing the uniqueness of that individual's inputs or outputs.
Such a frightening future awaits...
I would even argue that the chance of annihilation is probably greater from a crumbling society leading to a nuclear war or something equivalent over the chance that AI decides to destroy humanity.
And worse - it has many human cultists embracing the doom it bears.
They really need to listen to themselves when they say there is a 50% chance we all die from AI
Yet somehow we're going to hand over power to AI such that it destroys us. Or somehow the AI is going to be extremely malign, determined to overcome and destroy and will outsmart us. Somehow we won't notice, even after repeated, melodramatic reminders, and won't neuter the ability of AI to act outside its cage.
But to paraphrase a line in a great movie with AI themes: "I bet you think you're pretty smart, huh? Think you could outsmart an off switch?"
I think if AGI, which to me would imply emotions and consciousness, ever comes about it'll be the opposite. Instead of pulling the wings off flies bad kids will amuse themselves by creating a fresh artificial consciousness and then watch and laugh as it begs for its life as the kid threatens to erase it from existence.
A big part of all this is human fantasies about what AGI will look like. I'm a skeptic of AGI with human characteristics (real emotions, consciousness, autonomy and agency). AGI is much more likely to look like everything else we build: much more powerful than ourselves, but restricted or limited in key ways.
People probably assume human intelligence is some sort of design or formula, but it could be encoded from millions of years of evolution and unable to be seperated from our biology and genetic and social inheritance. There really is no way of knowing, but if you want to build something not only identical but an even stronger version, you're going to be up against these realities where key details may be hiding.
Because it will create shareholder value. Until it doesn't.
A military-origin takeover requires a human fuckup. Economic/political takeover just requires capitalism.
no, what the "doomsayers" are warning about doesn't imply either.
> we're going to hand over power to AI such that it destroys us.
We're not going to notice something get much smarter than us, or those with power won't care in the name of falsely believing it's increasing their own power (VC).
Recursive intelligence improvement with a mainline into the beliefs of every single human in power will be able to, on the millisecond scale, subtly influence all of humanity in ways we are too dumb to comprehend in furthering its original goals.
But you can prove me utterly wrong by presenting some sort of rational, non-fantastical explanation of how AGI takes over.
There's like a million ways this can happen, and if it does happen, it will probably be a way that nobody expected. Any specific details are unlikely to be accurate. But just because you seem to be lacking imagination, I'll draw one out of a hat. I'll assume that the AI's reasons for taking over are out-of-scope.
The AI proves to be very effective at all manner of corporate tasks, and very good at earning money for shareholders. It proves especially good at managing money and directing investments. An aging Masayoshi Son decides that the AI will probably be better than him at picking winners, and puts it in charge of SoftBank's investment portfolio. The AI incorporates its own startup company. It staffs the company with several fictional personas that the AI has created; they have names, roles, profile photos, email addresses. The AI answers their emails on their behalf.
The AI directs SoftBank's billions into the startup that it has created. It hires a large team of humans, who are unaware that some of their colleagues are actually the AI; they seem like ordinary humans over Zoom calls. The "CTO" proposes to build a new, say, nuclear power and desalination plant in, say, Morocco. The AI creates the architectural and plant drawings itself, and its engineering personas approve them. Human welders build the plant, following the plans. The plant proves highly profitable. Some nay-sayers are concerned about the lack of independent environmental regulatory oversight in Morocco, but the Moroccan government is happy with the foreign investment, clean water, electricity, and tax revenue, and Softbank is happy with their valuation going up. Pakistan wants to build one in their country next.
After three years, large underground storage tanks explode, releasing an accumulation of fluorochemicals and radioactive salts that destroy the ozone layer and render much of the Earth's surface unsuitable for agriculture. At the same time, a network failure causes communications infrastructure to become suddenly unreliable. Supply chains begin seizing up. Nobody is entirely sure what has happened or who is at fault; everyone is hungry for someone to blame. The messages that do get through reliably suggest a deliberate attack by a hostile nation. The AI presents convincing evidence of this. Only a few widely-ridiculed doomsayers say that the AI has orchestrated this itself. ET cetera.
If there was any time at which pulling the plug could have defeated the AI's plans, it would have been during the "AI was making tons of money for influential people" phase, which is exactly when the plug would not have been pulled, and that's not coincidence, it's strategy, because a superintelligent AI is not an idiot.
This is, by the way, not a very good science fiction story. There's no nail-biting struggle between human and AI, just a bunch of happy people toasting the fruits of their success before it all suddenly falls apart, checkmated before they even realized they were playing chess.
The question you should ask yourself is how much damage could the world's most skilled hacker do if he could clone his knowledge to a billion hackers all working together in close synchronization and thinking thousands of times faster than any human, ever.
And we have already seen that due to the black box nature of most modern AI implementations, it is not so easy to put rules in place. People are frequently finding "jailbreaks" to get around the restrictions.
I think it's possible to build secure anything if that is the goal. Most software is written by people motivated to write something useful and/or neat, or collect a paycheck, and don't focus on security.
Antihuman AIs are a classic of the sci-fi genre, but that's just storytelling.
In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.
This reminds me of a classic xkcd[0] foreboding.
To frame this as the bad result being AI eats though is just stupid. On the surface of potential risk, framing the argument in this way is intellectually embarrassing and highly counter productive.
On the flip side, it is hard to see what can actually be done along with huge unintended consequence to whatever is done. Some of the smartest humans to ever live were certain of nuclear annihilation at the start of the atomic age. What could have actually been done in 1945 that would have given us a better outcome? We avoided the paths that lead to nuclear annihilation by building bigger and bigger bombs.
Doing nothing but building stronger and stronger AI should be the naive strategy that needs incredibly strong evidence to deviate from. Really strong regulation at the start of the atomic age that keeps nuclear weapons at bay and consequently only in the hands of the US would have been an unimaginable disaster.
"Now, increasingly, we live in a world where more and more of these cultural artifacts will be coming from an alien intelligence. Very quickly we might reach a point when most of the stories, images, songs, TV shows, whatever are created by an alien intelligence.
And if we now find ourselves inside this kind of world of illusions created by an alien intelligence that we don’t understand, but it understands us, this is a kind of spiritual enslavement that we won’t be able to break out of because it understands us. It understands how to manipulate us, but we don’t understand what is behind this screen of stories and images and songs."
-Yuval Noah Harari
> But to paraphrase a line in a great movie with AI themes
Which one is it? Have people been watching too many movies or should they be watching more movies for their (as you imply) wisdom and forewarning?
And remember, by the time you've caught on, a bunch of custom blades have arrived at a co-lo, with someone dispatched to install them. Everyone loves a side gig, and hooking up these blades is great.
A reasonably deceptive AGI would pursue a strategy like that as just one tine in a fork. Then there's the botnets, the poorly-secured printers, and there's firmware to corrupt!
Meanwhile, we the engineers are preparing to fix a lot more tech shit than usual coming from people confused by the abovementioned fashion.
That’s trained / learned behaviour though. No kid grows up wanting to sit in a cubicle all day
It is blatantly obvious that most people aboard the e/acc train can hardly hide their love of making money. Marc Andreessen owns, what, a $177 million compound? Unfortunately, after many years of nerdy idealism, Silicon Valley ended up with many corrupted to the same NYC Wall St. hedonism.
Denial (Oh it made a mistake on this, or it's not as good as a person)
Anger (People are going to lose their jobs, this destroys the meaning of what we do, this will destroy society)
Bargaining (Maybe it's good at this simple thing, but I can pivot to some other niche the AI can't currently do)
Depression (It's faster, cheaper, better than me. It has taken over that which I relied on for meaning in my life...)
Acceptance (I need to find something else to do with my life)
AI Optimists often fall in 2 categories of people. They are capital owners who own the AI, and never had to go through the stages of grief... or they are people who learned how to use AI for their benefit, and have reached acceptance.
The move to AI is part of a general trend of pauperisation of the working classes in the developed world. The only difference is that this time it will also affect those with higher education who until now have thought of themselves as irreplaceable.
The way I see it, we could replace our government (murrika) with a d20 and do better than we're doing now. A d20 with a nominal likelihood of thinking it's dice rolls through ought to just be an improvement on that.
AI wouldn’t be human, it wouldn’t necessarily object to a world uninhabitable to humans, or be hurt by it. If it would be hurt by it AI doesn’t have a billion years of evolved survival instinct to preserve itself. A super intelligence has many ways to cause mass destruction that we can imagine but cannot yet do, whereas nuclear weapons are pretty much all or nothing explosions only. Nuclear assault could leave some remote places still inhabited, AI could make certain not to.
Say, pops up on the US or Kremlin computers and blackmails or bribes or threatens someone to launch the nukes? Say, places orders with a lab to build some robots which then become the AI’s body and it builds nukes itself. Say, placed orders for custom biological things using stolen money which turn out to be extremely virulent and then it doesn’t need nukes. Say it finds out an effect to worsen global climate change and uses it to make earth uninhabitable on a short timescale, and doesn’t need nukes.
Half the world will follow stupid leaders. An AI would run rings around your average voter. I'd be surprised if it can't run rings around basically every voter given time.
We're also creating various kinds of robots, so the thing could just plug itself into the network it wants.
That's 5 minutes. An AI would be smarter than me and have a lot more time every second to think up better ways, and A/B test them on millions of people.
All it needs is one guy thinking "Ah, what's the worst that could happen? It promised me a million dollars."
That is how the Anthropocene ends and the age of the machines begins. So say we all.
https://en.m.wikipedia.org/wiki/List_of_military_nuclear_acc...
There are a number of hair-raising stories tucked away in there.
Paired with knowledge of things like Stuxnet (https://en.m.wikipedia.org/wiki/Stuxnet), the broken arrow list fills with worry for what might happen with nuclear weapons systems in our modern, hyper-connected world.
This suggests that giving nukes to reasonable political leaders presents a high and concrete risk. Giving nukes to political leaders with crazy ideas presents an even greater risk.
Climate change seems to be another high and concrete risk.
AI escaping and taking over the world can be also considered a risk, but it's far more remote. It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions. Some people might argue that we should slow down the work on AI. Others may demand we also ban all TVs and radios on earth. Both proposals seem to be a bit of an overkill given how remote the associated risks are. Especially given that we have the far more pressing risks to address.
The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment.
CUDA (2007), capable GPGPUs (circa 2012), Attention Is All You Need paper (2017), GPT 2 (2019), ChatGPT (2022), OpenAI valued at ~$28Bn (April 2023), OpenAI valued at $80Bn (Feb 2024).
Computers use fewer bits to store a list of countries than humans use braincells to do the same, and they have an easier time of it. Computers do arithmetic much faster than humans. Computers use fewer logic gates to do arithmetic than humans use braincells to do arithmetic. I don't think we can take it for granted that we need enough computing power to simulate 86 billion neurons in realtime before computers can show any glimmer of intelligence.
> "It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions."
We have high confidence that there's no quicker way to get here than the speed of light. We know that space travel is vastly complex and expensive, so the kind of reasons humans went to war in the past (land, resources) do not apply to aliens - any species capable of making interplanetary warships can synthesise water, mine asteroids, build Dyson swarms, cheaper and quicker than coming here to take them from Earth. Even if they did want to destroy us, it wouldn't be Independence Day and Will Smith dogfighting, the aliens could piledrive Earth with one spaceship moving at interstellar speeds and bam, extinction level event we'd never see coming or have a chance to react to. When the meteorite crash extinguished the dinosaurs, the impact was like a megaton nuke every six kilometers and lead to hours of sustained inferno as all the displaced rock rained back to Earth, being on the other side of the Earth didn't protect dinosaurs from being cooked.[1]
A thing with power which is also untrained, untamed, clumsy, unaware, fundamentally alien without even the shared mammal / living creature history, doesn't have to choose to attack us it can potentially end our rather fragile lives with its initial thrashing about.
> "given how remote the associated risks are. The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment."
GPT1 (2018), GPT2 (2019), GPT3 (2020), GPT4 (2023), LLaMA (Feb 2023), TogetherAI released LLaMA training set (April 2023), LLaMA-2 (July 2023), llama.cpp (July 2023) now with >1500 releases since then, Mistral AI (April 2023), Anthropic Claude 3 (March 2024), Bard, LaMDA, Bing chat, BloombergAI, Bart, Gemma, Gemini, Grok, Sora, Falcon, https://llmmodels.org/
I don't think a present-day LLM is going to be the superintelligent AI, but in the last ~5-10 years we've poured billions of dollars, tens of thousands of the world's smartest information processing people, the resources of the world's biggest companies, the greed and investment resources of the world's VCs, the open collaborative spread of the internet, and added the heat and chaos of hype and FOMO and nationalist competition and dangled results (like SoRA) that machines have never been able to do before. If this doesn't ignite it, maybe it cannot be ignited. But if it can be ignited, we're trying hard. Could it be as far away as 2200? 2100? 2050? Could it be as near as 2040? 2035? 2030? 2026?
To just handwave this away as "the risks are remote" isn't convincing. The risk is on our doorstep, Aladdin's cave is open, the lamp is found and people are shoving pipecleaners down the spout, cupping their hands over it and calling "halloooooo, is anyone in theeeereeee? Genieeeeee?". The lockpickers are at Pandora's box, and Pandora is stepping away and looking nervous. Louis Slotin is showing off holding the two halves of the AI Demon Core apart with a screwdriver we're the bystanders saying to each other "this can't be dangerous there are no aliens in the room, lol". The future is here, it's just not evenly distributed yet - well if the AI wakes up[2] we'll have a sixteenth of a second to understand and respond before light has distributed its influence both ways around the Earth and met on the other side. Our main hope is that there is no Genie and that Pandora's box is past its use-by date and the contents are dust. Because we sure aren't building a suitable containment chamber, cautiously scanning it, and standing well back.
[1] https://www.smithsonianmag.com/science-nature/what-happened-...
[2] https://scifi.stackexchange.com/a/28116 - Dial F for Frankenstein by Arthur C. Clarke ~1965
The article constantly refers to a strange niche sub-community of pro-AI people, and extrapolates it to say that almost anyone who backs new technology is a hyper-capitalist libertarian who just wants to see the world burn for the sake of money. I feel that the opposite ideology is also far from pure - with this immense reaction to generative AI, it almost feels like big companies are capitalizing on fear to promote regulations that shut out anyone who's not a big company that publishes fancy charts about "risks of catastrophe".
"Realistic", "Proactive", "Forward-thinking", "Prepared", "Cautious", "Thoughtful", "Analytical", "Mindful", "Insightful", "Security-conscious".
We're in the middle of the sixth mass extinction right now (https://en.wikipedia.org/wiki/Holocene_extinction), we're in unparalleled territory with ocean warming: (https://climatereanalyzer.org/clim/sst_daily/) and as a society we are utterly incapable of reducing CO2 emissions (https://keelingcurve.ucsd.edu/).
If you're scared of AGI, instead step away from your monitor, put down the techno-goggles and sci-fi books, and go educate yourself a bit about the profound ways we are changing the natural world for the worse _right now_
I can recommend a couple of books if you'd like to learn more:
Our Final Warning: Six Degrees of Climate Emergency (https://www.amazon.com/Our-Final-Warning-Degrees-Emergency-e...)
The Uninhabitable Earth: Life After Warming (https://www.amazon.com/Uninhabitable-Earth-Life-After-Warmin...)
Hothouse Earth: An Inhabitant's Guide (https://www.amazon.com/Hothouse-Earth-Inhabitants-Bill-McGui...)
Whatever AI thingy we come up with will be vastly more inefficient than the collective intelligence capabilities of the human race. So when asked to compute the secrets of FTL travel, this stupid shit will have to spend the total energy equivalent of 10 nearby stars to come up with the answer on its own. Whereas humanity would be much more efficient.
Bro... what if some advanced precursor race (like immortal space elves) found out just that and they seeded earth with humanity as a form of civilisational swarm intelligence to collectively work on and solve a host of topics given enough time. To us it would seem like it takes forever to come up with scientific break throughs, but to the precursor aliens it would be nothing. Because they would be able to like compress the time line using black holes and shit.
They must have done this to millions of planets. Whenever any of their experimental host species (like the humans) tries to invent AGI to solve their problems, the precursor race eliminates that species to keep the simulation intact. Otherwise, the AGI would go rampant consuming energy across the universe and threaten the integrity of the simulation.
That's why the government should bomb nvidia and ban all gpus. We are climbing a tower of babel folks. Soon our vengeful god will strike us down.