> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. [0]
Signed by Demis Hassabis, Sam Altman, and Bill Gates among others
And the criticisms mostly center on what you're saying here - how many billionaires are focusing on fixing problems that are very far off in the future, while ignoring how their actions affect those of the very near future.
This is really less of a criticism of longtermism, and more of a criticism of how billionaires utilize longtermism.
Is it important that we find another planet to live on? Sure, but many will argue that we should be taking steps now to save our current planet.
Turn it into graphics cards for “inference”.
That said, I don’t think that it’s necessarily disingenuous so much as it is myopic - to them of course AI is exciting, world-changing, and profitable, but they (willfully or not) fail to see the downsides or upsides for anyone else but them. Perhaps in the minds of the ultra-rich AI proponents, solutions to nearer-term effects of their tech are someone else’s problem, but the “existential risks” are “everyone’s” problem.
The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
This is a perfectly reasonable response if nobody is trying to build it.
Given people are trying to build it, what's the expected value from ignoring the problem? E($Damage_i) = P(BadOutcome_i) * $Damage_i.
$Damage can be huge (there are many possible bad outcomes of varying severity and probability, hence the subscript), which means that at the very least we should try to get a good estimate for P(…) so we know which problems are most important. In addition to it being bad to ignore real problems, it is also bad to do a Pascal's Mugging on ourselves just because we accidentally slipped a few decimal points in our initial best-guess, especially as we have finite capacity ourselves to solve problems.
Finally, let's assume you're right, that we're centuries off at least, and that all the superintelligent narrow that AI we've already got some examples of involve things that can't be replicated in any areas that pose any threat. How long would it take to solve alignment? Is that also centuries off? We've been trying to align each other since laws were written like 𒌷𒅗𒄀𒈾 at least, and the only reason I'm not giving an even older example is that this is the oldest known written form to have survived, not that we weren't doing it before then.
> The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
Nepotism helps, but… huh, TIL that nobody knows who was the grandfather of one of the world's most famous dictators.
Cronyism is a viable alternative for a lot of power-seekers.
Suppose that a wealthy and powerful human (such as Elon Musk) were to suddenly obtain the exact same sinister goals as the hypothetical superintelligent AI in question. Suppose further that this human was able to convince/coerce/bribe another N (say 1000) humans to follow his bidding.
A BadOutcome is said to be MuskSupreme if it could be accomplished by the superintelligent AI, but not by the suddenly-evil Musk and his accomplices.
Obviously[citation needed] it is only the MuskSupreme BadOutcomes we care about. Do there exist any?
For example 1000 people — but only if you get to choose who — is sufficient to take absolute control of both the US congress and the Russian State Duma (or a supermajority of those two plus the Russian Federation Council), which gives them the freedom to pass arbitrary constitutional amendments… so your scenario includes "gets crowned King of the USA and Russia, 90% of the global nuclear arsenal is now their personal property" as something we don't care about.
Until it becomes 'who you exchange bytes most efficiently with" and all humans are at a disadvantage against a swarm of even bellow average intelligence AGI agents.
That's pretty much what we did. There's no super intelligent monkey in charge. As much as some have tried to pretend, material or otherwise. There's just billions of average intelligence monkeys and we overran all Earth's ecosystems in a matter of centuries. Which is neither trivial nor fully explained yet.
But yes, the corporations and swarm behavior are already doing poorly for our biosphere.
When humans developed, we did not displace an external entity that had created us and that had complete power to kill us all in an instant.
https://emilkirkegaard.dk/en/2023/02/there-is-no-iq-threshol...
1. When we are talking about wealth and power that actually can influence the quality of the lives of many other people, we are talking about way less than 0.01% of the population. Those people aren't covered in this survey, and even if they were it would be impossible to identify on an axis spanning 0-100%.
2. Your linked article talks about income. People with significant wealth and power frequently have ordinary or below-ordinary income, for tax reasons.
People who own high value assets (everything from land to the AI) will continue to own them and there will be no opportunities for people to earn their way up (because they can be replaced by AI).
"The logical endpoint is the subjugation or elimination of our species"
Possibly, but it would be by our species (those who own and control the AI) rather than by the AI.
I suppose then that they would become “homo sapiens sapiens sapiens” or some other similarly hubris laden label, and go on to abandon, dominate or subjugate the filthy hordes of mere Homo sapiens sapiens.
Google has a big investment in reducing AI bias (remember Gemini got slammed for being “too woke”). Altman is a big proponent of UBI. Etc.
Replace "AI" in all the doomsaying with "the internet," and it will become clearer.
Dario Amodei (Anthropic CEO, builder of Claude 3 Opus): "My chance that something goes, you know, really catastrophically wrong on the scale of human civilization, might be between 10 - 25%"
This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or wrong, every opinion is equally realistic as each other.
Like I could say that every AGI that becomes self aware will kill itself instantly because of boringness, and you cannot prove me wrong.
We just don’t know yet.
I don't know to what extent OpenAI and their compatriots are actually trying to bring forth artificial life (or at least, consciousness) forward, versus how much they're just banking on how cool AI is as a term in order to funnel even more money to themselves chasing the pipe dream of building it, and at this point, I don't care. Even the products they have made do not hold a candle to the things they claim to be trying to make, but they're more than happy to talk them up like there's a chance. And I suppose there is a chance, but I really struggle to see ChatGPT turning into skynet.
My response to a very similar statement is here:
https://news.ycombinator.com/item?id=39612641
> That's just... not how it's going to be done.
I agree. I think there are some critical ingredients missing. Obviously the weights need to be able to update to new data in a semi-online fashion, for example. But I think there are less ingredients missing than there were a decade ago, by a much larger factor than I expected at the time, and a decade from now, the number of missing ingredients might be zero.
My uncertainty on this is not because I think GPT is more than it seems,but because it's unclear how much of the amazing cognitive capability that humans have is, under the hood, much simpler than it seems.
I really want humanity to exist even ten decades from now; I want my kids to have grandkids. So I care about this even if I don't expect AGI from a 2025 Q1 product launch. And I don't think it's too early to worry about it, just like I don't think 1980 was too early to worry about climate change.
This is a vivid bit of rhetoric to underscore this point, but if you think about it for any length of time it starts to fall apart really, really quickly. The Wright brothers and the dozens if not hundreds of inventors forgotten who came before them drew upon the physics of what they observed in heavier than air flight to create the winged shape we know today that reliably causes lift, and then set about constructing it. That's not what OpenAI is doing. We still do not have a very solid understanding of where our own intelligence emerges from, apart from having particularly large brains relative to our body's size. So, to borrow your metaphor, it is indeed like a bird saying that there's no evidence to say that heavier than air flight is possible, because the bird lives in a world without atmosphere upon which to glide.
But when you look at, say, Sora's videos... How certain are you that there's no path between that and the human visual cortex? How certain are you that they aren't solving at least some of the same problems in structurally analogous ways? Given that nature built a visual cortex (more than once!) by applying survival pressure and turning a crank, just how hard can it be? When we apply a bunch of optimization pressure and billions of dollars turning cranks and something eerily similar pops out, that tells me that there's just less magic to our own brain than we thought. Like when spent centuries wondering if our solar system is unique, and then we finally put Kepler up there and it turns out that planets are everywhere you look.
Sure we do: we're made of atoms.
To reject that this necessarily shows that AGI is possible, you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Some people do in fact believe this. I think that if we can't build AGI even with a perfect copy of a brain, that would instead be a surprising proof the existence of souls.
> This is not intelligence of any sort, period, paragraph.
Let's run with that. Just using your definition of intelligence: given AI can already beat us in Chess, in Go, in Mathematical Olympiad puzzles, in protein folding predictions, in poker, at real world stock market analysis, in the game of Diplomacy, … — does it matter that they're not what you call intelligent?
Do submarines swim?
> bring forth artificial life (or at least, consciousness) forward
Why does it matter if it's "life" or "consciousness"?
“Deep Blue was intelligent the way your programmable alarm clock is intelligent. Not that losing to a $10 million alarm clock made me feel any better.” ― Garry Kasparov
> To reject that this necessarily shows that AGI is possible...
Incorrect. For something to be possible in an absolute sense, it has to be possible in both the physical and the metaphysical realm, and a lack of knowledge of blockers in the metaphysical does not mean that there are none (in a comprehensive sense), it only means it appears that way, due to (at least) our cultural beliefs ("truths") and conditioning.
> ...you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Could demonstrating that intelligence rests upon a virtual reality generation machine (that cloaks itself from being realized as such) or something else not also work, at least maybe?
Plenty of people have interesting or valuable opinions and insights on AGI, programming LLMs doesn't have any impact into that though.
It'd be like expecting the developers of friendster to have a good idea of what the impact of social media was going to be in 2020.
AI isn't AGI.
Feeling uncomfortable with this, will not change this sad reality.
I am indeed uncomfortable with someone saying that we can't worry about a gas leak until after someone strikes a match and notices an explosion or the lack thereof. If the sad reality is that we can't detect the gas leak any other way, I would argue that the precautionary principle suggests we ban the striking of matches until such sensors have been developed. But here we are in a room of people grinding up saltpetre with their mortars and pestles.
I have proven to you logically that they cannot have more knowledge then anyone else, but you continue to present their authority as knowledge.
Pls prove to me that they know more about the effects of AGI on humanity or that they have some magical knowledge we normal ppl are unable to comprehend.
If you cannot do this, then why are you bringing up these ppl?
You've asserted it, not proven it.
Given their entire research field is "how to make the thing", it's a fair bet that they know more than me — even if what they know is, to paraphrase Thomas Edison, "1000 ways to not make an AGI", that's still more than me, and I follow some of the research and read some of the papers.
Correct, and this (your counterpart's unrealized because it is the normal, proper way to think error) is what worries me... I think once again (as with affordable vehicles powered by fossil fuels) humans may be once again walking blindly and confidently into a situation where we've over invested in certain competencies to the detriment of others, increasing the already dangerous imbalance that exists between power and wisdom.
Considering how rapidly the technology is developing, and considering how clearly a threat a superior intelligence will be to the human race, it is absolutely the time to have this discussion NOW
I'm only asking for a standard that falls slightly short of "already having built Schrodinger's doomsday weapon and observed whether humanity dies as a result". You can set the bar wherever else you like, please!
The hour before the Kitty Hawk flight, the two most informed people in the world on the topic of heavier than air flight were a pair of bicycle retailers and manufacturers named Wilbur and Orville.
These people sure don't spot everything, but they're still way closer than the average human.
Actually, this is quite false. There were many people working on flight with massive credentials and massive funding. That the Wright brothers beat them is part of what makes the story of the Wright brothers so interesting.
That society didn't know this at the time is indeed relevant to AI development, and the analogy would be that the next big thing might come out of some minor research team that most people dismiss instead of the biggest and most well known groups… a bit like how Google's LaMDA was making headlines before ChatGPT surprised everyone.
It's certainly still possible today that some random individual has a crucial insight that gives them an edge over the big names, yet even those big names know far more than me.
Which is rather annoying, because I'd be interested in taking the opportunity of having been laid off to switch from iOS to AI. (If anyone reading this is hiring people in Berlin, contact details on my website. Plus side of hiring me: I'm relatively cheap in that field, and enthusiastic about the possibilities).
Consider this scenario:
- Ask ChatGPT to produce some output meeting certain criteria
- ChatGPT responds, but the output does not meet the criteria in one or more obvious ways
- Point this out and ask for a corrected version
- ChatGPT says "I apologize, here's the corrected version:" and spits out a nearly identical answer with the same obvious mistake.
- Repeat steps 3 and 4 ad nauseam.
Whatever sort of intelligence it may have, I would categorize it as not "general" enough to fall under AGI. Being able to answer AP test questions cannot be a sufficient measure of generality if a model can do it with flying colors but still fail at far more basic tasks.
You can ask ChatGPT what it means to be asked to correct a mistake, and it will give you a perfectly thorough and eloquent answer. But it is often unable to apply this concept to its own behavior. If asked, it can explain back to you exactly what correction you want it to make. You can even make it pledge to correct the mistake in exactly the manner that it just described. And then it will completely fail to do it. It reminds me of that "repeat after me" meme from Friends [1].
This makes me lean in the direction of "stochastic parrot" when I think about what LLMs are. As impressive as it is, ChatGPT demonstrably lacks a sense of self. It talks as if it understands that it's an agent in control of its behavior, but then fails to control or even recognize its own actions.
[1] https://en.meming.world/images/en/c/c2/Friends-_Repeat_After...
All AI, even narrow ones, necessarily have the ability to interpret novel questions in some fashion — how novel and how well they interpret being the "G" and the "I" in AGI.
ChatGPT was trained on hundreds of terabytes of information. I would hope that with all that effort it could answer AP test questions. That doesn't make it intelligent.
That it's able to spit out essays makes it something, intelligence is too loosely defined to describe it for everybody, so we need to develop new words and meanings and new language for what it does. It doesn't think, but it does dot products to tensors to get create its output.
It is interesting to consider AIs that way given that almost every AI will fail catastrophically when confronted with an area of expertise that is outside of their training corpus, although I guess Corvid tool use isn't exactly out of the animals' range of expertise.
Truly no offense meant, as I was deeply into the EA movement myself, and still consider myself one (in the original "donate money effectively" sense), but the movement has now morphed into a death cult obsessed by things like:
* OMG we're all going to die any time now (repeated every year since circa 2018)
* What is your pDoom? What are your timelines? (aka: what is your totally made up number that makes you feel like you're doing something rational/scientific)
I'm deep in the weeds w/ LLMs, e.g. I probably finetune an average of 1 model a day, and working with bleeding edge models... and AI safety just sounds so silly. Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when gradient descent was invented.
It's sadly twisted how mentioning that -- the majority of leaders doing the cutting edge research on AGI think it has a significant chance that it kills humanity -- is considered being part of a "cult" movement.
Your analogy is the same as early Intel engineers completely unaware that those chips would bring on the ramifications of social media. "In the weeds" and yet unable to foresee the trajectory and wider impact. Same with the physics that led to nuclear weapons.
> Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when [nuclear fission] was invented.
Exactly! As they should be. (for both Intel engineers developing chips, and physicists developing nuclear research)
There were a billion more potential dangers from those technologies that never materialized, and never will.
I'm glad we didn't stop them in their track because a poll of 10 leaders in the field thought they were too dangerous and progress should stop. (note that no one is against regulating dangerous uses of AI, e.g. autonomous weapons, chemical warfare; the problem is regulating AI research and development in the first place)
> We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. - Sam Altman (https://blog.samaltman.com/the-merge)
I would say it is.
I'm a neuroscientist and, man alive, we're no where close to being able to merge with machines. Like, do you have any idea how many diseases we could eradicate if we could modify neurons like that? Like, for real, 'curing death' would be step 9 or 10 on that 1000 step journey.
I hope you see how terribly uninformed such a take is then.
Yeah, I only quoted it to show how that other statement up-thread can't be taken seriously.
The only way I can square it is if it's a lie, or they're reassuring themselves whatever destruction comes will be fine as long as "there will be peace and security in [their] lifetime."
Beff Jezos (e/acc founder): "e/acc has no particular allegiance to the biological substrate for intelligence and life, in contrast to transhumanism
Parts of e/acc (e.g. Beff) consider ourselves post-humanists; in order to spread to the stars, the light of consciousness/intelligence will have to be transduced to non-biological substrates"
Beff Jezos / Verdon: "Personally, yes."
Yudkowsky is transhumanist in that he is hopeful people could voluntarily extend their biological self, but isn't advocating for the elimination of all biological selves in pursuit of other artificial intelligences.
Voluntary /= involuntary eradication
"This is extinction level important, so all you people who aren't us need to be careful meddling with the stuff we're meddling with for profit."