OpenAI Insider Estimates 70% Chance AI Will Destroy or Tragically Harm Humanity
futurism.com
futurism.com
"Good lord, what is happening in there? Life-changing AI implementations that could destroy all human life as we know it? Localized entirely within your laboratory?"
"Yes"
"May I see it?"
"No"
Of course this dog and pony show just got NVIDIA past $3 trillion, it's like crypto on steroids.
Just for reference, this is from nine years ago:
And this is from five years ago:
"You may not need hospitalization for respiratory problems for which there is pain or even full internal combustion. Breathing is particularly hard in the high-pitched high pitch, low pitched pitches. Air you inhale and exhale can cause a shortening in your breathing, so a ventilator is best. Doping can be administered on your own. If you fall, you must be at least 1 kilometer away from you to be considered for the testing. Your turn-of-the-seat's softerness may be a sign that something has gone wrong. Your depth should not exceed the mean of your seat height when wearing the seatbelt."
And for two... LLMs are also getting to be pretty old. BERT and GPT-2 are both more than 5 years old, and still have fundamental issues that we can't be sure are solvable. They lie confidently, they conflate facts and fiction, they create entirely made-up scenarios when asked about real-world happenings. It should be extremely alarming that the only successful AI strategy to-date has been scaling-up an already inefficient model.
AI will improve with time, but it seems entirely plausible to me that we've already hit the proverbial "bathtub curve" of progress. I genuinely cannot imagine what a "generational leap" in LLM technology would look like, outside of fixing the hallucination issues.
You sound like you have no clue that five years are the blink of an eye.
Cryptocurrency apologists also used this line of logic. "Yes, today the value of crypto is nothing... but imagine where it will be in five years!" Then we all wait 5 years, and cryptocurrency is still the victim of it's own mindset. Like cryptocurrency, I think LLMs have "technically" solved what they set out to do; generate readable text. But you need more than a solution in search of a problem; there isn't really that much demand for marginally truthful text in the same way completely decentralized currency isn't really necessary for the average well-meaning civilian. I'd even go further and argue that introducing AI into our daily lives requires you to replace something else, something that was probably more accurate and better-designed than the AI replacing it.
If 5 years is a "blink of an eye" for the industry, the entire field will be dead within 18 months. VC just moves that fast.
You clearly have some very specific models in mind. Even if the latest 4B and 8B models don’t move the needle on the “results you would champion” metric, this does not advance your argument that the state of the art hasn’t significantly progressed from 5 years ago.
> I would legitimately argue
I’ll bet you would!
Because this to you is "generate readable text"?
https://m.youtube.com/watch?v=MirzFk_DSiI
Sorry to be so direct, but you're in denial.
There's a reason why GPT-4o is not taking over YouTube and social media with it's incredible capabilities; nobody cares.
Well, if it encodes a world model, and can work on its own encoding, loop it into a critical revision of its contents and you have the Real Thing - something that is informed and reasons.
That a full world model is there is the preliminary condition. So, first you have to establish that an LLM actually draws a world model in its inner structure.
Then you have to refine that world model, and be able to refine its intentionally spawned branches (e.g. "Imagine scenario X"...). If the iteration of the refining is successful, you achieved the goal.
The point is implementing critical thinking; that the initial world model is flawed is the known starting point: at the beginning, it only knows it has "heard" a lot of information.
If it has no concrete reasoning, how is this looped revision intended to be more-accurate than a zero-shot guess?
I agree with the fundamental principle of not trusting input data from the start, but that just puts us back in square-one where we generate non-authoritative results dependent on a series of autoregressive parameters. It might generate better-reasoned results, but the outcome will be equally unreliable and prone to hallucination.
It can work if the reasoning (the logic) is part of the world model and is similarly refined.
Then, like in the normal case, you gather a good amount of information than draw your conclusions based on you logic and the "wisdom" built. Were this automated, you would have an automated champion in judgement - doing its best with the imperfect data and function available. Like us, just better.
We started a very long time ago and you have to give it time. The current turn of events seems to have turned its back on some very critical aspects - and still, give it time.
Impatience at this stage seems very ill-posed.
Although you probably meant that "LLMs show no signs to "emerge" (through upscaling) what they should become".
The sun begins to die 5 billion years from now, so is this an estimate there's a 70% chance someone will "tragically harm humanity" with AI in the next 5 billion years?
Or are they positing humans will survive the death of our sun, and concluding between now and infinity, someone somewhere will tragically harm someone with AI?
If this is true someone at some point between now and infinity should really get on this, perhaps we might estimate that around 2.5 billion years from now, we'll need to write some regulations.
“Does it involve creation of technology that doesn’t exist now?”
“Does it involve physically impossible nanotech?”
“Does it involve assuming the personal safety of every person with a any degree of control over it?”
In theory, you're right. If an LLM was given complete control over ballistic missiles, there is a nonzero chance that something could go horribly wrong. But this is where we take two steps back and realize that no rational country will give AI complete control over their weapons. It would be an extremely human-based design failure to hand over control of nuclear weapons to a statistics model designed to be wrong.
And to the extent they do, it’s replacing a system which otherwise wasn’t particularly concerning itself with civilian casualties on the other side.
It is another available piece of imposing power that can be used irresponsibly.
Wake me up when they go to D.C. and start working for the people at government wages. This isn’t part of OpenAI’s marketing, but it’s still self-aggrandizement.
No, he hasn’t. He started a non-profit. I can’t find his pay.
Also, the Wikipedia seems wrong. The Alignment Research Center is not a 501(c)(3) charity [1], they simply accept donations through one [2]. They directly claim their evaluations team, METR, is a 501(c)(3) [3], but due to the name being so close to “metro,” I’m having trouble verifying that. (They also launder at least smaller donations through the middleman charity [4].)
The closest Cristiano has come to public service is his role at the NIST, an appointment which prompted a revolt amongst staffers given his ties to effective altruism [[5].
All a bit ironic for a pair of nonprofits preaching transparency.
[1] https://apps.irs.gov/app/eos/
[2] https://www.every.org/alignment?donateTo=alignment#/donate/c...
[3] https://www.alignment.org/donate/
[5] https://web.archive.org/web/20240607060738/https://venturebe... having trouble loading the original Venture Beat article
https://www.nist.gov/people/paul-christiano
The Alignment Research Center is, in fact, a 501(c)3 charity, here is their Form 990:
https://projects.propublica.org/nonprofits/organizations/863...
METR is also a 501(c)3. It doesn't have a Form 990 because it's so new it hasn't filed a return yet, but here is the Guidestar page:
https://www.guidestar.org/Profile/99-1219864
every.org is just a generic non-profit payment processor. Lots of people use payment processors, because managing payments is annoying. Calling this "laundering" is like saying that accepting third-party payments through Stripe is some sort of illegal sinister plot.
It is. I’m saying the sole evidence of outcome I could find was his staff revolting.
> Alignment Research Center is, in fact, a 501(c)3 charity, here is their Form 990
Thank you, I stand corrected. (Christiano’s compensation is an admirable zero.)
> every.org is just a generic non-profit payment processor
Sure, and given ARC and METR are 501(c)(3)s, it’s fine. Charity to charity. If either weren’t, and were simply non-profits, it would mean a charity was lending its tax-exempt status to a non-charity. That would be sketchy. But as you’ve shown, it’s not what’s going on here.
Some more discussion last week: https://news.ycombinator.com/item?id=40579104
https://news.ycombinator.com/item?id=40576018
https://news.ycombinator.com/item?id=40574355
The letter site: https://righttowarn.ai/
[1] https://www.nytimes.com/2024/06/04/technology/openai-culture...
https://www.lesswrong.com/posts/EwyviSHWrQcvicsry/stop-talki...
If someone inside OpenAI had said something like "I predict there is a 70% chance our current corporate culture and leadership will harm or destroy humanity", that would be a much more compelling argument.
It's what you do with the technology that counts.
Because tech people want to build, are paid to build, and "it is difficult to get a man to understand something, when his salary depends on his not understanding it."
I guess any group can become the NRA when their controversial interest comes under scrutiny.
>Is there any technology you can't say that about?
Yes: all the technologies humanity has invented except for AI and maybe possibly nuclear bombs.
Or am I misinterpreting your question?
Look, it may well be something he believes, and he’s free to prognosticate (or market) however he likes, but I see absolutely nothing to support the number outside of his own opinion.
Besides, there’s no time limit on p(doom), so it’s completely unfalsifiable (“on a long enough timescale…”), and it’s about the destruction of humanity which means it’s unprovable as well. That, in my view, makes his 70% guess a sensational statement lacking scientific merit.
> There’s a [arbitrary number] percent chance that [technology] will destroy or catastrophically harm humanity
Try these: social media, the Internet, the large hadron collider, Starlink, Neuralink, iPhones, iDrones, quantum computers, regular computers, the 2038 bug, the Y2K bug, electric cars, gasoline cars, the great firewall of China, the not so great firewalls of asbestos, mRNA technology, gain of function research, nuclear bombs, nuclear energy, paper clip manufacturers, scissors.
I’m not saying it’s true that these have a 70 percent chance of destroying or catastrophically harming humanity, but couldn’t you make the argument?
Unless I'm mistaken, even GenAI hasn't really changed anything worthwhile. Most of the changes I see are a) most of the scummy and creepy internet ads are now AI generated b) endless spam is now AI generated and c) the latest round of tech layoffs used GenAI as an excuse. Has there been an actual revolution I've missed?
...and so, if GenAI hasn't lived up to the hype, how am I supposed to believe that this industry will make actual existing AGI?
The open letter this article is based on still cites, "existential risk."
Define what AGI is. Show us evidence the technology is on track to implement it.
This whole "AI safety" scene is becoming a circus for conspiracy theories and speculation.
As far as I can tell we're reaching the point of diminishing returns in training LLMs... which is already rather destructive.
What's telling is that we don't see similar "safety" committees there trying to protect water in distressed ecosystems, regulate building/zoning of data-centres, energy usage; protecting workers used in alignment and training; protecting workers displaced by the technology and poor policies, etc.
Source is actually here: https://www.lesswrong.com/posts/xDkdR6JcQsCdnFpaQ/adumbratio...
Did you try turning it off and back on again?
"Oh, but how will we ever get to the stars without it?"
Do you know what type-III civilizations don't appreciate? Stupidity.
Now imagine that AI has access to anybody's search history, plus that person is using a trackable phone, who's to say that a small FPV drone cannot be sent to get the job done? The way it's done in Ukraine?
I think we as humanity are facing mortal danger RIGHT NOW, not in the future. AI is killing humans on almost industrial. Especially because, well, some people might argue that's the entire goal. So AI is making it much easier.
Mass extinction through technology has been a known reality since decades. The technology remains the enabler, the responsible remains the human actor.
That's an awful lot like "it is difficult to get a man to understand something, when his salary depends on his not understanding it." Some tech people just want to keep building, consequences be damned. Mentally shifting the responsibility one step further down the road is just a way for them to dodge responsibility.
It's worth remembering the developer is also "human actor." He's still responsible if he builds a dangerous technology, and leaves it for someone else to push the button.
The atomic bomb? But Nazis had the V2.
The videogame? But the creator rewarded the creative, entertainment and possibly artistic results more than the idea of collective time of addiction, or maybe thought "better than chemicals"...
The axe? It is for chopping trees. Dynamite? Mines. Death ray? To defend yourself from wolves etc.
Sure there will be cases in which someone develops something inherently and only harmful, but it is not the general rule.
Don't reason about the safety of potential future technology from the safety of past technology, generally. That gets you into nonsense like "my highly contagious bio-weapon won't destroy civilization because the axe didn't." You've got to look at the particularly characteristics of the new technology, with a cold and realistic understanding of human nature.
Still, it is pretty uncommon in general to develop a «highly contagious bio-weapon»: when it happens - and it happens - the destructive context is clear. But there are also very many cases in which the productive or destructive adoption of an otherwise neutral technology is critical. The progresses enabled by Yann LeCun: recognize targets, or thieves, or citizens, or tumors...