I know people at METR and Anthropic.
When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.
When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.
Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.
Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
The people still there necessarily think they can make a difference.
I never applied to any of them because I didn't think I could make a difference.
I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone.
I don't expect it to be an important idea.
> Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before.
My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number.
It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important.
When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles.
What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively?
If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
HAH
I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
And how does that relate to the ask for oligopoly licensing within global democracy?
“We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
I would call it more of a self-selecting one. Anthropic is basically hiring people with that mentality. I'm pretty sure that most of them do sincerely believe it, too. I'm skeptical about Dario himself though. The man had an opportunity to show moral backbone, and failed to do so; why should I trust him on that again?
> And how does that relate to the ask for oligopoly licensing within global democracy?
They are basically saying that they'll stop if everybody else does, which requires some kind of global enforcement mechanism.
> “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
The difference between nuclear bomb and AGI (as understood by the likes of Anthropic) is that the latter triggers the technological singularity that renders any runner-ups moot. That is, so long as AGI is developed, we're going to get our robot overlords either way, but whoever gets there first gets to define their ethical system. If that is one's perspective, and if one sincerely believes that they are the only ones who can do it right, it's a coherent argument. It's just that the premises are very arrogant.
Your analogy with nukes actually works better for the position that AI development needs to be unconstrained because otherwise we'll lose the arms race to China. That is basically a repeat of https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_lette.... I honestly don't know where I am on this. Realistically, if AI is indeed a power multiplier - and with all the recent security stuff it's hard to not see it that way - then an arms race feels inevitable, especially given the current worldwide political situation. I could believe in sincere international cooperation on this back in 1990s, but there's way too much saber rattling all around for it to work (and note that this goes both ways, i.e. China can similarly not be certain that US isn't secretly developing more powerful AI even if we do publicly announce a freeze).
That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
Selection effect.
Everyone who thinks "the biggest difference I can make is staying in/joining/founding new AI research lab" does that.
Everyone who thinks "the biggest difference I can make is leaving/whistleblowing", does that.
Treating both groups as the same by virtue of employer is the goomba fallacy.
Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator is a necessary step to safely prevent anyone else making unsafe AI.
Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
What happens when everyone else (who is not so careful) gets to the same threshold three months later? How does them getting there first stop that happening?
It's an utterly self-serving argument and it's not even internally consistent.
Megalomania is not evidence of either a problem or a solution.
Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this.
God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture.
This is exactly the opposite of what Dario proposes.
Are you likeminded?
In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately:
there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry.
With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice.
And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.
They're not proposing anything concrete, and when they do, what do you think the proposal will be? Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies? Or are their proposals going to be aimed at everyone running open Chinese models?
> Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies?
This is compatible with the language being used, but I suspect they're not going to do that.
And if a threat to the human race does come from AI, it's going to come from OpenAI/Anthropic. Hypercapitalist, secretive, in bed with the government, plus multiple real documented hackings of open source infrastructure already.
We're a hell of a lot safer with China doing the same research out in the open and making it available to anyone. The choice might well be: one or two superintelligent autonomous AIs at OpenAI/Anthropic - or a lot of smaller ones, unable to be controlled but also coming out of a diverse set of environments.
One of those leads to a stable ecosystem where we can all coexist, the other is genuinely terrifying. But make no mistake, from OpenAI/Anthropic this is all motivated by their stock price - when you're in the silicon valley mindset, it distorts your reality. They've convinced themselves that everyone's safer if they stay on top and in control, conveniently ignoring how that benefits them, and I don't believe them for a minute.
The oil corporations were publicly claiming to support carbon taxes, while also secretly fighting actual implementations of carbon taxes.
All the communist/hippy stuff was done by people a couple of steps removed from the actual companies with obscure money trails. The official statements were much more sophisticated propaganda that if you weren't paying attention to who they were paying in the background would make them seem reasonable stewards of the climate transition.
Yet here we are talking about some vague "trust me bro" instead and you making some vague insinuation of climate change denial.
But that's an aside. Do you think we should treat them as liars or threats?
I answered that with the analogy you called "spin" and "avoiding the question", and completely misunderstood because "vague insinuation of climate change denial" is almost the exact opposite of my point ("what if the oil companies were screaming from the rooftops about the problem" is as far from denial as you can get).
Threats. Like they claim to be.