re your last sentence: why do you disapprove of funding research into AI safety
When most EA/Rationalist folks discuss AI Safety. They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible
Now there’s nothing inherently wrong with funding research of that sort.
The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today.
This existential measuring contest is just so odd and unnecessary. I really don’t care if you think AI Safety is a legitimate problem. I personally don’t but it’s when they begin to downplay others actual catastrophes that many of us take issue.
Have you seen anything to indicate otherwise? The IPCC reports don't contain even a whiff in that direction.
To the best of my knowledge zero people who know anything about climate change think its existential.
Is it:
1. All life on the planet dies
2. All advanced life on the planet dies?
3. All humans die?
4. Advanced civilization is destroyed irretrievably?
5. Advanced civilization is destroyed for a long period of time?
etc
And I'm not sure the group "people who know anything about climate change" are neccesarily better equipped to answer this question - it's a complex systems question.
I'm not actually sure who is equipped to answer it accurately.
"Millions to billions of people die; large percentages of the rest are displaced and/or have their lives made significantly shorter, harsher, and less certain as war, famine, disease, and natural disasters become vastly more common around the globe."
And there's really very little question that that's where climate change is leading if we don't get it under control soon.
With dangerous research, when a conclusive demonstration of possibility is achieved, it is usually too late to do much in the way of prevention. That's why we don't expect the FDA to "prove" that harm from a new medication "is even possible" before regulating it; rather, we expect the pharmaceutical company to provide evidence for its expected harmlessness before we even make the attempt. Think of the FDA what you like, this is certainly a better standard than putting the burden of proof on harmful outcomes.
From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge.
This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.
My take on it is the other way around: every step of the argument seems plainly obvious.
So either it's less obvious than you're claiming, or you are taking the threat of creating an evil AGI not seriously enough according to your own beliefs.
I just don't care that much about things that kill everyone. I don't think almost anyone does. In the Cold War, the vast majority of the population was not doing absolutely everything to achieve bilateral disarmament. This would lead you to believe that they didn't really expect nuclear war. But maybe they were just apathetic and helpless.
This isn't an accurate representation, really. The letter was written exactly because it was clear that a fission bomb is very likely within the short-term reach of current technological and industrial capability. This is very much not the case with AGI.
The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.
Supporting your argument with an inaccurate thing is not 'a difference of degrees'. It's a difference between a good argument and a bad one.
My argument is that for contested and dangerous outcomes, proof of possibility is an unreasonable standard, in good part because it massively underspecifies the goalposts.
It's not, it's just a turn of phrase you've latched on to. There's no discussion of a standard of proof, etc. The comparison between the Szilard Einstein letter and 'AGI safety' is specious and it's one you brought up! You have to outright misrepresent what the letter was about just to make it. It was not a letter about a 'hypothetical danger they had no way of proving'. There just isn't any reasonable reading of the letter, the context, the history in which that's an accurate statement.
By any reasonable standard, at that point the possibility of nuclear fission was well established. But the argument is that there is no central committee that defines the reasonable standard and checks theories against it. It's all just opinion, and that specific opinion can be arbitrarily goalshifted.
I'm not saying the letter was about a hypothetical danger they had no way of proving, I don't think that. But that's specific to my interpretation of those terms, and I could easily see the exact same argument levered against the letter if that debate had been done in public.
“In a high-emissions scenario where little is done to curb planet-heating gases, global mortality rates will be raised by 73 deaths per 100,000 people by the end of the century”
https://www.theguardian.com/us-news/2020/aug/04/rising-globa...
That's an assessment based on what we know, but there's a lot we don't. For instance, it's possible that climate change could lead to a large scale collapse of food chains that could pose an existential risk to humanity.
It's also plausible that climate change will lead to resource shortages that result in wars between nuclear powers, and the nuclear calamity that follows kills us all.
I'm sure some people do this, and it's bad.
But to be clear, when you say some researchers are actively ignoring how to reduce harms of AI today... I'm not sure what you mean by "actively" ignore, but don't most researchers just by default ignore this, except for the few who are active in this field?
In other words, I think it's totally legit to focus on long-term AGI risk, and totally ignore current short-term AI risk, as a personal career-choice, just like it's ok to not even work on AI at all. Unless someone is actively trying to cause less resources to go into short-term AI risk, what's the problem?
> They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible
I mean, we exist as (one would hope) intelligent beings. Why wouldn't AGI be possible? Your frame makes it seem like the default is that it isn't possible, which is wrong IMO.
I really don't understand AGI skeptics. Unless you think reality has some non-physical character that can't be reproduced outside of biology, it obviously follows that a physical machine can reproduce what another physical machine (human body) is doing.
Right now what's threatening the most people is war, famine and diseases. Climate change is a tangible future threat but I'd bet a couple $ that the JSO organizers are full EA proponents and are flying private jets to convince people to throw soup into works of art
Like increased rate of hurricanes due to climate change?
One does need to seriously consider low probability, high impact events, like pandemics, in one's future planning and resource allocation.
This is what I'm replying to. And yes Climate Change is a worrying problem, but it won't be solved by people throwing tantrums in museums
This is a bad analogy. The mechanisms and risks of climate change are reasonably well understood as are the solutions available to us now and even some possible future solutions.
When it comes to AGI, what's the risk? Something bad might happen? The mechanisms of AGI arising is also unknown since it has never happened before and it's not even known if it's possible. Finally, what's the solution? Prevent AGI research? Make sure every AGI project has a killswitch? How do you prevent something when you don't know if it is possible nor how it will arise?
Compare this to the concrete, measurable things we can do to slow down climate change and "AGI safety" seems like a make-work money pit.
Ergo at this point in time money spent to pay people to contemplate it is almost certainly entirely wasted.
Related to the topics of AI apocalypse and putting things off because they seem far away: I personally find it amusing to contemplate a strong AGI being developed in 2035 and wiping out humanity. Why? Because it would eliminate the Y2038 problem, retroactively vindicating all the engineers who only allocated 32 bits to timestamps. It would mean the justification of "oh, it's so far off, the world will have changed unimaginably by then", would for once turn out to be correct!
Its not at all clear that future actual AI will be GPT7 now with 10,000x as much processing power. In fact not much seems to be clear at all. It would seem that the insight to control future tools will come from the experience in building said tools with little chance that it will just accidentally suddenly become skynet. If we doom ourselves thus it will be a long laborious process with years of striving, setbacks, and thousands of people involved.
We should focus on the basic science and let it become clear what avenues exist over the probable decades it will take to even reach the vicinity of our goal. Money spent specifically on paying PhDs to imagine how to secure a technology we don't have and don't remotely understand is probably wasted.
The possible outcomes here are "probably we waste a lot of money" and "the world ends."