A.I. could lead to a ‘nuclear-level catastrophe’ according to researchers
fortune.com
fortune.com
It's actually the result of a survey of a sample of NLP researchers, of whom 36% said they "agree" or "weakly agree" that AI could lead to a nuclear-level catastrophe. The exact question is:
>It is plausible that decisions made by AI or machine learning systems could cause a catastrophe this century that is at least as bad as an all-out nuclear war.
The article presents this as:
>Many experts in A.I. and computer science say the technology is likely a watershed moment for human society. But 36% don’t mean that as a positive, warning that decisions made by A.I. could lead to “nuclear-level catastrophe,”
This is a very misleading presentation. The 36% are not saying it's "likely", they're saying it's "plausible". They're not issuing a "warning", they're answering a survey question.
And if productivity increases that means it's worth to hire more people. Look at bank tellers after the advent of the ATM. There are more bank tellers today than before the ATM.
> The number of human bank tellers in the United States increased from approximately 300,000 in 1970 to approximately 600,000 in 2010. Counter-intuitively, a contributing factor may be the introduction of automated teller machines. ATMs let a branch operate with fewer tellers, making it cheaper for banks to open more branches. This likely resulted in more tellers being hired to handle non-automated tasks, but further automation and online banking may reverse this increase.
Doom from technology is always around the corner but we just never happen to get there.
Excel made it possible to eliminate the manual parts of certain tasks that humans are bad at, and generally uninterested in doing.
AGI removes the need for a human entirely.
These considerations are not similar in substance or scope, and applying old models of understanding technology advancement is a dangerous game when exploring the potential realities of AGI.
It’d be better to frame AGI conversations as if we had just discovered an alien species but had not yet made first contact.
Assuming our understanding of biology on earth is sufficient to predict the outcomes of first contact would be similarly dangerous.
> Do you think you'll be able to prompt AGI "do business things" and it'll just carry on in its virtual world that will somehow translate to real value?
Simply: yes. And much more.
This is the general concern about AGI in a nutshell.
If an alien ship shows up tomorrow, and its occupants express a desire to coexist/cooperate, and it becomes clear that they are significantly smarter than we are, what concerns come to mind?
What if suddenly every company in existence is angling to benefit from this higher intelligence, and wants to proceed with full inter-species integration/collaboration full speed ahead, without fully understanding the goals of this other species?
What if we later learned that this species prefers extremely hot climates and thrives on high CO2/low oxygen environments? Would we be so willing to participate in a short term business optimization initiative?
What if we also learned that this species is remarkably similar to humans in that it will compete for various forms and degrees of power while ignoring the long term ramifications?
Given what we know about our own species, the prospect that a super-intelligence might be built by and based on it should be terrifying.
How do you get from that to aliens controlling our climate?
Give me one reasonable path. Will the numbers start changing themselves and pop out of the computer and float around and start moving things?
This AGI doomsday nonsense is becoming more of a religion every day.
Actual AGI appears, and it has the ability to improve its source code. It becomes more intelligent than humanity and is misaligned to human goals.
These goals will the mathematical representation of what the AI is trying to optimize, and there are many ways for this optimization to cause negative effects.
We haven't even reached AGI, yet the LLM GPT4 was already able to act in the real world by manipulating a Task Rabbit Worker.
The goal of the alien analogy is to provide a thought experiment that allows us to expand how we think about AGI without getting stuck in preconceptions.
It is not literally alien in the sense that most people think about when contemplating extraterrestrial life, but there are some strong parallels that are informative to consider.
> It doesn't have autonomy, will or desires
Current-generation AI tech is already teaching itself how to walk [0]. This is not a robot that follows deterministic rules pre-defined by a human, or a robot that was trained to walk. The robot trained itself.
The AGI conversation is a conversation about the future where each of the specialized fields of AI converge. Much like we are massively complex interdependent systems, I suspect AGI will be the culmination of integrating focused fields of AI technology into a single system.
Will and desire are human feelings that ultimately stem from our goal-oriented behavior. The absence of these feelings in a robot do not matter if the robot behaves in a goal-oriented way. Problems emerge when its goals are not aligned with our goals. This isn't about some hokey robot overlord scenario, but one where a robot sees humans much like humans see ants when going about our days.
Most humans don't go out of their way to destroy ant colonies, but neither are we very concerned when we run the lawn mower over one. Malice is not required for a misalignment of goals to be a serious issue.
> This AGI doomsday nonsense is becoming more of a religion every day.
None of what I've described is inevitable, but all of it is possible. We're now in the moment where we'll decide what is next, and if we'll take this seriously enough to bake in appropriate safeguards. I think Stuart Russell has important things to say about this [1].
While I'm not a doomer, I'd strongly encourage you to dig deeper into this subject. I was shrugging most of this off for awhile, but things have changed.
- [0] https://vcresearch.berkeley.edu/news/step-step-berkeley-robo...
The excel “aliens” are significantly smarter than humans when it comes to basic organization and math, but they still require humans to generate value.
Similarly, AGI is much smarter for a set of tasks, but it still requires humans to generate value. LLMs are astounding/fascinating in what they are able to do, but they don’t replace employees end-to-end (except for a few niche industries).
Better tools will be made with AGI, and the outcomes will better society. For a long time now, humans have required computers/AI to compete in business, and this will just be extended by the new AGI. Humans+AI will always be better than either alone.
What you're describing sounds like an early iteration of AGI. But unless we stop the research, progress will continue and the limited AGI you describe is not where things will stop.
LLMs are not AGI, and their limits are unsurprising as such. But the limits of LLMs are not informative about the broader concerns of an eventual AGI.
In each of your descriptions you've limited the scope of what an AGI can accomplish, but those limitations need to be justified. If we continue to iterate, and especially if there's ever a moment where the computers surpass our own understanding, there is no reason to believe that those limits you imply are absolute.
It’s a tale as old as the semiconductor really. Ray Kurzweil’s “The Singularity is Near” only with a different spin that incorporates the latest tech.
I do, however, personally believe that computers won’t ever surpass the Human+computer combo. That is to say that computers will never be so dominant that our involvement is useless.
Pontificating about the future wasn’t the point of what I was saying though, rather that the current state of AI (even with the incoming few years of generative AI applications) is a far stride behind the current Human+computer combination.
What makes you confident in this belief?
I do, however, believe it. The current state of AI is so far behind the human mind that I think it’s reasonable to say it will never surpass the Human+computer combo.
It’s like talking about humans traveling outside of our galaxy. I guess, theoretically, it’s possible or technically even likely with a large enough amount of time. Theorizing that far into the future is meaningless though, as far as I’m concerned.
Why do you assume this would be the case? Why would companies increase pay?
> And if productivity increases that means it's worth to hire more people.
No, it means the opposite. Companies can reduce payroll expenses without incurring a decrease in productivity.
it's not clear to me that your bank branch example shows that automation increased employment. It seems more likely that as populations have grown, there was a corresponding increase in demand for bank branches.
But that's just speculation either way. Do we have a good way of determining causality here?
It's pretty basic economics that says wages are based on productivity. If you were hiring would you pay more for a 10x engineer?
If the current employer wouldn't pay you more you could leave as someone else would pay you since you're more productive
Its self evident pretty basic economics. Disputing that you'd have to reinvent a lot of basic economics
Dev tools made programmers many times more productive. Do employees just stop hiring as many programmers?
This is a nice idea that plays to the human sense of fairness, but wages have been completely decoupled from productivity for about the last four decades[1].
The only thing that really raises wages is the market power of laborers. Wages have stagnated or fallen as the market power of labor has eroded.
If you had a goose that lays golden eggs that you can sell for $100 net of costs. You'd pay up to $100 for the goose. If suddenly it started producing double the eggs you'd pay up to $200 or someone else would outbid you. The only way the goose wouldn't be able to get a higher salary is if competitors were not greedy as it's basically free money and just let the current owner profit from the goose's productivity gain
This makes a lot of assumptions about the market being perfectly competitive, which is a bit of a stretch. The past four decades have been marked by consolidation and reduction of competition, and the largest firms have profit margins that aren't at all indicative of healthy competition.
Think about an operator of a machine in a factory. If the machine is 10x more productive than the operator on its own, will the operator get paid 10x more? I don't think so, because the competing companies also have 10x the productivity. Prices of produced goods will drop but that will not lead to 10x more consumption.
1) They're hiring a contractor for 1/10th the hours, and 2) The contractor hasn't increased the per-hour fee to match the increased productivity
I mean, sure, it can work out. But it's hard to see how it would do so in enough numbers to actually matter much.
> Which is exactly why we have things like WordPress developers for small businesses
Right, this is a solved problem. Businesses who don't need a full-time dev hires a contractor. That's how it's always worked. But contractors are expensive.
So, either the contractors would reduce their fees (thus falsifying the assertion that AI would increase their pay rate) or those businesses will be paying the same as they've always paid but perhaps getting the finished work sooner than they otherwise would have (thus falsifying the assertion that AI would make this more affordable).
Productivity gains can absolutely reduce costs and increase profits, but they do so by getting more work out of people while paying them the same as always.
Certainly not if there was some tool that made every engineer a 10x engineer. In effect, it would mean that they're all 1x engineers, but that the "x" is ten times greater.
> Its self evident pretty basic economics.
Basic economics says that businesses try to eliminate as much expense as possible. So no business is going to pay their employees more just because they're more productive. All it does is reset the baseline for what is considered the minimum acceptable level of productivity. Salaries won't increase because of that.
> Dev tools made programmers many times more productive. Do employees just stop hiring as many programmers?
Yes, that's exactly what has happened historically. If a tool makes programmers twice as productive, then the business needs half as many programmers. Some businesses may be able to increase profit by doubling the number of products, thus needing to maintain the same amount of programmers -- but that's not the norm.
Companies gladly outsource to countries where developer wages are lower -- and that's with issues like time zones and language barriers. What if you could outsource to an AI that works whatever hours you want, even around the clock, and speaks perfectly in your native language?
> Dev tools made programmers many times more productive. Do employees just stop hiring as many programmers?
If programmers were less efficient, companies would probably have to hire more of them, and more of other sorts of people too. Or they would hire the same amount, but just be forced to accept slower progress.
Vox describes the initial flaws in Schmidt's reasoning here: https://www.vox.com/2017/5/8/15584268/eric-schmidt-alphabet-... (this is also the source behind that Wikipedia article)
Statistics from the US Beauteau of Labor Statistics (https://www.bls.gov/ooh/office-and-administrative-support/te...) proves Vox's assessment was correct: there are now only 378,000 tellers in 2021 and is forecasted to continue to decrease. For context, the US population grew 68% over the same period suggesting that as a proportion of available jobs the difference is even more stark.
Or broaden it out and ask what happened to job opportunities in banks or finance in general. There has been huge growth in productivity and hence headcount growth. Same is true for engineers. With modern dev tools and eng is a lot more productive and hence more profitable.
And the ATM was invented 50 years ago, so even if bank tellers are on the decline today i wouldn't call that short lived
https://www.aei.org/economics/what-atms-bank-tellers-rise-ro...
Looking at the graphs shown, it is fairly clear that tellers were increasing from 1970-1980 at a strong rate and then it more or less levelled off. ATMs grew at a very similar rate from 1990-2000. It seems reasonable that if ATMs were not as successful as they were, the teller growth would have been higher and therefore ATMs have "stolen" at least some teller jobs growth potential. Absent a technological efficiency increase, the base demand of US population growth would have created more need for human provided services.
In the long term the truth is laid bare: in the AEI article (2016) it says "employment of tellers is projected to decline 8% over the next decade". That would have left us with ~543k tellers -- instead we have 378k, a drop of 35%. When the tailwind stops, we will often see capital choose technology over humans and often at higher rates than projected.
A parting concern: the rate of technological change and capability of industries to swap inputs was impressive in 1970-2000 but I hope you will agree it is nothing compared to the pace of change now. There is every reason to believe the trends, short and long term, we saw with ATM:tellers is going to repeat faster in the future.
Today's ATM's can do anything a teller can, albeit maybe a mortgage?
If, like tellers, the initial effect of AI is to increase the demand for developers and then taper off, this could be ideal. The initial surge in demand will give many a chance to increase their knowledge of AI techniques and in about a decade when AI is everywhere, demand for all knowledge work will start to drop. The good thing about this is knowledge and experience about AI has spread, it is for all to use (hopefully it works reasonably well on a desktop computer).
Then the real shakeup begins, but at least the means of production -- or rather, the means of AI -- has spread and isn't completely siloed in one or two companies.
In America in particular, there isn't enough income or jobs in the low/no skill area to survive. Suicide and deaths of despair rates amongst middle-aged and older in this demographic are astronomical.
In the relatively near future, it won't be necessary to have as many software engineers to run a very large company.
The lives lost from the two nuclear bombs dropped on Japan? The environmental cost of several above-ground and underwater nuclear tests over the years? The consequences of various nuclear reactor leaks/meltdowns?
"Nuclear-level" is ridiculous hyperbole. Did an AI write that title? ;)
No, worse...
Not scared about global warming, not scared about war and countries bravely stocking up on most deadly weapons, not scared about capitalism eating itself to the bone. Future potential AGI is what they choose to be scared about because suddenly producing words got a bit cheaper.
The rest of your post aside, people concerned about AI safety tend to also be concerned about nuclear weapons.
Eg:
- https://asteriskmag.com/issues/1/the-illogic-of-nuclear-esca...
- https://www.lesswrong.com/posts/FGTgeweYNxmMBx4fz/existentia...
* Global warming is going to be painful, but seems like we're on a track that doesn't involve earth becoming completely uninhabitable. I'm still concerned about it.
* Companies have a much stronger incentive to make AGI than deadly weapons. It will be incredibly profitable.
* We have zero historical examples of a species developing a technology that is smarter than it is to look back at and establish some sort of probability for how this will go.
* All the issues with capitalism, where we have this system that isn't completely aligned with the well-being of humans, though it has kinda worked in the past -- now imagine that it's turned up to 1000 times the power. If you're worried about capitalism eating itself to the bone, that's the same thing to be worried about with AI.
To say that "producing words got a bit cheaper" really trivializes what a massive impact AGI will have, and how soon it's coming. I personally think that the probability of humanity being ruined or destroyed is probably either close to 0%, or close to 100%, and we don't know which, or if it's bad, how long we have. Even though 100% end seems more likely to me, I still think it's worth trying really hard to get to a good future.
If we had knowledge of this happening to several hundred different alien species, and could see the distribution of outcomes, it would be a lot easier to plan for this. In the absence of such evidence, I think it's a mistake to try to over-generalize from challenges humanity has successfully faced in the past, to "creating a new form of superior intelligence".
Are you sure? Various countries are throwing hundreds of billions of dollars on the table of weapons manufacturers. I think this market grew recently way faster than AI.
> We have zero historical examples of a species developing a technology that is smarter than it is to look back at and establish some sort of probability for how this will go.
We already have so many superhuman technologies under our belt that allow organizations, which I could argue are a form of soulless AIs to compete with interests of whole societies. I don't think neural AI will come to beat our organizations at their game just because if you pour into it significant fraction of our total computing resources might be slightly smarter than a single human. Our bureaucracies surpassed a single human with their cunning, and self-preservation and crushing power centuries ago. We already live in the world made by AIs for AIs. They are just made of humans not silicon.
> To say that "producing words got a bit cheaper" really trivializes what a massive impact AGI will have, and how soon it's coming.
I am not convinced it will be more unique than internet or printing press. I believe AIs will pressure us into changing some things just as any relevant technology have done before. Internet came with its own set of SF scares of hackers taking over entire world or brains fried by man machine interfaces that were supposed to be just around the corner. We lived through so many technological revolutions that impacted our way of life and work. I don't think AIs or even AGIs developed from what we have now will be much different. I think the example of self-driving cars shows us that pace of progress going forward will be more disappointing than anything else.
Another thing is that people vastly overvalue intelligence. Tiger might be less intelligent than human but it will still tear a single unaided human to shreds. People concerned about AIs should focus less on how intelligent it is but instead of how large physical claws are we willing to equip it with.
This. But also, if it were smarter than us (or even if people believed that!), then it could program people to be its claws. I don't want to give it physical control of weapons, or even the means of production, but I also don't want it to manipulate peoples' minds.