The illusion of AI’s existential risk
noemamag.com
noemamag.com
Speaking of… one scene in the book has humans communicating with an alien that appears to have “learned” human speech by training a non-sentient LLM on human ship-to-ship transmissions, and then “fine tuned” it to achieve the desired communication goal without ever actually understanding what the LLM is saying.
This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s! That’s proper science fiction, right there.
Back to the economic aspect: Several characters in the book had to “butcher themselves” with implants and enhancements to remain economically relevant in the age of AIs. It’s that… or you’re packed away in storage. Useless.
PS: Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”. You could use this to scare whales away from ships without understanding what specifically the LLM was singing to them!
I'm a huge fan of Peter Watts' work, but to be fair that is at the very least a close-cousin to the Chinese Room thought experiment that's been around for a while.
The details might differ, but it's worth thinking about whether the implications make sense. Watt's certainly asks a lot of good questions which other orders have also asked - i.e. Accelerando[1] is very interested in the idea of "what happens to mankind when economic systems can no longer be participated in by recognizably human minds?" - taken out to the limits of known physics.
[1] https://www.antipope.org/charlie/blog-static/fiction/acceler...
^^^ You read the full-text here, I highly recommend it.
The concept wasn’t new in 2006 either.
Not really seeing the “now that’s sci fi!” emotion here.
Most of our historical compute problems were coupled to network reliability, performance. Those bottlenecks are largely solved; oh wow explosion in AI progress.
This isn’t new it’s more like the Higgs; just needed to wait for hardware.
What’s interesting in the culture series in light of the AI alignment debates is that the culture aligns every mind to its task at “birth”. A picket ship, falling outside the normal moral constraints, is a very different set of personalities and morals than a GSV such as wisdom like silence
An interesting aspect of the Culture Minds is that they're described as hybrids of human-like / neural-net / organic intelligence and also direct numerical compute and storage capacity. They can both "think" and "compute", something few if any current AI models attempt.
A well-known limitation of current LLMs like ChatGPT is that they are atrocious at even basic arithmetic, whereas a $2 pocket calculator from the 1980s outperforms them in this respect!
I'd love to see an attempt at a hybrid architecture where the neurons are also connected to a numeric compute component of the architecture.
Think of parsing an expression such as:
let foo = 5 + 10
"let", " foo", " =", " 5", " +", " 10"
[1309, 22944, 796, 642, 1343, 838]
[LET, ID("foo"), EQ, NUM(5), PLUS, NUM(10)]
What if we lifted this restriction and made an AI embedding where every token had both an ID and a value? For literal text just set the ID to NaN or whatever.
E.g.: assume that "integer constant" is ID 1000, then an LLM might be fed this input instead:
[1309, 22944, 796, 1000, 1343, 1000]
[ NaN, NaN, NaN, 5, NaN, 10]
Could be interesting...
As a layman with respect to LLMs, I feel like Blindsight and its perspective of the chinese room puzzle is required reading. I wonder if people working in AI feel the same and I'd be really curious to know how they feel about the book.
I don't think it is described as non-sentient in the book, but only as lacking consciousness.
Consciousness not being required for sentience is one of the two major themes of the book (the other being that consciousness is an evolutionary dead end).
Still your example is very relevant. I very much believe that ChatGPT, for example, is not conscious. The question is if it has some amount of sentience.
Sentience is a power of consciousness which is a power of the body which is a power of the cells of the body.
There is no dualism other than in the imagination: one could imagine some fuzzy distant "forest" without thereby imagining trees -- so much for the imagination.
Similar to the book Solaris (there is a new English translation, and I recommend both the US and Russian movies).
Apologies, I have a bad habit of mixing up those words, despite supposedly knowing better... at a conscious level.
Huh.
I wonder if I write some things subconsciously?
Decoding animal communication using machine learning is a popular research topic. See the current issue of Science, for example: Using machine learning to decode animal communication (https://www.science.org/doi/10.1126/science.adg7314)
But don’t worry, TED Talk attendee. The obsoletariat mostly won’t be Americans. The median member of this class will probably be Chinese or Indian. So you can continue your performative concerns about Roko’s Basillisk or whatever topic of distant, paralyzing uncertainty is overflowing from the next room into mine.
This goes back all the way to the Luddites, who weren't actually anti-technology/progress. They were opposed to how its benefits were captured and by whom.
PvP - someone will use it against me
PvE - someone will do it better than me
and EvP - it will take something away from our human experience
None of those really capture the weight of tradition though. Loss of control is a hard pill to swallow. Feels better to keep on doing the old thing that kinda works, instead of the new unknown.
The other immediate dangers are social and political. When one person with resources can run an AI-enhanced troll farm and social media PR engine - not even remotely science fiction - we have a serious problem.
Those threats are already politically and culturally existential.
And that's before anyone has even fitted a gun to an AI-powered autonomous drone. Or gunship.
AI is inherently conservative because it reinforces hierarchy. The AI-poor will have far less political, cultural, and personal leverage than the AI-rich.
Essentially it will have the same effects as money as a cultural and political practice - but much more so.
I would also expect a huge uptick in bureaucracy from AI, so lots of new jobs will spring up.
If 99% of consumers no longer have money to spend, GDP will definitely not be maximized.
You can talk all you want about companies selling to companies, etc., but at the end of the day everything on earth is owned by a human eventually.
Even high-frequency trades between two hedge funds are done with capital that was supplied by Limited Partners who probably are acting on behalf of pensioners.
The scenario you're describing is one of extreme wealth inequality, which is a real problem, but one that's supposed to be solvable through democratic means. Stopping technological progress isn't going to solve it.
[1] https://www.mcsweeneys.net/articles/macroeconomic-changes-ha...
This concept became particularly controversial during the Cambridge Capital Controversy [0].
If you believe that GDP is an exogenous measure of what society can produce, you run into oddities like global GDP changing trajectory in 2008 [1] (i.e. we never 'caught up' but did the US housing crisis really cause the human race to lose our ability to create more goods and services?).
On the other hand, if you believe demand drives GDP, then why don't we just demand ourselves into more wealth? Shouldn't natural constraints like resource scarcity then drive GDP (i.e. the original underpinning of the broken window fallacy)?
In either case, the broken window fallacy is far from an agreed-upon axiom.
[0]https://www.aeaweb.org/articles?id=10.1257/08953300332116501... [1] https://data.worldbank.org/indicator/NY.GDP.MKTP.PP.CD
Well, a huge portion of our economy is in fixing the old. Why waste so much money on healthcare for humans when we could use it to build bigger and better things. We have whole portions of the economy like food production whose entire purpose is to keep people alive. What a waste. The far more efficient option is to eliminate those people and let machines do the work. Even if GDP goes down, ultimate productivity goes up.
The same can be said for entertainment, hospitality, most education (we spend 20+ years educating people before they can meaningfully contribute) and marketing. A broken window theory might say these industries contribute to the economy and give people jobs, but actually they accomplish little.
Because that's the goal of our economic system! If everything could be made with no human labor requirement, then the price of these goods is going to trend towards zero. The disruption is going to cause pain, but ultimately that's what we've always been after.
No, it isn't. Profit is meaningless, money is inherently worthless.
Capitalism exists to encourage the production of (new) goods and services. The system is set up to encourage people to take their capital and invest it for this purpose.
Investments help business ideas that would otherwise never have been made, because the person with the idea couldn't afford it. In return the investor gets a slice of the potential future profits. But the goal of the system is the products the company creates.
Money is worthless. It only carries value based on what you can trade that money for. If I have $10, but a cup of coffee costs $1000, then my $10 are meaningless. But if a house cost $1 then my $10 is very valuable. And what you can trade your money for are the goods that are produced by the economic system.
Most investment is done by converting anticipated future value (modulo some degree of risk), and encapsulating it into an asset price, which is relative to the opportunity cost of alternative investments, and the current discount rate.
And capitalization doesn't have a goal beyond honoring those future returns (or "embedded growth obligations"). It doesn't care whether those returns come from innovation, production, rent-seeking, or even a "greater sucker" Ponzi scheme. Investors (and their financial agents) operate almost entirely on Make Number Go Up ("buy low sell high"), and any incentives that result in win-win increases to human flourishing are entirely secondary effects.
The goal of the investor isn't to provide goods & services, it's to get profits in whatever way is possible. He doesn't give a shit about goods and services. This tends to diverge from the interests of non-investors.
We have an investor class that is presently largely in control of our government and our society, whose interests as a part of the capitalist system are currently at a few percent remove from our interests as people who enjoy goods & services. That divergence, as well as the accumulated privilege opposed to our interests that this class derives from control of our government and our society, appears to grow over time with uninterrupted peacetime improvements in automation, and AI is promising almost unimaginably large improvements over the next decade.
The status quo socially liberal, economically neoliberal society is not sustainable even as a polite fiction in the face of that kind of shift.
Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world
As we no longer weep for serfdom-slavery, you don't have to weep for wage-slavery, but you do have to replace it with something else more functional if our society is not to completely collapse. And it's looking like we will be empowering our de facto leaders with all the blood and treasure in the world to stop us from replacing it with anything more functional. Every civil war in the world up to this point has relied upon human soldiers with their own interests and allegiances, faced with a brutal choice of risking their lives to kill their countrymen - people who have the option of not pulling the trigger. This is not going to be the case for much longer.
And that's all assuming we don't get paperclipped, assuming that Ayn Rand is the penultimate cosmic horror and there is nothing darker in the possibility space, no pressure on the right side of the Drake Equation. I don't know about that.
I would argue that our economic system is built on the accumulation of power, not an ultimate goal of zero marginal cost production. That's why cheap drugs keep getting expensive, among a litany of other examples.
You can go buy decent t-shirts with no logos for €2.50 right now. And yet most people walk around with t-shirts that cost €10-50 with some logo on them.
This becomes apparent when you look at common food like potatoes, milk, cheese, bread etc. All the competition more or less provides the same product for about the same price. If they could go cheaper while still being profitable they would.
But it isn't...it's the office knowledge worker. Anyone cruising in the middle bucket where you don't do creative research, but you do sit in front of a computer. AI is coming for those groups.
The blue collar worker? They'll be fine for decades. Anyone who gets out there and manipulates the physical environment, won't be replaced any time soon.
In fact, if you watch The Rules for Rulers, there’s the possibility of turning the entire world into a 3rd world corrupted structure: a few people at the top enjoying the riches of full automation, a few more people with guns to protect them from the unwashed masses, and everyone else.
If democracy really comes from the fact that we need everyone’s labour to make a country, we may actually lose it as automation makes most of us economically irrelevant. We need an alternative before the window closes. If we’re lucky it won’t ever close, but to be honest I won’t bet on it.
Though I reckon an organised unwashed mass could most likely do something.
All this stuff you mention relies on the the „fact“ that capitalism and guaranteed private property rights are absolute truths - when in fact they aren’t, actually more like a temporal fashion wave in some parts of the world, and faith in this destructive obsession in „the economy“ is on the decline finally. :-)
Amusing, usually rationalists are made fun of because the concerns about Roko's Basilisk are not performative.
However, when the agricultural sector became more productive the 'winners' wanted bigger houses, travel, education, healthcare etc and this provided enough opportunity for the displaced workers to find jobs. These new sectors, over time, even offered more wealth and better working conditions to their workers than the old agricultural sector.
Some of the farmers pushed out of agriculture were in a truly terrible position. Some effectively exiled into other countries (e.g highland clearances in Scotland) or left with no choice beyond military service in poor conditions.
Other ex-farmers were pulled into the high wage new economy sectors. I imagine this was a comparatively happier experience, although not without its difficulties. For example, the shipbuilding or railway workers. In a more modern example, the Chinese factory worker.
But even given that the kids will be alright, my point was that the breadwinners displaced by progress do not find an alternate income - they rely on the next generation to adapt.
Farmer-landowners are much more financially resilient to technological change because (generally) the same technologies which reduced the value of their labour increased the value of their property.
I suspect this difference in historic land ownership had a big impact on subsequent politics and it's something I'm looking to read more about.
I might suggest that drawing boundaries around a plot of land and calling it "owned" was the beginning of politics, as it represents collective agreements and mechanisms to arrive at them. Have you dug into Elinor Ostrom's work around the tragedy of the commons being an unrealistic model given that people don't just let other people's cows overgraze? We create rules and sanctions before the resource collapses (or rather, we try to; and that attempt to prevent collapse could be called government).
Anyone doing physical labor is going to be just as necessary in a world of artificial mind, it's those knowledge workers of the service economy that will find their skills automated.
> AI is not competing for resources with human beings.
A superintelligence would absolutely be competing for resources (mainly electricity and cooling) with human beings.
> Rather, we provide AI systems with their resources, from energy and raw materials to computer chips and network infrastructure.
A superintelligence will be able to convince anyone it wants to do anything for it. Even the previous President did a pretty good job at causing adulation and democracy-threatening personal loyalty from people he had not actually met in person. Imagine how amplified and personal the effect could be.
> For now, AI depends on us, and a superintelligence would presumably recognize that fact and seek to preserve humanity since we are as fundamental to AI’s existence as oxygen-producing plants are to ours. This makes the evolution of mutualism between AI and humans a far more likely outcome than competition. Moreover, the path to a fully automated economy — if that is the goal — will be long, with each major step serving as a natural checkpoint for human intervention.
The author is literally relying on us outsmarting the superintelligence (collectively, with coordination!). It doesn't sound like they've come to terms with the concept of superintelligence at all. Or with the state of human global coordination problems, come to think of it.
> AI cannot physically hunt us.
This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process.
> AI’s impact on the climate is up to us.
Our own impact on the climate is up to us. Our collective decision-making is.. suboptimal.
> If we really think that superintelligent AI presents a plausible existential risk, shouldn’t we simply stop all AI research right now? Why not preemptively bomb data centers and outlaw GPUs?
You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.
I'm stunned at the number of people who try to make this argument. The operators of Radio Télévision Libre des Mille Collines never had to leave their broadcast studio. Goebbels never had to do anything but get in front of a microphone or a typewriter. Still, millions were violently killed. We bicker about LLMs' ability to write C code that will compile, but their best abilities are to gaslight, emotionally manipulate, lie and create FUD.
Why compete with humans on Earth if there's the whole Solar system to exploit?
I love these type of thought experiments though...
What is the evidence for such an extravagant claim? Is there a correlation between IQ delta and persuadability that has no degradation as the lower IQ rises? Especially given that there are plenty of people far more intelligent than the President in your example who are much less persuasive.
So for any given person, consider how well the most effective persuader out of 8 billion people would be at persuading them. Maybe they can't always persuade them of anything, but pretty close. This is the lower bound for an AI's maximum theoretical abilities, and it's already enough to rule the world.
This "just so" claim has always struck me as quite a reach. Humans are manipulable, yes, especially by (perceived) group pressure, but there are still limits. Especially if the human can easily just mute the AI's output. It's like when people believe that a sufficiently clever hacker can get a given program to do anything. There are millions of security vulnerabilities, yes, but you're unlikely to find a RCE in the program `true`. But instead of the 80s wizard hacker trope we're going back to the older B-movie hypnotist who can take full control of any person by talking
> You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.
There's a few links in the article - which person are you referring to? Being the most prominent voice calling something unsafe is not an inherently valuable qualification. The most vocal opponents of nuclear power tend to have a poor understanding of it
You could reimagine it as "convince most people to do anything, including killing people who are unconvinced" if you prefer.
> There's a few links in the article - which person are you referring to?
Sorry, this link, which is attached to "bomb" in the article: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
> Being the most prominent voice calling something unsafe is not an inherently valuable qualification.
That's true. But the article does something odd instead of making that critique -- it presents bombing GPU data centers as a reductio ad absurdum, suggesting if you really believed in AI risk then you would have to be willing to consider doing that. The people who really believe in AI risk already are willing to consider it, though.
I see this argument against existential risk alliance – that AI needs humans to provide it with resources- this we can hold it hostage. I like to point out that power plants are increasingly since the pandemic remotely operable- that AI will increasingly be able to hold us hostage by hacking the infrastructure we need to survive. Throw in the ability to use some robots and bio weapons to keep humans from taking them back easily.
The other aspect is that AI doesn’t have to properly conclude ahead of time that such an attempt has a high probability of success. Its program just has to decide that this will help it achieve some objective.
For example, ASI may be able to automate manufacturing really quickly, maybe be able to convince humans easily to do what it needs, etc Moreover, why ai would not be competing with us for resources? Materials on earth are finite.
Also, most likely scenario is not evil AGI, as it outlined in the article, but indifferent. And basing reasoning about AI on Darwinian evolution is barely useful.
Also, talk about this tribal people AI is ironically even worse then silicon valley narrow viewpoint: it is Californian liberal virtue signaling.
I think we're only like this to animals because we are animals (we forgot), and we compete for the same type of resources they need, water, protein, calories, we're not intelligent enough to get ourselves out of this situation, but if we could, we could.
If we could we'd be able to grow tasty meat in a lab, reverse climate change, restore forrest, synthesize water cheaply, live in floating cities above beautiful habitats,,,etc etc. We're just not that smart so we do dumb shit, like factory farm.
We still compete for space and energy at the very least.
Humans are sympathetic to animals because we are alike. We don’t want animals to suffer. However, will an AI be sympathetic to humans because we „think alike“? We don’t know.
I do wonder if the quest to build AI will teach us a lot about why the thing we take for granted exist. Why have fear, limits, empathy, anger etc.
I suppose bacteria is a bit like this but it’s limited in its capabilities to evolve too quickly and wipe everything out?
I just think that we not only need to work on all of the more immediate problems but also worry about less immediate problems that are existential risks.
Also people always say "AGI" and they mean different things but I am guessing most now mean something like a digital living human/simulation of human when they say that. Mixed in with some blurry connotation of it automatically becoming godlike in power.
That stuff is very speculative. But we don't need AI to get to that level for it to be very dangerous. Just imagine that we have an open source GPT that is something like 33% smarter than GPT-4 and less brittle. Make the output 50 times faster than human thought and suppose that we can run it very inexpensively, in a "swarm" of agents cooperating.
Then you have a type of superintelligence that does not require any really speculative AI advance -- hyperspeed reasoning -- based on the current technology.
If that is widely deployed for military and industrial decision-making then something like a computer virus could create existential risk.
Also, realize that all these systems really need to emulate some of the core functional aspects of animals, like a self-preservation instinct or desire to control resources, is the right instruction and relaxation of guardrails. So when they can reason a bit better and at hyperspeed with collaboration between them, they don't need to be alive or anything to be dangerous.
Look at the history of increases in computing efficiency. It is easy to imagine 50 or whatever times output speed increase in less than 5 years. And even though they are not human level now, GPT-4 has proven that these types of systems can have strong problem solving ability.
There is this image of super-human, much faster, much broader thinking in the collective. When a bog standard human, drunk and in a foreign country, can do plenty. By this measure, chatGPTs are plenty intelligent enough - just not yet connected enough to tools, a wallet and vague orders or intentions.
The general discussion is definitely prematurely focused on some sort of end-boss fight, while AI and large scale data collection are causing serious harm and privacy concerns to real humans already. It’s only reasonable to expect that this increases in the future.
I would like to see more discussion focussing on these pressing issues.
Also a pause on AI development is not an option, not a solution and is digressing from the issue at hand. Big capital with their hands on big valuable data will support anything that distracts here.
Finally, I do think that AI also brings many positives, I am not against the technology itself at all- we shouldn’t be.
of course it isnt physically impossible. AI requires large compute, large resources. you cant advance the state of the art inconspicuously. the only reason you say it isnt an option is because you dont consider AI an existential threat, really, and therefore extreme measures arent justified. but they actually are.
“Hey fellow nation states, let’s not create nuclear bombs, but let’s pause developments and keep the current power dynamics fixed please” is completely unrealistic. Mind that creating nuclear weapons actually requires technical operations that are factors more complex than “large compute” in many aspects… like… enriching uranium and working with supercritical masses.
> you cant advance the state of the art inconspicuously.
What evidence has led you to believe this?
Google "robodebt australia" for what happens when government uses machine derived decisions to penalise the poor.
It's hard to play out a scenario where this leads to actual human extinction.
This isn’t the Yellowstone super volcano that might blow in the next billion years.
People who talk like this are idiotic at best.
Granted, the other concerns are real too but the accusation that the concern of existential risk is being used to hide the other known problems is dangerous in the extreme.
"People who talk like this are idiotic at best."
https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-i...
The goalposts will continue moving until morale impairs.
[0] https://news.sky.com/story/mathematicians-hail-breakthrough-...
All the forecasts I've seen have the period of danger spread out over at least 20 or 30 years, and no one claims to understand the topic well enough to be able to tell whether a particular model is dangerous even if given complete freedom to examine the source code and interview the creators.
Our inability to predict is in large part the basis of the pessimism/alarm because it means that the leading labs won't know when to stop.
"How many years after 1945 ... would it take for the Soviet Union to get the bomb? Framed this way, it was a much easier question than forecasting the advent of AGI: The Americans were concerned with only one adversary and the development of a single technology which they already knew all the details of. American predictions of Soviet proliferation offer a highly constrained case study for those who would undertake technological forecasting today."
It really isn't. There are people quitting their jobs, protesting AI companies, and inducing panic attacks in children over their irrational fears:
It is easy to find people who believe that it is possible (with some low probability) that AI will end the human race in the next 12 mo. (I'm one of them.) I haven't seen anyone however put most of their probability mass in the next 12 or 24 mo, and I'd like to find out what their reasoning is.
I think the criticisms of the Boring Company are more "small diameter tunnels are a bad way to build high-throughput underground infrastructure, and autonomous cars will not solve that"
Those tunnels are 12 feet diameter. Boring Co's tunnels are 12 feet diameter.
A modern subway system needs considerably larger tunnels.
It's more like political polarization where people dwell on something they don't agree with to the point where they treat it as if the world is going to end because of it, instead of in context. AI is more like the political party they don't like winning, potentially undesirable from a certain point of view but they see it as "existential".
I think it's important to see it in this light, instead of trying to actually debate the points raised, which are absurd and will only give credibility where none is due.
Which category would you put Geoffrey Hinton in? https://www.utoronto.ca/news/risks-artificial-intelligence-m...
https://www.rsna.org/news/2022/may/global-radiologist-shorta...
https://mindmatters.ai/2022/08/turns-out-computers-are-not-v...
"Deep learning will replace radiologists within 5 years" isn't as inherently crazy as it probably seems in hindsight. Sure, it didn't pan out, but I know a vet who works for an Ai company classifying scans for training and he says first hand the models have gotten a lot better. That said, he's still not worried about being replaced.
https://en.m.wikipedia.org/wiki/Appeal_to_the_stone
“Appeal to the stone, also known as argumentum ad lapidem, is a logical fallacy that dismisses an argument as untrue or absurd. The dismissal is made by stating or reiterating that the argument is absurd, without providing further argumentation. This theory is closely tied to proof by assertion due to the lack of evidence behind the statement and its attempt to persuade without providing any evidence.
Appeal to the stone is a logical fallacy.”
Any serious attempt at AI would: need to be trained on reality - like people are trained; need to overcome the cardinality barrier - digital systems can only approximate analogue systems; and need to demonstrate spontaneous language emergence driven by survival in a social environment.
I don’t see why a strong AI _needs_ to be trained specifically on reality. Imagine an alien race that sees world in X-Rays and neutrinos. An alien AI researcher claims that a “true intelligences” must train on the same inputs, and all those systems trained on the part of the spectrum between infrared and ultraviolet only fake intelligence.
I don’t see why intelligence has to be analogue in any way. Analogue systems are inherently noisy and contain limited amount of useful information, so you can always faithfully approximate them using digital representations.
If I am not mistaken Deepmind has shown spontaneous emergence of communication around 5 years ago when training agents in multi-player team shooters.
the key observations: imagine a human with intuition so good and so powerful that you could compose an essay without thinking, as a reflex. people dont understand how insanely powerful this AI would be if it had an intellect to match its intuition. its coming. people dont understand that intuition is a necessary part of an intellect, that were already half way to true AI. these LLMs are a bundle of kindling.
From a history-of-technology standpoint, the state of the art for generative AI went from LSTMs to ChatGPT overnight. There was no warning. Nobody could have looked at the Attention Is All You Need paper and predicted how much it would improve the state of the art.
And worst of all, we already know that AIs can be hostile. What else is a public corporation's corporate bureaucracy but a Chinese Room running an AI whose goal is to maximize quarterly dividends? What else is a government but another Chinese Room trying to survive? We talk about corporations being faceless or friendly, we ascribe them wants and needs and behaviors and habits, we give them names, and they suffer natural selection and mutation and reproduce and evolve over time so as to select from those bureaucracies that are most fit for survival. And this class of AI is already responsible for global climate change and the Cold War and so many other things that might already be on the path to push us into extinction.
LLMs are nowhere near as humanlike as a corporation. Corporations are bound by regulations and capitalism and basic human decency. It is absolutely possible to build an LLM whose fundamental goal is to kill people. All you have to do is tell ChatGPT to pretend it's playing a game of global thermonuclear war, and some joker on 4chan would do that in a heartbeat.
So what? Why does this matter?
We have no way to predict when the next revolution in AI capability will occur.
We have no way to predict what the next revolution will be.
We have no way to predict the capabilities of the next generation of generative AIs.
We have no way to predict how the next generation of AIs will "think", insofar as they do.
And because we have no predictions, we have no way to prepare.
So the real question is, how much risk are you willing to tolerate? Are you wiling to take a coin flip on it? A spin of the cylinder? A roll of the dice? Because that's what we're looking at here - the next generation of AIs has no reason to be any less inimical to human existence than the examples we already have, and AI will _only_ get better.
and i never saw a single person on HN or anywhere else raise concerns or even appreciate the true impact GTP would have today in 2018 when GTP was released and samples available! people coundnt get it right even when it was right in front of them. how can they think for a single second theyre getting it right now?
to my credit i recognized immediately what was going on in 2018 and to my knowledge i was the only person leaving comments warning people and begging people to support a pause of research. so by the only material metric, i am more qualified to make predictions than anyone here or even the experts who all confidently predicted that what is happening now would never happen in our childrens lifetimes.
i will never forget when hinton saw the light. when he started warning of danger and using the key words that are at the heart of the issue. and i realized i had been ahead of him. just because youre an expert in one thing, even the implementation of ML or AI, doesnt make you an expert on the broader consequences of AI.