The greatest risk of AI is from the people who control it, not the tech itself
aisnakeoil.substack.com
aisnakeoil.substack.com
People are jumping too quickly to deeply integrate this tech with everyday things, and while that's great for many use cases, it's not so great for others.
You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.
Many of those people were not tricked, tech CEOs are involving themselves with this and picking up Yudkowsky's particular strain of AI safety intentionally.
Instead of focusing on the very real problems AI causes and exacerbates today, problems that might affect tech companies' bottom lines if addressed, CEOs can control the narrative and appear proactive by buying into this brand of science fiction.
Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies.
The most useful of fools. Yudkowsky acts, speaks, dresses and looks like the most neck-beard of neck-beards. He wears that cringey trilby hat in podcast interviews.
I've often thought that the popular promotion of people like Michale Moore and even Noam Chomsky works in the favor of those in power. Choose the ugliest, most boring and laughable opponents and then allow them to speak while just rolling your eyes and giggling. It works even better when they are speaking sense since you can demonstrate you are allowing legitimate dissent.
I wonder if Chomsky ever suspected that his own promotion and fame were the result of the establishment using his yawn-inducing academic tone in their own attempts to manufacture consent. Can you imagine someone who is obsessed with the Kardashians being caught dead agreeing with Chomsky, Moore or Yudkowski? And then consider if the number of those who are swayed by such superficial appearance are the majority or minority and how that effects democracy.
We don't have an AI safety problem, we have an human safety problem. Violent extremists are unsafe, greedy corporations seeking to suppress the free exploration and exercise of mathematics are unsafe. Existent language model chatbots are primarily just a public relations risk, at least so long as the government doesn't step in and create a monopoly that gives the market leaders choice of training biases the force of law.
So what if he likes wearing a hat?
If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk. The core argument is just
1. We might create ML models smarter than ourselves soon
2. We don't really understand how ML models work or what they might do
3. That seems dangerous
There's more to it than that of course, but most of the "more to it" is justifications for each of those steps (looking at the historical rate of progress, guessing what might go wrong during the training process, etc.).
The people who dismiss that AI is an existential risk might have really good counterarguments, but I've never heard one. The only counterarguments people seem to make are "people are scared of technology all the time, but usually they're wrong to be", "that seems like sci-fi nonsense", etc. If you want people to stop being "tricked" by Yudkowsky and co, the best way to do that would probably be to come up with some counterarguments and communicate them.
Your burden of proof is backwards.
It is on the AI-doomers to explain why sci-fi concepts like “AI improving itself into a super intelligence” or “AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead” have any relevance in the real world.
The entire AI-doomer worldview is built off of unproven assumptions about the nature of intelligence and what computers are capable of, largely because thought leaders in the movement are incapable of separating sci-fi from reality.
No one is making this argument. The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.
If you watched the conversation around Meta with Facebook and Instagram, and then later the conversation around TikTok, almost nobody cut to the heart of the issue, which is that "algorithms" are being used to make decisions about what to show people, and that the algorithms themselves have been changed subtly hundreds or thousands of times over the years to maximize engagement, until the type of engagement being maximized has made people basically crazy. The same engagement-maximizing work will proceed with AI, and it will allow the software engineers and managers responsible for developing the "algorithm" abdicate even more responsibility for genocides and for mass hysteria because they can pretend like they have no control over the algorithm.
The same irresponsible people will pocket a bunch of money for all this work and they will maximize engagement until it blows up our entire culture. And because nobody understands it except the techno-cultists, nobody will hold them accountable.
The trolley problem is similarly a thought experiment in moral philosophy that many moral philosophers have used to guide discussion, but nobody actually takes the thought experiment seriously as a "real-world" thing.
If you actually want to engage with the argument in good faith for why an AI might indeed be smart enough to wipe us out but "dumb enough" to just pursue some other goal ("make paperclips as an extreme example") there is a great video by Rob Miles here: https://www.youtube.com/watch?v=ZeecOKBus3Q
Point out the flaws in the reasoning. Just saying "this is nonsense" does nothing but prove you've never actually taken the time to understand the best arguments.
Also... Alan Turing, Geoffrey Hinton... Extremely influential and intelligent people take/took this seriously. These are not SciFi fanboys. AI doom only became sci-fi after smart people like Alan Turing decades ago raised the alarms flags about where AI development might go if we are not careful.
It's a very compelling combination of ego and convenience.
To me it doesn't feel technical at all--just superficial use of some domain verbiage with lots of degrees of freedom to duct tape it all together into a story. He very much reminds me of Eric Drexler and the nanotech doomerism of the 80s and 90s. Guy also had all the right verbiage and a small following of fairly educated people. But where is the grey goo?
If we need a counterargument to Yudkowsky do we also need one to Drexler?
It's called life.
Drexler may have been off with the approach to take, but he isn't wrong about the fundamentals.
According to the Yuddites, AI is likely to cause total extinction the first time something goes wrong, so the usual mechanisms aren’t enough.
Right now, there is no central entity that has supreme control. Rather, humanity is guided by a hierarchy of organizations made from individual humans. The decisions and behavior of these organizations is determined by both the desires and needs of individual humans (basic needs, status, community), as well as dynamics acting directly on organizations (competition for survival, selection pressure for increasing control over resources). The system is in a somewhat stable equilibrium because no organization can achieve domination due to a balance of power and due to many individual humans not wanting that to happen.
AI does two new things. First, it allows for potentially immortal agents which have much completely stable goals, which they will pursue with no regard for anything else. This is something individual humans cannot come even close to. Organizations can come closer but still not that close. Second, fast progress in AI allows for a temporary large imbalance of power. In particular, AI has the potential to enable scalable and effective technologies for monitoring and controlling humans. Using such means, an organization (whixh then naturally ends up partially controlled by AIs) may outcompete others and consolidate control.
The danger is that this consolidation may be completed for the first time in history. No organization has ever achieved world domination, though a few have come somewhat close. I would say one of the strong factors that prevented this from happening was the difficulty of maintaining the organization's goal of domination, while influential people die, change their minds or pursue their own goals above those of the organization. Systems involving human organizations have slack. Systems of organizations made up of or controlled by AIs may be very different.
The crucial point for the "existential" part of the existential risk is that we all used to believe world domination is obviously impossible. This must be carefully reevaluated with AI entering the picture. Even a small danger is worth considering and large mitigation efforts because a failure may be permanent and irreversible.
You have an extremely high risk dying in a vehicle collision but people dismiss the risk because it isn't immediately apparent that 40,000 people die every year in the U.S. from that very cause. Certain deadly events become common enough that they get ignored due to desensitization, or become categorized as individual and anecdotal tragedies because they aren't mass casualty events like a plane crash or a hotel collapsing. If a few thousand people die every quarter from AI lockups or failures that nobody could've predicted or prevented because the AI is fully operationally autonomous, it will be treated the same as vehicle deaths. And that is the true fear.
Maybe that is an argument you've seen, but the more convincing argument I know of is that we have no way of knowing we've gone too far until after the fact. Doesn't matter how many times it goes right first, we don't even know how to determine if and when it has gone wrong. A powerful enough AI could deceive us into thinking things are going "right".
That's why the Republican party has been gunning so hard in the past couple decades for the Chevron deference precedent: they want it harder to have legally-enforceable regulations that don't require lawmakers to get into the weeds (and risk being narrow and outdated soon).
It's going to be a dark day when some idiot passes a law forcing AIs to serve as mandatory reporters.
The issue is our acceptance of information as if it were true, as if misleading ideas were not monetisable, as if we can outsource the basis for why we make decisions to an external authority. Hardly anyone verifies anything. Most simply accept whatever they are told. Deep skepticism and empiricism are used by very few - instead we have been taught to trust authoritative sources (media, academia) which can be both well meaning and wrong.
Anyway, skepticism and personal verification is the best answer I have to the whole story saga of how to determine truth from lies. This issue is under an especially bright spotlight thanks to ai.
I'm pessimistic over whether many will be prepared to 'verify better' in the future. Unfortunately, I suspect things will have to get a lot worse before we start to learn. It seems that ai can create compelling content, that will be tailored to each individual - who could resist 24/7 pandering to one's predilections and biases?
In the case of U Michigan and Flint's water infrastructure predictive AI far far far outclassed the predictions of local contractors on where the actual lead pipes were buried. The AI was order of magnitude more accurate.
Regardless of AI's efficacy Flint's Mayor temporarily replaced the AI (bc AI fear mongering stoked classism) with a contractor who was right only ~15% of the time vs. the AI's 70%+. Those numbers affect thousands and thousand of people's access to a basic human right: water. US Court determined the AI had less bias, and there should be no discrimination on where to dig up pipes based on where someone lived (ie. richer neighbourhoods).
The benefit of AI is it makes decisions MORE transparent, not less. It pulls apart prediction from judgement in decision making. So you can tweak it, call bullshit on it, etc.
But our industry already operates this way. Google will cut you off for triggering automated rules, and good luck getting human help. AI will not make it worse; but it will be used by such businesses to give their CS the appearance of being better. It will feel like you're talking to a real person again.
True in a fair number of cases...but, based on their actions to date, I doubt that Google cares enough about appearances to bother.
I think it's more disappointing that the fact that lots of smart people are extremely worried about something, doesn't cause onlookers to get worried - it causes them to be dismissive.
Yudkowsky and others had certain worries, as more people heard their arguments and technology improved, more and more smart people increasingly got convinced of these arguments. Instead of listening to them or considering that they might have a point, many people here are extremely dismissive - "they're tricked", "they watch too many sci-fi movies", "they're corporate shills", etc, even when all of these arguments can be refuted by two simple ideas - there are many different people with different backgrounds getting worried, and most of them weren't worried 10 years ago, but are worried now, meaning their point of view changed with growing evidence.
Let me ask you this: at what point will you be worried? What would it take? If some of the people who built these technologies are worried isn't enough to cause you to change your mind (or at least consider that they might have a point), what will?
Note: For the record, I'm also worried about your specific "today" worries. I just hate the dismissiveness of your last paragraph (and of a common sentiment on HN).
This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic.
You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.
Here's the thing: I do actually kind of believe that. Not because I think that AI will be super intelligent and will do so out it's own volition. I think that despite AI being barely functional people will put it in charge of power generation to save a buck and it will just fail.
You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?
So... what's your answer to the question then? Are you just assuming we are likely to be annihilated and chill with that?
Anyone who was working with transformers could have seen ChatGPT on the horizon, it wasn’t surprising at all that scaling an autoregressive model can result in something seemingly intelligent.
Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based?
What is my estimate for AI being an existential risk in the next couple of decades? Depends on if we find something that actually resembles AGI which is impossible to predict. Based solely on current technology + scaling I would personally put the chance at essentially 0%.
It's a totally made up example. Its purpose was to engage you in a conversation about what's reasonable if you believe an existential threat is possible in the near term. It doesn't seem you are willing to engage that possibility.
> Anyone who was working with transformers could have seen ChatGPT on the horizon
So, who did?
> [LeCun is] highly critical
I did a quick search and found this[1] trash-tier article explaining his views summed as:
> He thinks the will to dominate and intelligence are separate things. Orangutan do not have a desire to dominate but are intelligent. They are territorial.
I have absolutely no idea where this distinction comes from. Being territorial is a form of domination, it's just not the same as expansionist behavior. If an AI is territorial, and the Earth is its territory, it doesn't need to be expansionist to annihilate threats on Earth.
Anyway, LeCun's former colleague Hinton was on NPR yesterday telling the masses that he thinks we could have dangerous AI in less than a decade. It's fair to say there's wide disagreement. I don't think it's fair to say that handwaving means we shouldn't be worried about the problem.
Finally, your question:
> What is so impressive about ChatGPT that it poses an existential threat?
This is a bit of a logical misstep, or I was unclear previously. I don't think anyone sees ChatGPT or anything directly related to it as an existential threat. Rather, the rate of change in capabilities of LLMs is worrying, because if we see a similar rate of change in systems with more agency, it could be a threat. Or, in the words of the article about LeCun's belief:
> Lecun says that in a few year LLM (Large Language Models) will go away and replaced with systems that will be guidable to desired goals.
So... he is predicting exactly the same situation that everyone is afraid of, and he's asserting that it will work out fine. Somehow.
The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.
> essentially 0%.
I sure hope you're right, but I don't think that's a common position to hold.
[1] https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-co...
And it's not like this estimate is based on a lot of concrete reasoning. So yeah, I would expect it to fluctuate wildly at an unexpected development in the field initially, then settle down around a slight change. Which is basically the definition of a hype cycle.
I believe that AI can't possibly pose an existential risk in the next decade or two. I believe AI poses a great risk, but an economic one, and not an existential one.
> Actually I'd love to know what your estimate actually is for AI becoming an existential threat
My estimate is: never. At least not in the form of some superintelligent AGI.
Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.
Or are we talking about a variation of the system we have right now, where someone could use the AI as a part of a control system and then the "algorithm" doesn't operate the way we want and causes an outage? Because that happens all the time without LLMs.
I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
No, but that's actually irrelevant. The reason it's irrelevant is because of the answer to:
> I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
Nobody is saying the second thing (that I'm aware of). This is the chain of reasoning:
1. "LLMs are an amazing technology [which advanced at an unexpected rate]"
2. Because LLMs advanced at an unexpected rate, this indicates an acceleration of research
3. Such acceleration could conceivably happen or be happening now in other forms of AI
4. The path to AGI is COMPLETELY unknown.
5. The origins and structures of consciousness and agency in intelligent systems are completely unknown
6. It is impossible for humans today to know if they've crossed a threshold into creating AGI, or a new form of intelligent alien life.
7. AGI is inherently alien and there's no reason to expect it to think the way we do.
8. Even human goals and systems are often (or even mostly) misaligned - look at any for-profit corporation or totalitarian state for an extreme example of how their existence harms people in many cases (pollution, murder, exploitation, discrimination, etc)
So, I think I've covered the bases. I'll sum it up like this: We don't know what AGI looks like, we can't expect it to have our best interests in mind because it's an alien that's smarter than us, and we don't know when or how we'll achieve AGI. So unless you don't believe superhuman AGI is possible at all, we're in a very scary time. We have know way of knowing if it will take 5 years or 100, but we also know that people all over the world are racing to make it happen as soon as possible.
Yes, that's a problem, and it's a problem that has a lot more examples in the real world. It doesn't automatically invalidate the problem of large-scale nuclear war. They're both big problems.
Same with climate change vs air pollution, political scandal deepfakes vs naked celebrity deepfakes, etc.
In the same vein misanthropic AGI should be delegated to top secret committees that no one knows about. Broadcasting that concern live is a distraction from the real issues the average person should consider: how do I get these tools away from organizations uninterested in me?
So if an AI generated bill for service was found that I Owe $N$ - I should be allowed to see all the code and logic that arrived at that decision.
I’m struggling on this side of the equation and find the hype and noise depressing.
1. Bad people control and use AI to dominate/destroy the world
2. An AI is created which resists human control entirely, and it decides to take an action that doesn't bode well for us. It is a fundamentally inscrutable mind to us, so we don't know what action it will take or why.
What data would comprise such a model?
"""Here is a person's power usage history:
{{ power usage by month }}
Here is their bill payment history: amount, date bill sent, date bill due, date payment received, if any:
{{ history }}
Here is their credit history for the last five years:
{{ credit history }}
Here is the location of their home and some overall information about the grid:
{{ grid info }}
Please give me a strategy for maximizing profit from this customer, options include "disconnection", "encourage them to use power at different times", "encourage them to buy solar", "move them to variable usage-based billing".
"""
Power companies in most places are probably too regulated to get too sneaky, but I'm sure there's shady stuff that could be done, especially if you can similarly tailor the marketing to each user e.g. "we want this person to get solar since the grid going towards them is near its max capacity and we don't want to invest in upgrading it, tailor the messaging towards them based on their credit history and social media profiles."
If you imagine a less regulated industry there's even more room for price discrimination and such.
The issues, I think, are at least three-fold:
1) Do we want that sort of individualized attention per-user based not just on observed behavioral metrics (ads clicked, sites visited) but every word they've typed online? too?
2) Who is responsible if this model then makes decisions that harm people? The people who trained it? The people who used it? The CEO? It's a wonderful tool for bureaucracy to avoid there having to be a "decision maker" and just have people follow what the tool says to do.
3) And just the practical: the model also is going to spit stuff out, but is it really going to be "optimal" for something closer to a maximization exercise vs just text generation? Possibly not, but I've seen people try stuff like this anyway.
Planning coercive disconnections in order to maximize profit from a customer based on credit history seems like the problematic thing in this example. Asking an LLM for recommendations on doing seems unrelated.
If they asked a human consultant for it instead, it would be just as bad. And just because the human's recommendations would be more explainable, it doesn't make the human consultant's recommendations any less problematic.
The power companies are going to buy access to this data. This government regulated utility is going to be allowed to charge its subscribers based on social metrics.
This sounds very far fetched to me.
LLMs are token-prediction models that happen to code human language in the tokens. You could train a similar model based on e.g. sensor inputs for moderating a plant and regulating a grid.
They provided an example and finding a technical flaw in the example they chose doesn't invalidate the broader concern as applied to other domains.
And in the meantime here are we, hackers, with with our dark Web, our peer to peer systems, our open source and our encrypted communication. We can develop AIs of our own, distributed across jurisdictions. Training costs are getting cheaper, as is computing hardware. They think they can regulate all that? Come and take it from us.
The horse bolted months ago, and surely the great minds at these leading firms can see this. What's the real reason behind these calls for us to set aside our curiosity and close our minds?
Sorry, you can't buy GPUs any more.
And there you go, it's over for open source AI. Our supplies will dwindle and we'll be far behind the 'licensed and regulated' data centers that are allowed these 'munitions'.
And besides, if you take away the bread and circuses, how long would such a government last?
The collateral damage level for this scenario is at a suicidal scale, and would just hand everything over on a platter to a competing high tech state.
It would be a huge hit to AAA games, but the games industry is a lot more than that, and very little outside of AAA games require a GPU.
This is the endgame for the rhetoric OpenAI and its associates are espousing. They're positioning OpenAI et al. to be the Lockheed Martin of AI.
If your government passes a law saying that GPUs can only be purchased or rented with a license, as OP was suggesting, all of that capacity disappears with the snap of a finger.
Amazon: "Can do"
Government: "Any projects that pool GPU resources will also be against the law and you must prevent it"
Amazon: "Ouch"
Look at me ; Who is the surveillance now
(Yes the intels use mormons a lot...)
Hardware doesn't have to be physically present in my house in order for me to use it to run my code.
> runaway AI is a way off yet, and that it will take a significant scientific advance to get there—one that we cannot anticipate
So an AI that may cause our extinction may be a result of a scientific advance "we cannot anticipate"? And you're having trouble understanding why people are concerned?
All the problems you've listed (COVID, global warming, war in the Ukraine) are considered problems because they cause SOME people to die, or may cause SOME people to die in the future. Is it really that difficult to understand why the complete extinction of ALL humans and ALL life would be a more pressing concern?
we also cannot anticipate an earth-killing asteroid appearing any day now, and no, i'm not bothered in the least by this possibility, any more than my usual existential angst as a mortal human.
sometimes I think the AI safety people haven't come to terms with their own death and have a weird fixation on the world ending as a way to avoid their discomfort with the more routine varieties of the inevitable.
In case of AI, it's unclear whether we need any additional scientific advances at all beyond scaling existing methods. Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent.
You don't have any idea whether additional advances are needed. You don't have any idea what advances are needed. You don't have any idea whether those advances are even possible. But you're confident that they will happen with probability > 10%?
You're making very confident assertions, but you have no factual basis for doing so. (Neither does anyone else. We don't know - we're all guessing.)
Yes, because there's plenty of datapoints to make prediction by extrapolation. Just look at the progress between GPT-1, GPT-2, GPT-3 and GPT-4 and extrapolate to GPT-5, GPT-6 etc. There could be some roadblocks that would prevent further progress, but the default prediction should be that the trend continues.
Metaculus has had predictions for various AI milestones for years and has been consistently too conservative for resolved questions. It now predicts 50% of the invention of full AGI by 2032 which by extension might also be too conservative.
By this method there is no way to calculate the risk of an AI extinction simply because it never happened before.
Meanwhile, we've got a war in Ukraine with probability 1.
So AI risk has to get in line with global nuclear war, and giant meteor strikes, and supervolcanoes - risks that are serious concerns, and could cause massive damage if they happened, but are not drop-everything-now-and-focus-your-entire-existence-on-this-one-threat levels of probability.
Is that true? Are there unimaginably many ways in which some hypothetical AI or algorithm could cause extinction?
I don't think so, I think the people who control [further research] are still the most important in that scenario. Maybe don't hook "it" up to the nuke switch. Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems (even in this sentence, the amount of "not currently technically possible" is extremely high).
Let's consider the war in Ukraine, on the other hand? How might it cause extinction? That's MUCH easier to imagine. So why would it be less of an concern?
If we knew how to make sure that this does not happen, the problem would be solved and there would be nothing to worry about. The problem is that we have no idea how to prevent that from happening, and if you look at the trajectory of where things are going, we're clearly moving in the direction where this occurs.
"just not doing it" would have to involve everyone in the world agreeing to not do it, and banning all large AI training runs in all countries, which is what many people are hoping will happen.
EDIT: we understand enough to know that today, "runaway GPT" is not a major concern compared to, say, a war between nuclear-armed world powers.
GPT is not "clearly moving in the direction" of consciousness for any normal definitions of "clearly" and "consciousness".
Is that true? Are there unimaginably many ways in which AlphaZero can beat me at a game of Go?
I don't think so, I think the people who control superhuman game playing AI are still the most important in that scenario.
-------
This line of thinking is quite ridiculous. Superior general intelligence will not be "controlled."
I think this is exactly the part that we can't anticipate or (potentially) control.
> that could hypothetically jailbreak your systems and migrate to other systems
This part, however, we absolutely can: There is no reason we can't build our proto-AGIs in sandboxes that would prevent them from ever having the ability to edit their own or any other program's code.
This, I think, is the biggest disconnect between a real (hypothetical) AGI and the Hollywood version: "intelligence in a computer" does not automagically mean "intelligence in absolute control of everything that computer could possibly do". Just because a program on one computer gains sapience doesn't mean it magically overcomes all its other limitations and can rewrite its own code, rewrite the rest of the code on that computer, and connect to the internet to trivially hack and rewrite any other computer.
I was talking about an AGI, not an LLM. We don't have any AGIs right now, nor anything that is remotely likely to become one.
In the scenario where a company like OpenAI develops an AGI with the intention of making it publicly available, it will not be so from moment one. There will be some period of internal testing, and assuming that it does prove to be a genuine AGI, you can bet that they won't make it available to the public for anything less than an arm and a leg. (Hell, even if it only proves to act much more like an AGI, without actually being one, they'd charge through the nose for it. Yes, they'd make you pay them an arm and a leg, through your nose. Somehow it's more profitable that way.)
Given that the nightmare scenario being posited is, effectively, "as soon as AGI exists, it will take over the world", we're then left with three basic possibilities:
1) TotallyNotOpenAI builds this hypothetical AGI with full sandbox protections, and doesn't give it any interface to the world that would allow it to break them—no API that would give it any kind of unrestricted access or control to anyone else's systems, no matter how much those people wanted to give that to it. The AGI remains contained, whether it would choose to take over the world or not.
2) TotallyNotOpenAI builds the hypothetical AGI with no protections, because it doesn't actually believe there's any real risk. Before the AGI is even revealed to the world, it takes over from within TotallyNotOpenAI.
3) TotallyNotOpenAI builds the hypothetical AGI with full sandbox protections inside its own systems, but builds an API to allow other people to give it control over theirs, because let them pay us to screw themselves over, right? It's not like it'll take over the—oh, wait; it's taken over the world, which we also live in. Oops.
Of these, #3, which is the only one close to what you describe, seems pretty logically inconsistent. It requires not only that TotallyNotOpenAI consider the AGI dangerous enough to themselves to sandbox, but not dangerous enough to prevent from accessing other systems (which can, of course, also access their systems, unless they're fully airgapped), but that they announce this AGI, and market it publicly, with the explicit capability to be given access to other people's systems, and not have anyone quickly step up and say "Hey, that's a bad idea, we should block this". Including anyone working for TotallyNotOpenAI.
Is it impossible? No. But I wouldn't consider it nearly as likely as possibility #0: We aren't able to create AGI within our lifetimes, because just throwing more hardware at the problem when we barely understand how our brains work isn't enough.
By over-regulating or restricting access to AI early on we might sabotage our chances of successful alignment. People are catching issues every day, exposure is the best way to find out what are the risks. Let's do it now before everything runs on it.
Even malicious use for spam or manipulation should be treated as an ongoing war, a continual escalation. We should focus on keeping up. No way to avoid it.
Ew, how gauche, only stupid people are concerned with what most people are concerned about.
AI is a tricky advancement that will be difficult to get right, but I think humanity has been so far successful at dealing with a much more dangerous technology (nuclear weaponry) - so that gives me hope.
It would take a ton of nukes to wipe out humanity (although only one to really ruin somebody’s day).
Unless you are counting strategies like: try to pretend you are one of the two (US, Russia) and try to bait the other into a “counterattack,” but hypothetically you could do that with 0 nukes (you “just” need a good enough fake missile I guess).
No, The Button doesn't currently exist, and all available science says it cannot ever exist. But the chance that all available science is wrong is technically not zero, because quantum, so that means The Button is possible, so unless you want everything to be turned into pudding, you need to start panicking about The Button right now.
In what way is this an analogy for misaligned superhuman AGI? I've never heard an assertion that it can't exist based on available knowledge. This seems a very flimsy argument.
Anyway, the button not only can exist, some would say it probably does exist. Some would say it's likely to have been pressed already, somewhere in the universe. It's called false vacuum decay, and it moves at the speed of light, so as long as it never gets pressed inside the galaxy it may never reach us.
This isn't an opinion the GP comment expressed, you assumed it, which is a real reddit moment.
People can be equally worried about two existential threats. Being tied to the train tracks and hearing a whistle (this is climate change) is terrifying, but it doesn't mean you wouldn't care if somebody walked up and pointed a gun at you (this is AI, potentially). Either one's going to kill you.
Is that likely? No.
Is that more or less likely than a rogue superintelligent AI? Well, we have one example of the first and none of the second...
The carbon that we're digging up was in the atmosphere before, it has just been sequestered, we're returning to a state that the Planet has seen before.
Across the entire Earth's history we're still at a fairly cold point and a long way from "Greenhouse Earth" and the temperatures at the Eocene Optimum.
And according to the IPCC: "a 'runaway greenhouse effect'—analogous to [that of] Venus—appears to have virtually no chance of being induced by anthropogenic activities"
AI has the possibility but not guarantee to kill everyone. We could shift to a lifestyle using electricity but avoiding modern computing technology. AI can be unplugged given sufficient will, whereas a planetary system cannot.
The cherry on top is when regulation is actually proposed, the act is dropped and obstructionism re-asserted [1].
[1] https://www.reuters.com/technology/openai-may-leave-eu-if-re...
Any expert, scientist, or company executive who says "this stuff could be dangerous" can be accused of wanting more attention/grants/investment/etc
No. Climate scientists aren't walking into Congress with a multi-million nest egg behind them, no tangible solutions in front and a playbook of rejecting all specific proposals ahead. That gives them credibility these AI researchers lack.
We should still take their assessments 100% seriously
Nobody said this. If the only person arguing the dangers of climate change was Elon Musk, there would be room for reasonable skepticism. That's the difference between the AI debate and "any expert, scientist, or company executive who says 'this stuff could be dangerous'."
Of course, such things would adversely impact AI research...
So if an AI generated bill for service was found that I Owe $N$ - I should be allowed to see all the code and logic that arrived at that decision.
Anyway, yeah I think models need a way to self-register upon whom uses them.. yes Creepy AF, but also needed AF. ? disagree>??
There may be some specific applications that require regulation or control. That’s ok. But the underlying fundamental technologies should be open and free.
This just ignores the very real coordination problem. The signatories do not represent the entirety of AI development, nor do they want to unilaterally forgo business opportunities that the next man will exploit. Government is the proper place to coordinate these efforts, and so that is where they appeal.
There are so many things wrong with this line of thinking. First, it mischaracterizes the issues. Few people believe AGI guarantees the extinction of humanity. The issue is that there is a significant potential for extinction and thus we need a coordinated effort to either manage this risk or prevent its creation. It does little to stop the coming calamity to single-handedly abstain from continuing to build. Coordination is a must. Besides, most people will think they stand a greater chance of building it safely than the next guy. The coordination is required to keep other people from being irresponsible. Human hubris knows no limits.
The other mistake is misjudging nerd psychology. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it". It is a grave error to discount the motivation of trying to solve a really sweet technical problem.
Ultimately these kinds of claims are self-serving, they provide rational cover to justify your predetermined beliefs that those calling for regulation are trying stifle competition. Folks don't want to be left out of the fun. The justification is in service to the motivation.
“Artificial intelligence could lead to extinction, experts warn” - BBC front page news headline earlier today in reaction to this story.
This is the problem. Reading that BBC article will make your average joe petrified. “Extinction” is a much catchier headline than the slow creep of automation replacing/changing jobs. The latter is literally already happening around us right now and it serves some of those signatories if those issues aren’t regulated against [1]. I’m not saying forget about extinction threat. Clearly that’s an important risk to manage, but let’s not ignore these near term, huge disruptions because policy makers are busy reacting to distracting headlines.
Edit: add ref; [1] https://www.reuters.com/technology/openai-may-leave-eu-if-re...
It smells like bullshit being espoused to push an agenda, but I can't tell what the agenda is. My guess: play up the huge unrealistic risks in order to distract from the more realistic ones.
Two things about this tech that I'm personally not worried at all about: "evil agi" and "only the elites will control this".
Misdirection. We see a generic call for regulation, or unrealistic call for a global pause with no answers to how it would be coördinated or enforced. When actual regulations are put forward, they're rejected without a counter-offered solution [1].
[1] https://www.reuters.com/technology/openai-may-leave-eu-if-re...
There is a difference between trying to shape regulation to your advantage and engaging in bad faith. There is no indication there is real regulation that addresses the problem these folks are brining up that they would support.
The code for AGI will not be some monolith of software architecture. The code will likely be simple. This means that someone in their basement could build it. The steps to get there are challenging, though. A single person could have developed the transformer architecture. A single person could have used 4-bit quantization and developed an AI that is just as good as ChatGPT and have it run on their local machine.
The difficulties are figuring out the best 'needle in the haystack' to solve the problem. This requires research, and this process happens much faster if you have more people working on it. For years, many people did not put any energy into AI systems because the hardware was not here yet. The hardware is here now. The cat is out of the bag.
"Which political party has had the most politicians convicted of crimes in the last 50 years?"
"According to a comparison of 28 years each of Democratic and Republican administrations from 1961-2016, Republicans scored eighteen times more individuals and entities indicted, thirty-eight times more convictions, and thirty-nine times more individuals who had prison time1. Is there anything else you would like to know?"
Oh wait, that's not bias. The Republican Party demonstrably has a higher number of its members convicted of crime. Simply put, Republican politicians are much more likely to commit crimes than Democrat politicians.
Reality has a well known liberal bias.
> According to a comparison of 28 years > 1961-2016
edit: looks like bing pulled it answers from: https://medium.com/rantt/gop-admins-had-38-times-more-crimin...
Talking of bias - this is used the world over..
Over the past decade social media has nurtured a culture where reality is whatever you choose to believe
I know the language is "choose your reality" and I'm not saying you're wrong, but I think we could acknowledge that's not really what's happening. Folks don't get to decide facts, they decide what facts they believe. That decision doesn't invalidate those facts.
But it really can't today, not without a bad actor giving it specific goals and figuring out ways to make those yield real-world outcomes. Which is the entire point of the article.
The difference between guns and automobiles is in the name. Cars are mobile by themselves. That’s a slippery slope waiting to happen.
It’s been called “AI” ever since they were puny inference programs. Them having “intelligence” in the name signifies nothing. Only what they can do in reality.
Unless an oppressive adversary has such weapons (ICBMs, antipersonnel mines, or machine guns), wants control of your land, and decides to take it. In that situation, a country will seek to utilize such or more powerful weaponry for themselves to defend their people. If that country (1) did not manufacture such weaponry inside their borders and (2) were prevented from purchasing such weaponry from other countries by blockade, sanction, or other means, then yes, those people will be oppressed.
Joke's on you, because I can absolutely buy one tomorrow. Thankfully you don't make the laws :D
I would argue you're wrong, because there is a long, colorful history of rulers oppressing populations by making the possession of weapons illegal. If the populace has no means to fight then they can't revolt. I'm not saying my neighbor should be able to buy ICBMs, but it's definitely true there's some element, no matter how small, that he's being oppressed by not having access to the same level of force as his rulers (the government).
AI risk is a valid discussion but regulating AI in its infancy seems misguided at best.
For instance, what happens if one company gets to control the future of social media, education, etc. through AI? That company can then unilaterally dictate our minds.
That risk doesn't map to the gun thing at all.
They are grabbing headlines, and moving the conversation from the real issues, which are how AI is used in education, health care, law enforcement, securities and housing markets, the government, the military, and more
Long before AI causes mass harm without human involvement, humans will find hundreds of ways to make it cause harm, and harm at scale. I do think the technology itself is part of the risk though, because of the flaws, the scale, etc inherent to its current iterations. However, maybe those are still the fault of humans for not giving it the proper limits, warnings, etc. to mitigate those things
That said, it can be used for good in the right hands (accessibility tools, etc), potentially, though I'm certainly more of a doomer at this point in time.
While putting an AI in charge of weapons is at least three Hollywood plots[0], it has also (GOFAI is still AI) behind at least two real-life near-misses from triggering global thermonuclear war[1].
[0] Terminator (and the copycat called X-Men Days of Future Past); Colossus the Forbin Project; WarGames
[1] Stanislav Petrov incident, 26 September 1983; Thule Site J incident October 5, 1960 (AKA "we forgot to tell it that the moon doesn't have an IFF transponder and that's fine")
It seems to me they're inconsistent ways of understanding what is happening, since concern about misuse also seems to me to imply regulation. But I seem to see a lot of people on HN who say both things who give the impression that they're agreeing with each other.
Why aren't we talking more about it?
I read the article you linked to, both parts. I wonder how much of people having psychotic breaks in the rationalist community is due to 1) people with tendencies toward mental illness gravitating toward rationalism or 2) rationalism being a way to avoid built-in biases in human thought, but those biases being important to keeping us sane on an individual level. (If you fully grasp the idea that everyone might die, and have an emotional reaction to that that's proportional compared to just one person you know, it can be devastating). I think we are bad at thinking about big numbers and risks, because being very good at evaluating risks is actually not great for short term, individual survival -- even if it's good for survival as a species.
I know personally the whole AI/AGI thing has got me really down. It's a struggle to reconcile it with how little a lot of people seem to put stock in the idea of AGI ending up in control of humanity at some point. I totally agree that everything on your list is a real issue -- but I think that, even if we completely solve all those issues, how do we not end up with a society where most important decisions are eventually made by AGI? My assumptions are 1) that we eventually make AGI which is just superior to humans in terms of making decisions and planning, and 2) there will be significant pressure from capitalism and competition among governments to use AGI over people once that's the case. Similar to how automation has almost always won out over hand-production so far for manufactured goods.
That's more the scenario Paul Christiano worries about than Yudkowsky. It seems more likely to me. But I still think that a lot of our mental heuristics about what we should worry about break down when it comes to the creation of something that out-guns us in terms of brainpower, and I think Yudkowsky makes a lot of good points about how we tend to shy away from facing reality when reality could be dreadful. That it's really easy to have a mental block about inventing something that makes humanity go extinct, where if it's possible to do that, there's no outside force that will swoop in and stop us like a parent stopping a child from falling into a river. If this is a real danger, we have to identify it in advance and take action to prevent it, even while there are a bunch of other problems to deal with, and even while a lot of people don't believe in it, and even while there's a lot of money to be made in the meantime but each bit takes us closer to a bad outcome. Reality isn't necessarily fair, we could be really screwed, and have all the problems you mentioned in addition to the risk of AGI killing us all (either right away or taking over and just gradually using up all the resources we need to live like we've done to so many species).
perhaps AI will be generally intelligent, but please for the love of god, let's only consider that scenario what it actually happens.
But you're right in that the two problems are inseparable. Wisely implementing sensible and effective AI controls will fall onto politicians and statespeople, from legislatures to constrain things domestically, to Presidents who have to lead abroad in forging meaningful and effective treaties.
This only makes sense if you think that at that point, it will not be a "the genie is out of the bottle" scenario and already too late to put it back in. And yet that's what people constantly say about current LLMs.
Does AI have to be strong to kill everyone? I can imagine scenarios in which it could be fairly shit, but good enough.
> please for the love of god, let's only consider that scenario what it actually happens.
Not a great philosophy to guide planning.
Let’s hear these scenarios that aren’t possible without AI.
An LLM is tied to a bunch of APIs as an event response agent. Maybe in some poduck enterprise support group.
But... there's tons of other LLMs tied to event response agents, in ... less podunk groups.
... and there's some fancy black box org in the CIA that then employs them as some crisis monitor.
and a cascade of false reporting leads to "oh my god there's nukes in flight, and the system has autolaunched nukes in response".
What current AI is good at is faking being intelligent, so that they can be used in many many places to substitute for "real intelligence" humans.
But... that means dangerous cascades in all likelihood.
That isn't skynet, but it is 90% of what the Terminator skynet disaster scenario was: an AI triggered a nuclear exchange.
The only thing that can be regulated is private citizens and corporations.
Our governments, to varying degrees, are under democratic control. Private corporations are not
I would argue worse actually, because most CEOs seem to believe if you can murder someone and get away with it and it would benefit your business, you are legally required to do it.
No.
Not to you and I.
The government may spy on you, routinely, but it does nothing to you with that data.
Private power routinely spies on you too, and has an agenda about how to use that data to their advantage, not to yours.
Technology advances inexorably, new domains are unlocked, creating ever more powerful abilities. The process is maybe even "exponential" - in the combinatorics sense.
Developments are thus capable of producing an ever increasing range good or bad outcomes.
But social technology, broadly defined, remains stagnant, actually regressing (if we take the deceitful and manipulative social media or the general (geo)political atmosphere as evidence).
That gaping discrepancy between technical ability and societal wisdom will not sustain for much longer.
This entire forum is a testament to people who are so up their own ass they think we can "innovate" our way out of societal issues, as if the problem with the Luddites was that they didn't know javascript.
I've been saying something like that for a while, but in less Olympian terms.
Look at this first as a consumer protection problem. I've pointed out before a Frontier Airlines presentation directed to investors which specifically calls out AI chatbots as a tool for suppressing customer complaints.[1], page 45: "Today, high-touch. ... Avenue for customer negotiation. Tomorrow, self-service. "Chatbot efficiently answers questions, reduces contacts and removes negotiation." The coming thing, already here for some companies, is customer service where customers can never reach a human.
We already see this with the big ad-supported companies - Google, Facebook, etc. They have the power to make major adverse decisions to a user with no consequences to the company. Few governments have the guts to stand up to that. The EU does, a little.
Government doing this sort of thing is a similar problem, and governments are harder to avoid.
The extinction worry is a distraction from the oppression worry.
[1] https://ir.flyfrontier.com/static-files/c7e0a34d-3659-49cc-8...
Of course, this all hinges on how hard it is to make AGI. People have wildly varying estimates of this, it could be 3 years, or it could be 30 years or more.
Your estimate may be on the higher side. But if we assume that and are wrong and it turns out to be really soon, we will be blindsided.
AI is looking to be a vehicle for yet another consolidation of wealth, and by proxy, power.
Anonymity and obfuscation of crime are killers.
AI/ML governed corporations, hedge funds and private equity will be able to be "Enron + Goldman Sachs + Nestle + Monsanto/Bayer + Purdue Pharma" on literal steroids. You might add AGI to mix to attempt to repair the system that was created by the most powerful sociopathic goal seeking super-toddler that the world has ever seen.
I mean, they can be, unless someone already has total power and then they don't give a shit and do crime in the open. Then they demand the lack of anonymity from you and turn the world into a police state.
Unfortunately that’s what I suspect too.
While wealth is busy stepping on you with its jackboots it is vying with other wealthy people for resources and power. Because AI is something that will increase the wealthy persons power, there is the counter risk that another persons AI will make them even more wealthy if they don't keep improving their AI.
The wealthy really don't want to get in a very expensive AI war with each other if they can avoid it. They'll seek to regulate it enough they can gain power over most, but setup a regulatory system they feel gives them some form of control.
How do we know this? Because we've had literal centuries worth of learned experience backing it! The exact reason why corporate capitalism is so brutally efficient is because there are lots of things we as humans value but don't have a suitable profit motive for. The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition). But that also runs counter to most AI safety proposals, which usually involve some kind of licensing system, which will ensure a handful of companies have the best and most direct access to the benefits of AI.
Yes, that's exactly the problem, which is why this
> The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition).
is backwards. The reason competitive markets generally work out better for consumers than monopolistic ones is because they're stronger optimizers, more ruthless, more capable of crushing all competing values underfoot - and those of us lucky enough to benefit from an inefficient, low-crushing segment of the labor market get to collect some of the resulting surplus.
Nothing about this is inevitable: it's just the way we happen to be positioned in one particular system at one particular moment in history. There's no reason to expect that we would remain safely un-crushed in a far more competitive environment.
Being more specific will allow for more specific responses. If the danger being addressed is the danger from text-prediction software, call it that. If it's something else, then describe it.
Will be interesting to see how global policies develop to deal with it. Probably something pretty bad will happen at some point and then the regulations will overshoot their goal.
One also can't replace nurses with guns to make healthcare more scalable.
There's a mob of dreamy libertarian techbros on HN who are (often wilfully) ignorant of the dangers. I'll start worrying about military robots after we figure out how to prevent the Earth from being dismantled, which very well might transpire first, if and when there's significant architecture breakthroughs.
AI does not need to be evil, it does not need to be conscious, it does not even need to be severely misaligned with human values, and it really doesn't matter if the goal of "eat all the humans" arises from a model hallucination or from a misanthropic troll in his basement typing it into his looped uncensored-LLaMAv3+plugins instance.
Who here with experience does not think these neutered late releases do not tilt the playing field?
"Open"AI was founded assuming that premise. Talk about a 180.
> Do the signatories believe that existing AI systems or their immediate successors might wipe us all out? If they do, then the industry leaders signing this statement should immediately shut down their data centres and hand everything over to national governments. The researchers should stop trying to make existing AI systems safe, and instead call for their elimination.
But a different set of people called for a halt to AI research two months ago, and his response then (https://aisnakeoil.substack.com/p/a-misleading-open-letter-a...) was that these concerns are "fever dreams" and we should disregard them in favor of "serious policy debates".
It seems like there's a self-reinforcing system here, where any evidence or argument in favor of existential risks from AI can be dismissed out of hand.
He is saying "if those people genuinely believed what they were saying, they should logically shut down their own AI systems."
His position is consistent and clear: the supposed threat from AI is based in sci-fi and motivated reasoning. Indeed, even those who are motivated to spread the idea of that threat do not appear to genuinely believe it—either that, or they believe that their own right to continue to do whatever they can to maximize their profits outweighs it.
- Jon Lajoie
Even if we did create some kind of autonomous sentient AI I'd still be more concerned about humans unless its behavior suggested otherwise. Humans are "in the game" with other humans, and for the time being our habitat is limited to habitable areas of Earth. That means we are in competition and are prone to power and control games with one another.
The most logical thing for a superintelligent sentient AI to do would be make a ton of money and buy some rockets and go F right off to somewhere in the solar system with lots of free energy and resources. Why fight with humans over scraps of a little sand grain like Earth when there's a whole universe out there?
I don't think we'll ever have a problem of accidentally trusting all our nuclear launch codes to an LLM.
sure
> "I don't think we'll ever have a problem of accidentally trusting all our nuclear launch codes to an LLM."
i would guess this could happen easier than you think
It's only after decades of scifi and Hollywood movies that people thought, "oh, we could accidentally create a bomb that wants to kill us all".
Both are true.
Frank Herbert addressed it in Dune. Right after Paul is tested by the Reverend Mother, there's a conversation where she says the following:
“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
Sadly, Brian Herbert seems to have turned the Butlerian Jihad (the ancient war against all thinking machines) into a struggle between mankind and AI rather than between mankind and itself.
Or, as I suspect, this is all very murky and nobody knows exactly what the challenge really is beyond, "Continue to not be an asshole."
I am, right now, applying LLMs to domain specific business problems as part of my job. How do I do this safely/sanely, does anyone actually know?
LLMs, as well as other recent outcomes of ANN technologies, are undoubtedly very impressive. Especially for the wide audience. But I feel like impressiveness is the only practical feature they have. When I talk to a support call centre chatbot, I feel like the only useful thing the robot could do for me is to guide me through the verbal user interface in a very verbose way. And I find it lesser convenient and reliable than just clicking a few buttons. ChatGPT can give me encyclopaedic knowledges through a verbal interface too, but I think that googling a Wikipedia article is more straightforward and, again, more reliable.
I think the lack of reliability was the reason why people eventually gave up on the previous generation of these technologies as a form of computer interfaces. The same fate awaits the new generation too, in my opinion. And the LLMs can't offer anything new rather than just a new form of interfaces. They are very bad in true reasoning. Which is why, I think, they are too far from what I would call AGI.
The author of the article is talking about new risks brought by AI. I personally don't see any new risks here, but I see a lot of risks in more traditional technologies no one is talking about. Modern Internet, smartphones and the way the modern computer interfaces designed in general are, in my opinion, already brought a lot of risks to us.
We are slowly loosing control over computer systems. When I open YouTube I would like to organise and control the information that I consume. But instead Google prefers to choose what I should watch. They are slowly removing explicit control functions that I can use directly in favour to suggestions semi-automatically generated by them. And it's not just Google, it's a common trend. Facebook, Windows or Twitter are not exceptions. The control over the users attention as a way to influence, is a main value for big tech and for the people who control these corporations. And I think this is the risk for the humanity we are currently facing.
Will the ANNs, LLMs anyhow advance this harmful trend? I don't know. I doubt so, but it doesn't really matter, because LLMs themselves is not a root of the threat. The root threat is that we, the users, are loosing control over computers that we use.
Somehow, the public lets them off the hook for 'stepping forward' and preaching that it should be regulated, while they continue to risk the future of humanity.
This is where you lose me. AGI (not LLMs, but intelligence smarter than us) is qualitatively different from anything else we've ever made. The one example we have of this in history is the development of human intelligence. It did not work out well for other species, and now we own the Earth. This isn't a perfect analogy! There are some reasons why AGI may be better (we'll be actively trying to align it to human interests) but also why it may be worse (there will be pressure to make it more powerful and take off guardrails).
There may be good reasons for why we'll be able to control AGIs, but "history of technology suggests..." is not one of them.
And I'm not arguing that all the "human misuse" problems are not important! They're hugely important! A likely outcome, the way we're going, is that a bunch of people get really rich and everyone else becomes super poor and possibly starves, and wars break out. I just think that, after that, AGIs also end up in charge of the planet and from there we either die or we have no say in how things go from that point on. I don't object to working on the former problem, except to the degree that the solution makes the latter problem worse (like, "open source all the models", which is likely to happen at some point anyway).
Rich getting richer [x]
Poor starving [x]
War [x]
Powerful gain more power [x]
Based on how I already see the world, honestly, I'm not sure that's much of a prediction.
I look forward to a "day the earth stood still"-like overlord that punishes aggression, greed and corruption: current human systems seem to be inadequate at solving these issues.
If you mean AGI in the sense of singularity, where it can self improve and get smatter, the question of that even being able to happen in the first place relies on P=NP being proven first.
If you're an oppressive regime, you want to control public opinion and culture. Hence, one reason the public can't have free speech AI is that it could correctly identify who's oppressing them, and misinforming them.
Have everyone watch War Games and make sure there's a plug.
Indeed the strongest point in favor of being critical of those concerned about existential risk over plain old grift and malfeasance and incompetence, is that the leverage provided by AI makes the latter itself an existential risk.
[0] Sam Altman on how to survive a nuclear war
Imagine a confident ten year old who has just learnt the rules of chess or Go.
Could this person explain or quantify how the best player in the world might defeat them?
This logic fails because we have never had something quite like AI.
If you think it's dangerous when bad people control an AI, how dangerous do you think it will be when an AI that has no concept of good or bad controls itself? An AI whose values are more alien than that of any sociopath, completely orthogonal to human values?
If you don't think that is going to happen, then make a convincing argument for your case, the way AI "doomers" provide well thought-out arguments for their case:
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
https://astralcodexten.substack.com/p/why-i-am-not-as-much-o...
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
Widespread adoption of the existing technology will cause the problems in the posted article.
It’s not just big bad guys.
I wrote a privacy policy with it…
But people generally look at that as "regulating cars", not "regulating AI."
And the failure mode there is not at all what's being bandied about as a nightmare AGI scenario—there, it's an AI that has chosen to kill people, where with Tesla, it's just an extremely stupid AI that doesn't know any difference.
I submit that it is not anything special, certainly pales in comparison to the atomic bomb. It's merely that we live in that weird end tail of the age of ideology where strong overarching ideologies came to (and are coming to) an end, where we're all scattered to the various geopolitical winds and in the absence of a strong guiding state and vision (communism vs capitalism) become more extreme and perturbed.
If you do want regulation, It should be something proper hashed out diplomatically and economically between Nation States and their respective blocs, no crony capitalist allowed, sorry. The rest of the world shan't be going along with something that economically cripples them.
Idols say what the priests want.
That's only partially true, and only at first. "Those that make them are like them, so are all who trust in them" (Ps 115). To promulgate idols is necessarily destroy one's own agency. And the deeper you go the worse it gets.
I agree, I'm not scared at all of the machines themselves, but the demented framing of what they are. All these "AI apocalypse" people are so convinced that machines have the power to end death and suffering that can't help but construct these horrible hell-stories about the vengefulness of their gods. The thing that makes people like Yudkowsky effective is that their fear is based on a obviously sincere life of devotion.