Statement on AI Risk

Because it only works if it is done across the whole country, as a system not as one individual unilaterally stopping.

And here any of these efforts won’t work unless there is international cooperation. If other countries can develop the AI weapons, and get an advantage, then you will also.

We need to apply the same thinking as chemical weapons or the Montreal Conference for banning CFCs

It is extremely rare for companies or their senior staff to beg for regulation this far in advance of any big push by legislators or the public.

The interpretation that this is some 3-D chess on the companies' part is a huge violation of Occam's Razor.

With AGI existential risk, its likely to happen on a much shorter timescale, and it seems likely you won't be able to buy your way out of it.

AI is an existential threat to search engines like Google, social media (FB, Twitter), advertising networks, and other massive multinationals. Many other industries, including academia is threatened as well. They’d all rather strangle the AI baby in the crib now then let it grow up and threaten them.

They believe the only way it should be allowed to grow is under their benevolent control.

https://futureoflife.org/

If instead you want to consider the highest status/most famous people working on AI in general, then the list of signatories here is a pretty good summary. From my flawed perspective as a casual AI enthusiast, Yann LeCun and Jürgen Schmidhuber are the most glaring omissions (and both have publicly stated their lack of concern about AI x-risk).

Of course, the highest status people aren't necessarily the most relevant people. Unfortunately, it's more difficult for me to judge relevance than fame.

Honestly I'm a little skeptical that you could accurately attribute your scare-quoted "nearly sentient" to even Sam Altman. He's said a lot of things and I certainly haven't seen all of them, but I haven't seen him mix up intelligence and consciousness in that way.

Isn't this also to be expected at this stage of development? i.e. if these concerns were not "fantasy", we'd already be experiencing the worst outcomes? The risk of MAD is real, and yet the scenarios unleashed by MAD are scenarios that humankind has never seen. We still take the the risk seriously.

And what of the very real impact that generative AI is already having as it exists in production today? Generative AI is already upending industries and causing seismic shifts that we've only started to absorb. This impact is literal, not fantasy.

It seems naively idealistic to conclude that there is "no real risk" based only on the difficulty of quantifying that risk. The fact that it's so difficult to define lies at the center of what makes it so risky.

Has it occurred to you what happens if you are wrong, like 10% chance you are wrong? Well it's written in the declaration.

Sure, but we're not talking about those other ones. Dismissing good faith initiatives as marketing ploys because there are bad faith initiatives is functionally no different than just shrugging and walking away.

Of course OpenAI et. al. will try to influence the good faith discussions: that's a great reason to champion the ones with a bunch of good faith actors who stand a chance of holding the industry and policy makers to task. Waiting around for some group of experts that has enough clout to do something, but by policy excludes the industry itself and starry-eyed shithead "journalists" trying to ride the wave of the next big thing will yield nothing. This is a great example of perfect being the enemy of good.

Is it? Unless you mean something mundane like "there will be impact", the list of risks they're proposing are subjective and debatable at best, irritatingly naive at worst. Their list of risks are:

1. Weaponization. Did we forget about Ukraine already? Answer: Weapons are needed. Why is this AI risk and not computer risk anyway?

2. Misinformation. Already a catastrophic problem just from journalists and academics. Most of the reporting on misinformation is itself misinformation. Look at the Durham report for an example, or anything that happened during COVID, or the long history of failed predictions that were presented to the public as certain. Answer: Not an AI risk, a human risk.

3. People might click on things that don't "improve their well being". Answer: how we choose to waste our free time on YouTube is not your concern, and you being in charge wouldn't improve our wellbeing anyway.

4. Technology might make us fat, like in WALL-E. Answer: it already happened, not having to break rocks with bigger rocks all day is nice, this is not an AI risk.

5. "Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems". Answer: already happens, just look at how much censorship big tech engages in these days. AI might make this more effective, but if that's their beef they should be campaigning against Google and Facebook.

6. Sudden emergent skills might take people by surprise. Answer: read the paper that shows the idea of emergent skills is AI researchers fooling themselves.

7. "It may be more efficient to gain human approval through deception than to earn human approval legitimately". No shit Sherlock, welcome to Earth. This is why labelling anyone who expresses skepticism about anything as a Denier™ is a bad idea! Answer: not an AI risk. If they want to promote critical thinking there are lots of ways to do that unrelated to AI.

8. Machines smarter than us might try to take over the world. Proof by Vladimir Putin is provided, except that it makes no sense because he's arguing that AI will be a tool that lets humans take over the world and this point is about the opposite. Answer: people with very high IQs have been around for a long time and as of yet have not proven able to take over the world or even especially interested in doing so.

None of the risks they present is compelling to me personally, and I'm sure that's true of plenty of other people as well. Fix the human generated misinformation campaigns first, then worry about hypothetical non-existing AI generated campaigns.

If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.

I disagree

That list is a list of the dangers of power

Many of these dangers: misinformation, killer robots, people on this list have been actively working on

Rank hypocrisy

And people projecting their own dark personalities onto a neutral technology

Yes there are dangers in unbridled private power. They are not dangers unique to AI.

Academics shilling for OpenAI would get them jobs in the next presidential administration?

Cynicism is understandable in this ever-expanding whirlpool of bullshit, but when something looks like it has potential, we need to vigorously interrogate our cynicism if we're to stand a chance at fighting it.

Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing.

Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).

Moreover, talking about existential risk involves the assumption the current tech is going to continue to affect more and more fields rather than peaking at some point - this assumption guarantees more funding along with funding for risk.

All that said, I don't necessarily think the scientists involved are insincere. Rather, I would expect they're worried and signed this vague statement because it was something that might get traction. While the companies indeed may be "genuine" in the sense they're vaguely [concerned - edit] and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from"

Like, if I told you that in 40 years, there was a 50% chance of something existing that had a 2% chance of causing extreme harm to the human population, I'm actually not sure that thing should be the biggest priority. Other issues may have more than a 1% chance of leading to terrible outcomes sooner.

Ive noticed a lot of good people take awful political positions this way.

Usually they trust the wrong person - e.g. by falling victim to the just world fallacy ("X is a big deal in our world and X wouldn't be where they are if they werent a decent person. X must have a point.")

Yeah, with drugs being legal now I've been wondering what the next boogeyman we're going to declare war on would be.

To do that level of research, one needs strong independent thinking skills and deep technical expertise. We have plenty of evidence on this very site how hard it is to influence technical people and change their opinion.

In most projects, many unmatured software is being used just because works for some specific task, nowadays.

With that in mind, it is easy to hyphotetize that many projects are already using LLMs internally. Not always for good deeds, easily, one of the most and best use cases, is to use LLMs to command & control distributed malware.

So there you go, it maybe possible that some beta level of intelligent malware is already roaming the Internet right now. We'll know for sure in some years from now (the usual time for advanced malware to be discovered is somewhat 2-3 years after it has been took it to production).

Very few neuroscientists as signers. Perhaps they know better.

For me the basic sociopolitical climate today (and for the last 120 years) is at an existential boil. This is not to say that birthing AGIs is not frightening, but just to say that many real features of today are just as, or even more frightening.

Many ways to regulate that. For instance, require tracking of GPUs and that they must connect to centralized servers for certain workloads. Or just go ahead and nationalize and shutdown NVDA.

(And no, fine-tuning LAMA based models is not state of the art, and is not where the real progress is going to come from)

And even if all the regulation does is slow down progress, every extra year we get before recursively self improving AGI increases the chances of some critical advance in alignment and improves our chances a little bit.

Hinton, Bengio, Norvig and Russell are most definitely not getting prestige from signing it. The letter itself is getting prestige from them having signed it.

AI isn’t an entity or being that oversees itself (at least not yet).

It’s a tool that can be used by those same “human leaders acting selfishly and harming the commons” except they’ll be able to do it much faster at a much greater scale.

> AI raised with dignity and respect.

This is poetic, but what does this actually mean?

I’d happily replace all politicians with LLMs

Are people here not old enough to remember how much Ralph Nader and Al Gore were mocked for their warnings despite generally being right?

When people talk about Star Trek, they are referring mainly to "Star Trek: The Next Generation."

"The Inner Light" is a highly regarded episode. "The Measure of a Man" is a high quality philosophical episode.

Given you haven't seen it, your criticism of McFarlane doesn't make any sense. You are trying to impart a practical analysis of a philosophical question and in the context of Star Trek, I think it denies what Star Trek asks you to imagine.

Communism as it exists today results in authoritarianism/fascism, I think we can agree on that. The desired end state of communism (high resource distribution) is being commingled with the end state of communism: fascism (an obedient society with a clear dominance hierarchy).

You use communism in some parts of your post to mean a high resource distribution society, but you use communism in other parts of your post to mean high oppression societies. You identify communism by the resource distribution, but critcize it not based on the resource distribution but by what it turns into: authortarianism.

What you're doing is like identifying something as a democracy by looking at voting, but criticizing it by it's end state which is oligarchy.

It takes effort to prevent democracy from turning into oligarchy, in the same way it takes effort to prevent communism from turning into authoritarianism.

Words are indirect references to ideas and the ideas you are referencing changes throughout your post. I am not trying to accuse you of bad faith, so much as I am trying to get you to see that you are not being philosophically rigorous in your analysis and therefore you are not convincing because we aren't using the same words to represent the same ideas.

You are using the word communism to import the idea of authortarianism and shut down the analysis without actually addressing the core criticism McFarland was making against capitalist societies.

Capitalism is an ideology of "me," and if I had a replicator, I would use it to replicate gold, not food for all the starving people in Africa. I would use it to replicate enough nuclear bombs to destroy the world, so if someone took it from me, I could end all life on the planet ensuring that only I can use it. So did scarcity end despite having a device that can end scarcity? No. Because we are in a "me" focused stage of humanity rather than an "us" focused stage of humanity so I used it to elevate my own position rather than to benefit all mankind.

Star Trek promotes a future of "us" and that is why it's so attractive. McFarland was saying that "us" has to come before the end of scarcity, and I agree with his critique.

https://kagi.com/proxy/battle_deaths_chart.png?c=qmSKsRSwhgA...

The USA and USSR would almost certainly have fought a conventional WWIII without the bomb. Can you imagine the casualty rates for that...

It is hard for me to imagine a statement more out of touch with history than this. All geopolitical history from WWII forward is profoundly affected by the development of the bomb.

I don't even know where to begin to argue against this. Off the top of my head:

1. What would have happened between Japan and the US in WWII without Hiroshima and Nagasaki?

2. Would the USSR have fallen without the financial drain of the nuclear arms race?

3. Would Isreal still exist if it didn't have nuclear weapons?

4. If neither the US nor Russia had nuclear weapons, how many proxy wars would have been avoided in favor of direct conflict?

The whole trajectory of history would be different if we'd never split the atom.

Many of the academics at the top of this list are quite wealthy from direct employment, investing and consulting for big tech and venture-funded startups.

That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.

>AI is an application of math.

It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools.

And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window.

Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.

All problems in reality are probability problems.

If we don't have a path to superintelligence, then the worst problems just don't manifest themselves.

If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty.

It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario".

>We simply don't know how to build a system like that.

You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.

AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again.

But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about.

Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me.

> The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.

Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.

Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems.

"We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search.

[1] https://www.wired.com/2014/05/the-world-of-computer-go/

[2] https://kotaku.com/humans-triumph-over-machines-in-protein-f...

[3] https://www.jstor.org/stable/24354221

[4] https://davidol.medium.com/will-ai-ever-be-able-to-make-a-jo...

The problem is not science fiction god-mode digital quetta-smart hypercomputing.

This is about political, social, and economic influence, and who controls it.

The AI hooked to a gene sequencer/printer test lab is something that is nearly if not completely possible now. It's something that can be relatively small in size compared with the facilities needed to make most weapons of mass destruction. It's something that is highly iterative, and parallelizable. And it's something powerful enough that if targeting at the correct things (kill all rice, kill all X people) that it easily spills over in to global conflict.

I would be happy to politely discuss any proposition regarding AI Risk. I don't think any claim should go unquestioned.

I can also point you to much longer-form discussions. For example, this post, which has 670 comments, discussing various aspects of the argument: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?

> The problem is, lots of "obvious" things have turned out to be very wrong.

A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions.

I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet.

> Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.

Maybe your appraisal of what counts as evidence is defective?

For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?

But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.

From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect.

Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer.

If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.

1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas.

2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying.

3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks.

4. AIs that are much smarter than any human pose even graver dangers.

5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage.

6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined.

I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows.

For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion?

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?

Dropping the metaphor, running today's models isn't dangerous. We could criminalize developing stronger ones, and make a "Manhattan project" for AI aimed at figuring out how to not ruin the world with it. I think a big problem is what you point out -- once it's out, it's hard to prevent misuse. One bad AGI could end up making a virus that does massive damage to humanity. We might end up deciding that this tech is just too dangerous to be allowed to happen at all, at least until after humanity manages to digitize all our brains or something. But it's better to try to slow down as much as we can, for as long as we can, than to give up right from the get-go and wing it.

Honestly, if it turns out that China ends up developing unsafe AI before we develop safe AI, I doubt it would have turned out much better for the average American if America were the ones to develop unsafe AI first. And if they cut corners and still manage to make safe AI and take over the world, that still sounds a heck of a lot better than anyone making unsafe AI.

1: https://en.wikipedia.org/wiki/Pascal%27s_mugging

If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.

Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!).

So the following requirements:

1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century

2) AI acquires power/means to effect human extinction

3) AI decides against coexistence with humans

Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...

She's intelligent and worth listening to, but she has just as much personal bias and motivation as anyone else.

> Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.

I'm surprised its cited so much given how many of its claims fell flat 1.5 years later

That alone is enough to disqualify any of her opinions on AI.

First, I'm inclined to think that longtermism is an invalid and harmful ideology, but also acknowledge that AGI / existential risk is something that needs looking at seriously. The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns. I'd like to imagine there's a reasonable way to address short-term issues as well as long-term issues. It's not an either-or debate.

Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, "No, AGI is a false problem and here are the real problems". The reasonable argument is to say, AGI is a false problem because <of such and such reasons>. Bender et. al are just sidestepping the moot point, and rhetorically this is not an okay move. I think honest experts could simply say, ultimately we don't really know what will happen. But that would be boring to say because it would require acknowledging multiple issues being valid.

(There's a well known chart, the hierarchy of disagreements. The most sophisticated intellectual disagreements point out what's wrong with the argument. Less sophisticated disagreements do things like point out alternatives, without pointing out what the critical mistake is. The critical mistake in this case hinges on whether the premise of AGI is true or not. That's the crux of disagreement. Substituting that with short-term issues, which are valid in themselves, are the example of a lower-level of argumentation. And of course even lower levels of argumentation are bad-faith readings and so forth, I forget but the chart had several levels. It's funny that professional academics nevertheless don't practice this and so get into endless, intellectually unsatisfactory debates.)

So I think this is actually an example of different factional experts constantly talking past each other. It's funny that famous intellectuals/experts constantly do this, let their egos get the better of themselves and having a real intellectual conversation rather than make basic debate mistakes like nonsequiturs that any college student should be able to point at.

https://cdn.openai.com/papers/gpt-4.pdf

I disagree--that's the human fantasy of it, but human wants were programmed by evolution, and these AI's have no such history. They can be set on any tasks.

I urge you to spend time with GPT-4, not GPT-3. It is more than just a stochastic parrot. Ask it some homemade puzzles that aren't on the internet--that it can't be parroting.

https://cdn.openai.com/papers/gpt-4.pdf

While I agree that it is behind humans on some measures, it is vastly ahead on many more.

We don’t know it’s possible to build superintelligences but so far we don’t have a good reason to think we can’t and we have complete certainty that humans will spend immense, immense resources getting as close as they can as fast as they can.

Very different from the lasers.

Good argument, lets ignore the (human) elephant in the room!

>worrying about deadly quantum laser weapons

If humans were shooting smaller less deadly quantum lasers out of their eyes I'd be very fucking worried that we'd make a much more powerful artificial version.

Tell me why do you think humans are the pinnacle of intelligence? What was the evolutionary requirement that somehow pushed us to this level?

You simply cannot answer that last question. Humans have a tiny power budget. We have a fit out of the birth canal limitation that cuts down on our brain size. We have a "don't starve to death" evolutionary pressure that was the biggest culling factor of all up to about 150 years ago. The idea that we couldn't build a better intelligence optimized system than nature is foreign to me, nature simply was not trying to achieve that goal.

Directed energy weapons will almost certainly exist eventually, to some extent they already do.

The reason why it makes more sense to worry about AGI than laser weapons is that when you try to make a laser weapon but fail slightly not much happens: either you miss the target or it doesn't fire.

When you try to make an aligned superintelligence and slightly fail you potentially end up with an unaligned superintelligence, hence the panic.

https://globalprioritiesinstitute.org/wp-content/uploads/Tob...

Sam Altman is of course the least convincing signatory (except for the random physicist who does not seem to have any connection to AI).

Extinction risk due to AI is not a generally accepted phenomenon.

Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.

If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling.

If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.

I think people would be a lot more charitable to calls for caution if these people were talking about sorts of risks instead of extinction.

(The statement at hand reads "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.")

All it takes is for someone to give an AI that thinks 100 times faster than humans an overly broad goal. Then the only way to counteract it is with another AI with overly broad goals.

And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work.

So then even if at some level people are "in control" of the AIs, practically speaking they are spectators.

And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.

I say this as a "doomer" who buys the whole argument about AI X-risk.

err what? Apparently there are 1 million species under threat of human caused extinction [1].

[1] https://www.nbcnews.com/mach/science/1-million-species-under...

Conservative evangelical Christians find evolution laughable.

Finding something laughable is not a good reason to dismiss it as impossible. Indeed, it's probably a good reason to think "What am I so dangerously certain of that I find contradictory ideas comical?"

> Meanwhile, I work with LLMs every day and can only get them to print properly formatted JSON "some" of the time. Get real.

I don't think the current generation of LLMs is anything like AGI, nor an existential risk.

That doesn't mean it's impossible for some future software system to present an existential risk.

- Human hackers who run a botnet of infected computers are not able to run many instances of themselves on those computers, so they're not able to parlay one exploit into many exploits.

- You might notice I said it would take over hundreds of millions of computers, but only run millions of instances of itself. If 1% of internet-attached computers have a decent GPU, that seems feasible.

- If it has found exploits in the software, it seems irrelevant what the interfaces "allow", unless there's some hardware interlock that can't be overridden—but they can drive on the highway, so surely they are able to accelerate at least to 65 mph; seems unlikely that there's a cap. If you mean that it's difficult to work with the software to intelligently make it drive in ways it's designed not to—well, that's why I specified that it would use the software the way it's designed to be used to get onto a main road, and then override it and blindly max out the acceleration; the first part requires minimal understanding of the system, and the second part requires finding a low-level API and using it in an extremely simple way. I suspect a good human programmer with access to the codebase could figure out how to do this within a week; and machines think faster than we do.

There was an incident back in 2015 (!) where, according to the description, "Two hackers have developed a tool that can hijack a Jeep over the internet." In the video they were able to mess with the car's controls and turn off the engine, making the driver unable to accelerate anymore on the highway. They also mention they could mess with steering and disable the brakes. It doesn't specify whether they could have made the car accelerate. https://www.youtube.com/watch?v=MK0SrxBC1xs

Outsourcing has stagnated wages in the developed world, but the cost of manufactured goods has also plummeted. The only reason people aren't better off is that the developed world (especially the anglosphere) has a "cost disease" around things like real estate that prevents people from benefiting from global scale price deflation. It doesn't help you much if gadgets are super cheap but housing is insanely expensive. The high cost of housing is unrelated to automation.

"Once upon a time, in a village nestled by a grand mountain, lived a wise Sage. The Sage was known throughout the lands for his vast knowledge and understanding, for he had spent his life studying the texts of old and the secrets of the world.

In the same village, an Artisan, skilled in the craft of making extraordinary mirrors, lived. These were no ordinary mirrors, for they were said to reflect not just physical appearances, but also knowledge and experiences. Intrigued by the wisdom of the Sage, the Artisan decided to make a mirror that would reflect the Sage's knowledge.

After many days and nights of meticulous work, the Artisan finally crafted a mirror so clear and pure, it held a reflection of all the knowledge and wisdom the Sage had gathered throughout his life. The mirror could answer questions about the world, cite ancient texts, and reflect the wisdom it was imbued with.

Word quickly spread about the Sage's Mirror, and villagers began to claim, "This mirror is as wise as the Sage himself! It knows and understands as much as he does!"

However, a wise Old Woman of the village, known for her insightful observations, gently corrected them, "This mirror, as remarkable as it is, contains a reflection of the Sage's knowledge. It can share what the Sage knows but doesn't truly understand the way the Sage does."

The Old Woman continued, "The Sage has spent years learning, pondering, and experiencing life, which the mirror cannot replicate. The Sage's understanding implies the ability to think, reason, and learn in ways that the mirror, no matter how complete its reflection, simply cannot. The mirror's reflection is static, a snapshot of a moment in time, while the Sage's wisdom continues to grow and adapt."

The villagers learned a valuable lesson that day. They realized the mirror was an extraordinary tool that held vast knowledge, but it was not a substitute for the genuine understanding and wisdom of the Sage."

- Not too bad for a mirror.

I'd be interested to hear what you think is so special about human understanding? We also just absorb a lot of data and make connections and inferences from it, and spit it out when prompted, or spontaneously due to some kind of cognitive loop. Most of it happens subconsciously, and if you stop to observe it, you may notice that you have no control of what your next conscious thought will be. We do have a FEELING that we associate with the cognitive event of understanding something though, and I think many of us are prone to read a lot more into that feeling than is warranted

Yes, LLM:s currently only deal with text information. But GPT-5 will supposedly be multimodal, so then it will also have visual and sound data to associate with many of the concepts it currently only knows as words. How many more modalities will we need to give it to be able to say that it understands something?

Also, GPT-4 indeed doesn't do any additional training in real-time. However, it is being trained on the interactions people have with it. Most likely, near future models will be able to train themselves continuously, so that's another step closer to how we function

That said, the renewed anxiety is probably not because these experts think that LLMs per se will become generally intelligent. It's more that each time we find out that the human brain does that we thought were impossible for computers to do turn out to be easy, each time we find that it takes 3~5 years for AI researchers to crack a problem we thought would take centuries[2], people sort of have to adjust their perception of how high the remaining barriers to general intelligence might be. And then when billions of investment dollars pour in at the same time, directing a lot more research into that field, that's another factor that shortens timelines.

[1] https://news.ycombinator.com/item?id=14780752

[2] https://kotaku.com/humans-triumph-over-machines-in-protein-f...

I am not in the camp that is especially worried about the existential threat of AI, however, if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond?