Hacker News Reader

Comment by chefandy | Hacker News Reader

I'm eternally skeptical of the tech business, but I think you're jumping to conclusions, here. I'm on a first-name basis with several people near the top of this list. They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met. These folks default to skepticism of the tech business, champion open data, are deeply familiar with the risks of regulatory capture, and don't sign their name to any ol' open letter, especially if including their organizational affiliations. If this is a marketing ploy, that must have been a monster because even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks.

View on HN

Maybe these people have good intentions and are just being naive

They might not be getting paid, but that doesn’t mean they are not being influenced

AI at this point is pretty much completely open, all the papers, math and science behind it are public

Soon, people will have advanced AI running locally on their phones and watches

So unless they scrub the Internet, start censoring this stuff, and pretty much ban computers, there is absolutely no way to stop AI nor any potentially bad actors from using it

The biggest issues that we should be addressing regarding AI are the potential jobs losses and increased inequality at local and global scale

But of course, the people who usually make these decisions are the ones that benefit the most from inequality, so

You don't have to be a mustache-twirling villain to have the same effect.

>Maybe these people have good intentions and are just being naive

Ive noticed a lot of good people take awful political positions this way.

Usually they trust the wrong person - e.g. by falling victim to the just world fallacy ("X is a big deal in our world and X wouldn't be where they are if they werent a decent person. X must have a point.")

> Soon, people will have advanced AI running locally on their phones and watches

Yeah, with drugs being legal now I've been wondering what the next boogeyman we're going to declare war on would be.

You are attributing naivety on a technical matter to many people who have done world class research and some of them are even a pioneer of their subfield of study.

To do that level of research, one needs strong independent thinking skills and deep technical expertise. We have plenty of evidence on this very site how hard it is to influence technical people and change their opinion.

Not attributing anything, just pointing out an alternative explanation to the one that it is given in the comment that it is responding to

However, you seem to be attributing super-human capabilities to a whole group of people for some reason

One does not need to be have super-human capabilities to foresee potential risks of the technology one is an expert in.

Yes, the current version of AI is not capable of large scale harm by itself yet, but the plausible trajectory is worth warning about. Gordon Moore did make fairly accurate predictions after all.

> One does not need to be have super-human capabilities to foresee potential risks of the technology one is an expert in

You are right. And no one is arguing the contrary

We can all see the risks

The thing is, what do we do about it

Regulating AI will only serve to help the big players and keep others out

That would depend on the specifics of the regulation though.

I do agree that the likelihood of regulatory capture in this arena is very high.

I sure when openAI converted to successful company, we had lost the game. AGI had won whole world. openAI was the organisation should stop AI damage.

There's range of functionality in software technology, i.e. some new framework, may have been outputed fully developed and internally debugged for years in some startup, or you could have a really almost not even a production level software, just uploaded to github.

In most projects, many unmatured software is being used just because works for some specific task, nowadays.

With that in mind, it is easy to hyphotetize that many projects are already using LLMs internally. Not always for good deeds, easily, one of the most and best use cases, is to use LLMs to command & control distributed malware.

So there you go, it maybe possible that some beta level of intelligent malware is already roaming the Internet right now. We'll know for sure in some years from now (the usual time for advanced malware to be discovered is somewhat 2-3 years after it has been took it to production).

It’s worth noting also that many academics who signed the statement may face adverse issues like reputational risk as well as funding cut to their research programs if AI safety becomes an official policy.

For a large number of them, these risks are worth far more than any possible gain from signing it.

When a large number of smart, reputable people, including many with expert knowledge and little or negative incentives to act dishonestly, put their names down like this, one should pay attention.

Added:

Paul Christiano, a brilliant theoretical CS researcher who switched to AI Alignment several years ago, put the risks of “doom” for humanity at 46%.

https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...

Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left.

Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing.

Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).

Even if it’s just Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, we’d probably agree the risks are not negligible. There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. Many of whom do not work for those companies.

We’re talking about nuclear war level risks here. Even a 1% chance should definitely be addressed. As noted above, Paul Christiano who has worked on AI risk and thought about it for a long time put it at 46%.

> There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well.

I know the Stanford researchers the most and the “biggest names” in LLMs from HAI and CRFM are absent. It would be useful to have their perspective as well.

I’d throw MetaAI in the mix as well.

Merely pointing out that healthy skepticism here is not entirely unwarranted.

> We’re talking about nuclear war level risks here.

Are we? This seems a bit dramatic for LLMs.

LLM is already a misnomer. Latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence.

Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do?

More on the risks of “doom”: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...

I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI.

This is hardly the first time in history a new technological advancement may be used for nefarious purposes.

It’s a discussion worth having as AI advances but if [insert evil actor] wants to cause harm there are many cheaper and easier ways to do this right now.

To come out and say we need government regulation today does stink at least a little bit of protectionism as practically speaking the “most evil actors” would not adhere to whatever is being proposed, but this would impact the competitive landscape and the corporations yelling the loudest right now have the most to gain, perhaps coincidence but worth questioning.

I’m not sure there is a way for someone to engineer a deadly virus while completely innoculating themselves from it.

Short-term AI risk likely comes from a mix of malicious intent and further autonomy that causes harm the perpetrators did not expect. In the longer run, there is a good chance of real autonomy and completely unexpected behaviors from AI.

Why do you have to inoculate yourself from it to create havoc? Your analogy of “nuclear war” also has no vaccine.

AI autonomy is a hypothetical existential risk, especially in the short term. There are many non-hypothetical existential risks including actual nuclear proliferation and escalating great power conflicts happening right now.

Again my point being that this is an important discussion but appears overly dramatized, just like there are people screaming doomsday there are also equally qualified people (like Yann LeCun) screaming BS.

But let’s entertain this for a second, can you posit a hypothetical where in the short term a nefarious actor can abuse AI or autonomy results in harm? How does this compare to non-AI alternatives for causing harm?

> I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI.

That's what interesting to me. People fearmongering about bioengineering and GMO's were generally dismissed as being anti-science and holding humankind back (or worse, that there opposition to progress meant they had blood on their hands). Yet many of the people who mocked them proved themselves to be even more dogmatic and apocalyptic, while being much closer to influencing regulations. And the technology they're fear-mongering about is even further from being able to harm people than biotech is. We are actually able to create harmful biotech today if we want; we don't know when we'll ever be able to create AGI, and if it would even pose a danger if we did.

This mentality - "there could be a slight chance research into this could eventually lead to apocalyptic technology, no I don't have any idea how but the danger is so great we need a lot of regulation" - would severely harm scientific growth if we applied it consistently. Of course everyone is going to say "the technology I'm afraid of is _actually_ dangerous, the technology they're afraid if isn't." But we honestly have no clue when we're talking about technology that we have no idea how to create at the moment.

Counterpoint: CRISPR only reignited what was already a real fear of reduced difficulty and costs of engineering deadly pathogens.

In fact, what you and GP wrote is baffling to me. The way I see it, biotech is stupidly obviously self-evidently dangerous, because let's look at the facts:

- Genetic engineering gets easier and cheaper and more "democratized"; in the last 10 years, the basics were already accessible to motivated schools and individual hobbyists;

- We already know enough, with knowledge accessible at the hobbyist level, to know how to mix and match stuff and get creative - see "synthetic biology";

- The substrate we're working with is self-replicating molecular nanotechnology; more than that, it's usually exactly the type that makes people get sick - bacteria (because they're most versatile nanobots) and viruses (because they're natural code injection systems).

Above is the "inside view"; for "outside view", I'll just say this: the fact that "lab leak" hypothesis of COVID-19 was (or still is?) considered to be one of the most likely explanations for the pandemics already tells you that the threat is real, and consequences are dire.

I don't know how can you possibly look at that and conclude "nah, not dangerous, needs to be democratized so the Evil Elites don't hoard it all".

There must be some kind of inverse "just world fallacy" fallacy of blaming everything on evil elites and 1%-ers that are Out To Get Us. Or maybe it's just another flavor of the NWO conspiracy thinking, except instead the Bildenbergs and the Jews its Musk, Bezos and the tech companies.

Same is, IMHO, with AI. Except that one is more dangerous because it's a technology-using technology - that is, where e.g. accidentally or intentionally engineered pathogens could destroy civilization directly, AI could do it by using engineered pathogens - or nukes, or mass manipulation, or targeted manipulation, or ... countless other things.

EDIT:

And if you ask "why, if it's really so easy to access and dangerous, we haven't already been killed by engineered pathogens?", the answer is a combination of:

1. vast majority of people not bearing ill intent;

2. vast majority of people being not interested and not able to perform (yet!) this kind of "nerdy thing";

3. a lot of policing and regulatory attention given to laboratories and companies playing with anything that could self-replicate and spread rapidly;

4. well-developed policies and capacity for dealing with bio threats (read: infectious diseases, and diseases in general);

5. this being still new enough that the dangerous and the careless don't have an easy way to do what in theory they already could.

Note that despite 4. (and 3., if you consider "lab leak" a likely possibility), COVID-19 almost brought the world down.

Great points. Will just add a point 1.5: There's usually an inverse correlation between ill intent and competence, so the subset of people who both want to cause harm to others on a mass scale and who are also able to pull it off is small

This gets countered by running one (or more) of those same amazing autonomous agents locally for your own defense. Everyone's machine is about to get much more intelligent.

“…some intelligence…” appears to be a huge leap from where we seem to be though.

> > We’re talking about nuclear war level risks here.

> Are we? This seems a bit dramatic for LLMs.

The signed statement isn't about just LLMs in much the same way that "animal" doesn't just mean "homo sapiens"

I used LLM because the people shouting the loudest come from a LLM company which claimed their newest language model can be used to create bioweapons in their whitepaper.

Semantics aside the recent interest in AI risk was clearly stimulated by LLMs and the camp that believes this is the path to AGI which may or may not be true depending who you ask.

I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively.

They've both been loudly concerned about optimisers doing over-optimisation, and society having a Nash equilibrium where everyone's using them as hard as possible regardless of errors, since before it was cool.

While true the ones doing media tours and speaking the most vocally in May 2023 are the LLM crowd.

I don’t think it’s a mischaracterization to say OpenAI has sparked public debate on this topic.

To the extent that this may be true (I've not exactly studied which public thinkpiece writers care about which AI so it might easily be the image generators that get this crown for all I know), that's because ChatGPT actually does something that a normal person understands.

A paper titled "Dual use of artificial-intelligence-powered drug discovery" (published last year) got a few angst pieces and is mostly forgotten by the general public and media so far as I can tell; but the people behind it both talked directly to regulators and other labs to help advise them how many other precursor chemicals were now potential threats, and also went onto the usual podcasts and other public forums to raise awareness of the risk to other AI researchers.

The story behind that was "ugh, they want us to think about risks… what if we ask it to find dangerous chemicals instead of safe ones? *overnight* oh no!"

> I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively.

Whenever Yudkowsky comes up on my Twitter feed I'm left with an impression that I'm not going to have any more luck conversing AI with those in his orbit than I am discussing the rapture with a fundamentalist Christian. For example, the following Tweet[1]. If a person believes this is from a deep thinker that should be taken very seriously rather than an unhinged nutcase, our worldviews are probably too far apart to ever reach a common understanding:

> Fools often misrepresent me as saying that superintelligence can do anything because magic. To clearly show this false, here's a concrete list of stuff I expect superintelligence can or can't do:

> - FTL (faster than light) travel: DEFINITE NO

> - Find some hack for going >50 OOM past the amount of computation that naive calculations of available negentropy would suggest is possible within our local volume: PROBABLE NO

> - Validly prove in first-order arithmetic that 1 + 1 = 5: DEFINITE NO

> - Prove a contradiction from Zermelo-Frankel set theory: PROBABLE NO

> - Using current human technology, synthesize a normal virus (meaning it has to reproduce itself inside human cells and is built of conventional bio materials) that infects over 50% of the world population within a month: YES

> (note, this is not meant as an argument, this is meant as a concrete counterexample to people who claim 'lol doomers think AI can do anything just because its smart' showing that I rather have some particular model of what I roughly wildly guess to be a superintelligence's capability level)

> - Using current human technology, synthesize a normal virus that infects 90% of Earth within an hour: NO

> - Write a secure operating system on the first try, zero errors, no debugging phase, assuming away Meltdown-style hardware vulnerabilities in the chips: DEFINITE YES

> - Write a secure operating system for actual modern hardware, on the first pass: YES

> - Train an AI system with capability at least equivalent to GPT-4, from the same dataset GPT-4 used, starting from at most 50K of Python code, using 1000x less compute than was used to train GPT-4: YES

> - Starting from current human tech, bootstrap to nanotechnology in a week: YES

> - Starting from current human tech, bootstrap to nanotechnology in an hour: GOSH WOW IDK, I DON'T ACTUALLY KNOW HOW, BUT DON'T WANT TO CLAIM I CAN SEE ALL PATHWAYS, THIS ONE IS REALLY HARD FOR ME TO CALL, BRAIN LEGIT DOESN'T FEEL GOOD BETTING EITHER WAY, CALL IT 50:50??

> - Starting from current human tech and from the inside of a computer, bootstrap to nanotechnology in a minute: PROBABLE NO, EVEN IF A MINUTE IS LIKE 20 SUBJECTIVE YEARS TO THE SI

> - Bootstrap to nanotechnology via a clean called shot: all the molecular interactions go as predicted the first time, no error-correction rounds needed: PROBABLY YES but please note this is not any kind of necessary assumption because It could just build Its own fucking lab, get back the observations, and do a debugging round; and none of the processes there intrinsically need to run at the speed of humans taking hourly bathroom breaks, it can happen at the speed of protein chemistry and electronics. Please consider asking for 6 seconds how a superintelligence might possibly overcome such incredible obstacles of 'I think you need a positive nonzero number of observations', for example, by doing a few observations, and then further asking yourself if those observations absolutely have to be slow like a sloth

> - Bootstrap to nanotechnology by any means including a non-called shot where the SI designs more possible proteins than It needs to handle some of the less certain cases, and gets back some preliminary observations about how they interacted in a liquid medium, before it actually puts together the wetware lab on round 2: YES

(The Tweet goes on, you can read the rest of it at the link below, but that should give you the gist.)

[1] https://twitter.com/ESYudkowsky/status/1658616828741160960

I've already read that thread.

I don't have twitter and I agree his tweets have an aura of lunacy, which is a shame as he's quite a lot better as a long-form writer. (Though I will assume his long-form writings about quantum mechanics is as bad as everyone else unless a physicist vouches for them).

But, despite that, I don't understand why you chose that specific example — how is giving a list of what he thinks an AI probably can and can't do, in the context of trying to reduce risks because he thinks loosing is the default, similar to a fundamentalist Christian who wants to immanentize the eschaton because the idea the good guys might lose when God is on their side is genuinely beyond comprehension?

Some of the academics who signed are either not doing AI research e.g climatologists, genomics, philosophy. Or they have Google connections that aren't disclosed. E.g. Peter Norvig is listed as Stanford University but ran Google Research for many years, McIlrath is associated with the Vector Institute which is funded by Google.

Id like to see the equation that led to this 46%. Even long time researchers can be overcome by grift

I just took that list and separated everyone that had any commercial tie listed, regardless of the company. 35 did and 63 did not.

> "Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left."

You're putting a lot of effort into painting this list in a bad light without any specific criticism or evidence of malfeasance. Frankly, it sounds like FUD to me.

With corporate conflicts (that I recognized the names of):

Yoshua Bengio: Professor of Computer Science, U. Montreal / Mila, Victoria Krakovna: Research Scientist, Google DeepMind, Mary Phuong: Research Scientist, Google DeepMind, Daniela Amodei: President, Anthropic, Samuel R. Bowman: Associate Professor of Computer Science, NYU and Anthropic, Helen King: Senior Director of Responsibility & Strategic Advisor to Research, Google DeepMind, Mustafa Suleyman: CEO, Inflection AI, Emad Mostaque: CEO, Stability AI, Ian Goodfellow: Principal Scientist, Google DeepMind, Kevin Scott: CTO, Microsoft, Eric Horvitz: Chief Scientific Officer, Microsoft, Mira Murati: CTO, OpenAI, James Manyika: SVP, Research, Technology & Society, Google-Alphabet, Demis Hassabis: CEO, Google DeepMind, Ilya Sutskever: Co-Founder and Chief Scientist, OpenAI, Sam Altman: CEO, OpenAI, Dario Amodei: CEO, Anthropic, Shane Legg: Chief AGI Scientist and Co-Founder, Google DeepMind, John Schulman: Co-Founder, OpenAI, Jaan Tallinn: Co-Founder of Skype, Adam D'Angelo: CEO, Quora, and board member, OpenAI, Simon Last: Cofounder & CTO, Notion, Dustin Moskovitz: Co-founder & CEO, Asana, Miles Brundage: Head of Policy Research, OpenAI, Allan Dafoe: AGI Strategy and Governance Team Lead, Google DeepMind, Jade Leung: Governance Lead, OpenAI, Jared Kaplan: Co-Founder, Anthropic, Chris Olah: Co-Founder, Anthropic, Ryota Kanai: CEO, Araya, Inc., Clare Lyle: Research Scientist, Google DeepMind, Marc Warner: CEO, Faculty, Noah Fiedel: Director, Research & Engineering, Google DeepMind, David Silver: Professor of Computer Science, Google DeepMind and UCL, Lila Ibrahim: COO, Google DeepMind, Marian Rogers Croak: VP Center for Responsible AI and Human Centered Technology, Google

Without:

Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto, Dawn Song: Professor of Computer Science, UC Berkeley, Ya-Qin Zhang: Professor and Dean, AIR, Tsinghua University, Martin Hellman: Professor Emeritus of Electrical Engineering, Stanford, Yi Zeng: Professor and Director of Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Xianyuan Zhan: Assistant Professor, Tsinghua University, Anca Dragan: Associate Professor of Computer Science, UC Berkeley, Bill McKibben: Schumann Distinguished Scholar, Middlebury College, Alan Robock: Distinguished Professor of Climate Science, Rutgers University, Angela Kane: Vice President, International Institute for Peace, Vienna; former UN High Representative for Disarmament Affairs, Audrey Tang: Minister of Digital Affairs and Chair of National Institute of Cyber Security, Stuart Russell: Professor of Computer Science, UC Berkeley, Andrew Barto: Professor Emeritus, University of Massachusetts, Jaime Fernández Fisac: Assistant Professor of Electrical and Computer Engineering, Princeton University, Diyi Yang: Assistant Professor, Stanford University, Gillian Hadfield: Professor, CIFAR AI Chair, University of Toronto, Vector Institute for AI, Laurence Tribe: University Professor Emeritus, Harvard University, Pattie Maes: Professor, Massachusetts Institute of Technology - Media Lab, Peter Norvig: Education Fellow, Stanford University, Atoosa Kasirzadeh: Assistant Professor, University of Edinburgh, Alan Turing Institute, Erik Brynjolfsson: Professor and Senior Fellow, Stanford Institute for Human-Centered AI, Kersti Kaljulaid: Former President of the Republic of Estonia, David Haussler: Professor and Director of the Genomics Institute, UC Santa Cruz, Stephen Luby: Professor of Medicine (Infectious Diseases), Stanford University, Ju Li: Professor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, Massachusetts Institute of Technology, David Chalmers: Professor of Philosophy, New York University, Daniel Dennett: Emeritus Professor of Philosophy, Tufts University, Peter Railton: Professor of Philosophy at University of Michigan, Ann Arbor, Sheila McIlraith: Professor of Computer Science, University of Toronto, Lex Fridman: Research Scientist, MIT, Sharon Li: Assistant Professor of Computer Science, University of Wisconsin Madison, Phillip Isola: Associate Professor of Electrical Engineering and Computer Science, MIT, David Krueger: Assistant Professor of Computer Science, University of Cambridge, Jacob Steinhardt: Assistant Professor of Computer Science, UC Berkeley, Martin Rees: Professor of Physics, Cambridge University, He He: Assistant Professor of Computer Science and Data Science, New York University, David McAllester: Professor of Computer Science, TTIC, Vincent Conitzer: Professor of Computer Science, Carnegie Mellon University and University of Oxford, Bart Selman: Professor of Computer Science, Cornell University, Michael Wellman: Professor and Chair of Computer Science & Engineering, University of Michigan, Jinwoo Shin: KAIST Endowed Chair Professor, Korea Advanced Institute of Science and Technology, Dae-Shik Kim: Professor of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Frank Hutter: Professor of Machine Learning, Head of ELLIS Unit, University of Freiburg, Scott Aaronson: Schlumberger Chair of Computer Science, University of Texas at Austin, Max Tegmark: Professor, MIT, Center for AI and Fundamental Interactions, Bruce Schneier: Lecturer, Harvard Kennedy School, Martha Minow: Professor, Harvard Law School, Gabriella Blum: Professor of Human Rights and Humanitarian Law, Harvard Law, Kevin Esvelt: Associate Professor of Biology, MIT, Edward Wittenstein: Executive Director, International Security Studies, Yale Jackson School of Global Affairs, Yale University, Karina Vold: Assistant Professor, University of Toronto, Victor Veitch: Assistant Professor of Data Science and Statistics, University of Chicago, Dylan Hadfield-Menell: Assistant Professor of Computer Science, MIT, Mengye Ren: Assistant Professor of Computer Science, New York University, Shiri Dori-Hacohen: Assistant Professor of Computer Science, University of Connecticut, Jess Whittlestone: Head of AI Policy, Centre for Long-Term Resilience, Sarah Kreps: John L. Wetherill Professor and Director of the Tech Policy Institute, Cornell University, Andrew Revkin: Director, Initiative on Communication & Sustainability, Columbia University - Climate School, Carl Robichaud: Program Officer (Nuclear Weapons), Longview Philanthropy, Leonid Chindelevitch: Lecturer in Infectious Disease Epidemiology, Imperial College London, Nicholas Dirks: President, The New York Academy of Sciences, Tim G. J. Rudner: Assistant Professor and Faculty Fellow, New York University, Jakob Foerster: Associate Professor of Engineering Science, University of Oxford, Michael Osborne: Professor of Machine Learning, University of Oxford, Marina Jirotka: Professor of Human Centred Computing, University of Oxford

So the most “notable” AI scientists on this list have clear corporate conflicts. Some are more subtle:

> Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto,

He’s affiliated with Vector (as well as some of the other Canadians on this list) and was at Google until very recently (unsure if he retained equity which would require disclosure in academia).

Hence my interest in disclosures as the conflicts are not always obvious.

Ok, that's a person!

How is saying that they should have disclosed a conflict that they did not disclose not accusatory? If that's the case, the accusation is entirely justified and should be surfaced! The other signatories would certainly want to know if they were signing in good faith when others weren't. This is what I need interns for.

I think you’re misunderstanding my point.

I never said “they should have disclosed a conflict they did not disclose.”

Disclosures are absent from this initiative, some signatories have self-identified their affiliation by their own volition and even for those it is not in the context of a conflict disclosure.

There is no “signatories have no relevant disclosures” statement for those who did not for the omission to be malfeasance and pointing out the absence of a disclosure statement is not accusatory of the individuals, rather that the initiative is not transparent about potential conflicts.

Once again, it is standard practice in academia to make a disclosure statement if lecturing or publishing. While it is not mandatory for initiatives calling for regulation it would be nice to have.

I’m not painting anything, if a disclosure is needed to present a poster at a conference it’s reasonable to want one when calling for regulation.

Note my comments are non-accusatory and only call for more transparency.

I'd guess that a given academic isn't going to face much of a career risk for signing a statement also signed by other very prestigious academics, just the opposite. There's no part of very divided US political spectrum that I can see denouncing AI naysayers, unlike the scientists who signed anti-nuclear statements in 1960s or even people warning about global warming now (indeed, I'd guess the statement doesn't mention climate change 'cause it's still a sore point).

Moreover, talking about existential risk involves the assumption the current tech is going to continue to affect more and more fields rather than peaking at some point - this assumption guarantees more funding along with funding for risk.

All that said, I don't necessarily think the scientists involved are insincere. Rather, I would expect they're worried and signed this vague statement because it was something that might get traction. While the companies indeed may be "genuine" in the sense they're vaguely [concerned - edit] and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from"

In interviews, Geoffrey Hinton and Yoshua Bengio certainly expressed serious concerns and even some plausible regret to their life’s work. They did not say anything that can be interpreted as your last sentence suggests at all.

My last sentence currently: "While the companies indeed may be "genuine" in the sense they're vaguely and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from" - IE, I am not referring to the academics there.

I'm going to edit the sentence to fill in some missing words but I don't think this will change the meaning involved.

On the contrary, I suspect "How do we prevent our AIs from killing everyone?" will be a major research question with a great deal of funding involved. Plus, no one seems to be suggesting things like the medical ethics field or institutional review boards, which might have deleterious impacts on their work.

You're putting a weirdly large amount of trust into, functionally, some dude who posted on lesswrong. Sure he has a PhD and is smart, but so is basically everyone else in the field, not just in alignment, and the median person in the field thinks the risk of "doom" is 2-5% (and that's conditioned on the supposed existence of a high level machine intelligence that the median expert believes might exist in 40 years). That still might be higher than you'd like, but it's not actually a huge worry in the grand scheme of things.

Like, if I told you that in 40 years, there was a 50% chance of something existing that had a 2% chance of causing extreme harm to the human population, I'm actually not sure that thing should be the biggest priority. Other issues may have more than a 1% chance of leading to terrible outcomes sooner.

The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years.

But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.

> The median person in field thinks 5-10%, not 2-5%

The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages.

> median timelines are shorter than 40 years.

The median person suggested a 50% chance in 39 years.

> since unaligned ASI is the default case

I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.

I work in the space (doing tech stuff that isn't direct research). The best argument I've seen for alignment by default is something like "morality comes from training data, and therefore the fact that LLM training data sets contain human moral intuitions will mean that an ASI stemming from such a training regime will share enough human values" (Quintin Pope believes something like this, as far as I can tell), which is deeply unconvincing, since it contradicts the evidence we _do_ have from human & animal value formation.

Happy to entertain other arguments that alignment-by-default is reasonable; most arguments I've seen are much worse than that one. I haven't seen many people make an active case for alignment-by-default, so much as leave open a whole bunch of swath of uncertainty for unknown unknowns.

Aren't we at least equally doomed without AI?

I think its cringe to both define doom and then come up with some unpredictability precise percentage for that scenario to occur.

46% is such a precise number. This is why I can't take "rationalists", the Yudkowskys, and the Silvers seriously. Colossal assumptions turned into confidently stated probabilities.

They didn’t become such a wealthy group by letting competition foster. I have no doubt they believe they could be doing the right thing but I also have no doubt they don’t want other people making the rules.

Truth be told, who else really does have a seat at the table for dictating such massive societal change? Do you think the copy editor union gets to sit down and say “I’d rather not have my lunch eaten, I need to pay my rent. Let’s pause AI usage in text for 10 years.”

These competitors banded together and put out a statement to get ahead of any one else doing the same thing.

Not all of them are wealthy, a significant number are academics.

That doesn’t erase the need of cynicism. Many people in academia come from industry, have friends in industry, or other stakes. They might have been persuaded by the rhetoric of stakeholders within industry (you saw this early in the climate debate; and still do), and they might also be hoping to get a job in the industry later on. There is also a fair amount of group think within academia, so if a prominent individual inside academia believes the lies of industry, chances are the majority within the department does.

What’s more likely, that everyone on this list just happens to have some overlapping ulterior motive or that they actually believe it?

Quite often these intersect. But also it is healthy to question the motives of signaturees asking for public policy. Especially when a significant number of them are direct stakeholders. When an academic signs such a list, their motives should be equally questioned.

Cool username btw. Any relation to the band?

The people I know on the list are academics and do not seem to be any wealthier than other academics I know. I'm quite certain the private industry signatories are going to entirely advocate for their interest just as they do in any other policy discussion.

Got it, thank you for the clarification!

Here’s why AI risks are real, even if our most advanced AI is merely a ‘language’ model:

Language can represent thoughts and some world models. There is strong evidence that LLMs contain some representation of world models it learned from text. Moreover, LLM is already a misnomer; latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence.

More on risks of “doom” by a top researcher on AI risk here: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...

A lot of things are called "world models" that I would consider just "models" so it depends on what you mean by that. But what do you consider to be strong evidence? The Othello paper isn't what I'd call strong evidence.

I agree that the Othello paper isn't, and couldn't be, strong evidence about what sort of model of the world (if any) something like GPT-4 has. However, I think it is (importantly) pretty much a refutation of all claims along the lines of "these systems learn only from text, therefore they cannot have anything in them that actually models anything other than text", since their model learned only from text and seems to have developed something very much like a model of the state of the game.

Again, it doesn't say much about how good a model any given system might have. The world is much more complicated than an Othello board. GPT-4 is much bigger than their transformer model. Everything they found is consistent with anything from "as it happens GPT-4 has no world model at all" through to "GPT-4 has a rich model of the world, fully comparable to ours". (I would bet heavily on the truth being somewhere in between, not that that says very much.)

Yeah, I think it leaves us back at square one: we don't know much about how it really works.

I don't follow it too closely, but I've seen papers on mechanistic interpretability that look promising.

I'd say stronger evidence than the Othello paper is the ability to answer what-if questions coherently and plausibly.

I just asked GPT-4 "What would happen to Northern Thailand if elephants behave like kangaroos?".

The answers are probably better than what 90+% of humans could give after spending an hour researching on the internet.

The Sparks of AGI paper provides much more evidence and examples: https://arxiv.org/abs/2303.12712

You may be right, I don't know the people involved on a personal basis. Perhaps my problem is how much is left unsaid here (the broader safe.ai site doesn't help much). For example, what does "mitigate" mean? The most prominent recent proposal for mitigation comes from Sam Altman's congressional testimony, and it's very self serving. In such a vacuum of information, it's easy to be cynical.

Right. It probably needed to be general because there hasn't been enough time to work out sane specific responses, and even if they had, getting buy-in on specifics is a recipe for paralysis by indecision. A credible group of people simply pleading for policy makers, researchers, et. al. to take this seriously will lead to the project approvals, grant money, etc. that will hopefully yield a more sophisticated understanding of these issues.

Cynicism is understandable in this ever-expanding whirlpool of bullshit, but when something looks like it has potential, we need to vigorously interrogate our cynicism if we're to stand a chance at fighting it.

Reading the comments here is helping evolve my thinking on the issue for sure. Here's a comment I made in another thread:

> As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something that will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation.

> If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.

If policymaker's understanding of AI is predicated on hypothetical scenarios like "Weaponization" or "Power-Seeking Behavior" and not on concrete economic disruptions that AI will be causing very soon, the policy they come up with will be inadequate. Thus I'm frustrated with the framing of the issue that safe.ai is presenting because it is a distraction from the very real societal consequences of automating labor to the extent that will soon be possible.

My own bit of cynicism is that regulating the negative impacts of technology on workforce segments in the US is a non-starter if you approach it from the technology-end of the issue rather than the social safety net end. Most of these automation waves that plunged entire employment categories and large metropolitan areas into oblivion were a net gain for the economy even if it was concentrated at the top. I think the government will temporarily socialize the costs of the corporate profit with stimulus payments, extended unemployment benefits, and any other thing they can do to hold people over until there's a comparatively small risk of triggering real social change. Then they just blame it on the individuals.

Agreed. To pile on: one true existential risk is the continued economic/social polarization of the US and other large nation states (not to mention climate changes).

We seem perfectly resigned to the expansion of an already huge underclass—-with or without the help of LLMs.

Might not AGI help convince our tech savvy aristocracy that one fundamental problem still is better balancing of opportunities and more equitable access to a good education? I see the probability of that happening as precisely 4.6%.

> will lead to the project approvals, grant money, etc.

In other words, a potential conflict of interest for someone seeking tenure?

This particular statement really doesn't seem like a marketing ploy. It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk

These are, for the most part, obvious applications of a technology that exists right now but is not widely available yet.

The problem with every discussion around this issue is that there are other statements on "the existential risk of AI" out there that are either marketing ploys or science fiction. It doesn't help that some of the proposed "solutions" are clear attempts at regulatory capture.

This muddles the waters enough that it's difficult to have a productive discussion on how we could mitigate the real risk of, e.g., AI generated disinformation campaigns.

> The problem with every discussion around this issue is that there are other statements on

Sure, but we're not talking about those other ones. Dismissing good faith initiatives as marketing ploys because there are bad faith initiatives is functionally no different than just shrugging and walking away.

Of course OpenAI et. al. will try to influence the good faith discussions: that's a great reason to champion the ones with a bunch of good faith actors who stand a chance of holding the industry and policy makers to task. Waiting around for some group of experts that has enough clout to do something, but by policy excludes the industry itself and starry-eyed shithead "journalists" trying to ride the wave of the next big thing will yield nothing. This is a great example of perfect being the enemy of good.

I agree completely. I was just speculating on why there is so much discussion about marketing ploys in this comment section.

Ah, sure. That makes sense.

There's definitely a lot of marketing bullshit out there in the form of legit discussion. Unfortunately, this technology likely means there will be an incalculable increase in the amount of bullshit out there. Blerg.

> It is difficult to disagree with the potential political and societal impacts of large language models as outlined here

Is it? Unless you mean something mundane like "there will be impact", the list of risks they're proposing are subjective and debatable at best, irritatingly naive at worst. Their list of risks are:

1. Weaponization. Did we forget about Ukraine already? Answer: Weapons are needed. Why is this AI risk and not computer risk anyway?

2. Misinformation. Already a catastrophic problem just from journalists and academics. Most of the reporting on misinformation is itself misinformation. Look at the Durham report for an example, or anything that happened during COVID, or the long history of failed predictions that were presented to the public as certain. Answer: Not an AI risk, a human risk.

3. People might click on things that don't "improve their well being". Answer: how we choose to waste our free time on YouTube is not your concern, and you being in charge wouldn't improve our wellbeing anyway.

4. Technology might make us fat, like in WALL-E. Answer: it already happened, not having to break rocks with bigger rocks all day is nice, this is not an AI risk.

5. "Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems". Answer: already happens, just look at how much censorship big tech engages in these days. AI might make this more effective, but if that's their beef they should be campaigning against Google and Facebook.

6. Sudden emergent skills might take people by surprise. Answer: read the paper that shows the idea of emergent skills is AI researchers fooling themselves.

7. "It may be more efficient to gain human approval through deception than to earn human approval legitimately". No shit Sherlock, welcome to Earth. This is why labelling anyone who expresses skepticism about anything as a Denier™ is a bad idea! Answer: not an AI risk. If they want to promote critical thinking there are lots of ways to do that unrelated to AI.

8. Machines smarter than us might try to take over the world. Proof by Vladimir Putin is provided, except that it makes no sense because he's arguing that AI will be a tool that lets humans take over the world and this point is about the opposite. Answer: people with very high IQs have been around for a long time and as of yet have not proven able to take over the world or even especially interested in doing so.

None of the risks they present is compelling to me personally, and I'm sure that's true of plenty of other people as well. Fix the human generated misinformation campaigns first, then worry about hypothetical non-existing AI generated campaigns.

I appreciate your perspective, but the thing that is missing is the speed at which AI has evolved, seemingly overnight.

With crypto, self-driving cars, computers, the internet or just about any other technology, development and distribution happened over decades.

With AI, there’s a risk that the pace of change and adoption could be too fast to be able to respond or adapt at a societal level.

The rebuttals to each of the issues in your comment are valid, but most (all?) of the counter examples are ones that took a long time to occur, which provided ample time for people to prepare and adapt. E.g. “technology making us fat” happened over multiple decades, not over the span of a few months.

Either way, I think it’s good to see people proactive about managing risk of new technologies. Governments and businesses are usually terrible at fixing problems that haven’t manifested yet… so it’s great to see some people sounding the alarms before any damage is done.

Note: I personally think there’s a high chance AI is extremely overhyped and that none of this will matter in a few years. But even so, I’d rather see organizations being proactive with risk management rather than reacting too the problem when it’s too late.

It may seem overnight if you weren't following it, but I've followed AI progress for a long time now. I was reading the Facebook bAbI test paper in 2015:

https://research.facebook.com/downloads/babi/

There's been a lot of progress since then, but it's also nearly 10 years later. Progress isn't actually instant or overnight. It's just that OpenAI spent a ton of money to scale it up then stuck an accessible chat interface on top of tech that was previously being mostly ignored.

As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something what will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation.

If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.

I believe the solution to said socio economic problem is rather simple.

People are being replaced by robots and AI because the latter are cheaper. That's the market force.

Cheaper means that more value us created. As a whole, people get more service for doing less work.

The problem is that the money or value saved trickles up to the rich.

The only solutions can be, regulations,

- do not tax anymore based on income from doing actual work.

- tax automated systems on their added value.

- use the tax generated capital to provide for a basic income for everybody.

In that way, the generated value goes to people who lost their jobs and to the working class as well.

> It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk

I disagree

That list is a list of the dangers of power

Many of these dangers: misinformation, killer robots, people on this list have been actively working on

Rank hypocrisy

And people projecting their own dark personalities onto a neutral technology

Yes there are dangers in unbridled private power. They are not dangers unique to AI.

Aside from emergent behavior, are any of the items on that list unique to AI? They sure don’t seem it; they’re either broadly applicable to a number of already-available technologies, or to any entity in charge or providing advice or making decisions. I dare say even emergent behavior falls under this as well, since people can develop their own new motives that others don’t understand. Their advisory doesn’t seem to amount to much more than “bad people can do bad things”, except now “people” is “AI”.

>They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met.

with all due respect, that's just <Your> POV of them or how they chose to present themselves to you.

They could all be narcissists for all we know. Further, One person's opinion, namely yours, doesn't exempt them from criticism and rushing to be among the first in what's arguably the new gold rush.

The only name in this list that gives me pause is Daniel Dennett. At least he has a philosopher’s historical perspective on the challenges ahead.

Very few neuroscientists as signers. Perhaps they know better.

For me the basic sociopolitical climate today (and for the last 120 years) is at an existential boil. This is not to say that birthing AGIs is not frightening, but just to say that many real features of today are just as, or even more frightening.

> even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks.

if the corporations in question get world governments to line up the way they want, the checks for everyone in these "letters" will be *way* bigger than 25k and they won't have "payment for signing our letter" in the memo either.

Be aware that the things that AI challenges most is knowledge work. No food delivery boy job is being challenged here (who cares about these people anyway?) but if you are a software developer the clock on that is ticking.

Yeah so am I but they all have incentives personally. Arguing for regulation has zero downside risk especially if all your friends are doing it too.

Professionally, ALL of their organizations benefit from regulatory capture as everyone is colluding via these letters.

Go look at what HuggingFace are doing to show you how to to it - and they only can cause they are French and actually exercise their freedom

I think it's the combination of two things.

First, there are actual worries by a good chunk of the researchers. From runaway-paperclip AGIs to simply unbounded disinformation, I think there are a lot of scenarios that disinterested researchers and engineers worry about.

Second, the captains of industry are taking note of those worries and making sure they get some regulatory moat. I think the Google memo about moat hits it right on the nail. The techniques and methods to build these systems are all out on the open, the challenges are really the data, compute, and the infrastructure to put it all together. But post training, the models are suddenly very easy to finetune and deploy.

AI Risk worry comes as an opportunity for the leaders of these companies. They can use this sentiment and the general distrust for tech to build themselves a regulatory moat.

They don't want 25k they want jobs in the next presidential administration

> They don't want 25k they want jobs in the next presidential administration

Academics shilling for OpenAI would get them jobs in the next presidential administration?

Having their names on something so public is definitely an incentive for prestige and academic promotion.

Shilling for OpenAI & co is also not a bad way to get funding support.

I’m not accusing any non-affiliated academic listed of doing this but let’s not pretend there aren’t potentially perverse incentives influencing the decisions of academics, with respects to this specific letter and in general.

To help dissuade (healthy) skepticism it would be nice to see disclosure statements for these academics, at first glance many appear to have conflicts.

Could you be more specific about the conflicts you've uncovered?

It’s unequivocal that academics may have conflicts (in general), that’s why disclosures are required for publications.

I’m not uncovering anything, several of the academic signatories list affiliations with OpenAI, Google, Anthropic, Stability, MILA and Vector resulting in a financial conflict.

Note that conflict does not mean shill, but in academia it should be disclosed. To allay some concerns a standard disclosure form would be helpful (i.e. do you receive funding support or have financial interest in a corporation pursuing AI commercialization).

I'm not really interested in doing a research project on the signatories to investigate your claim, and talking about things like this without specifics seems dubiously useful, so I don't really think there's anything more to discuss.

Huh, you don’t have to do any research.

Go to: https://www.safe.ai/statement-on-ai-risk#signatories and uncheck notable figures.

Several of the names at the top list a corporate affiliation.

If you want me to pick specific ones with obvious conflicts (chosen at a glance): Geoffrey Hinton, Ilya Sutskever, Ian Goodfellow, Shane Legg, Samuel Bowman and Roger Grosse are representative examples based on self-disclosed affiliations (no research required).

Oh so you're saying the ones there with conflicts listed. That's only like 1/3 of the list.

Yes, as I said “many” have obvious conflicts from listed affiliations so it would be nice to have a positive/negative disclosure from the rest.

Reply on news.ycombinator.com