Building an early warning system for LLM-aided biological threat creation | Hacker News Reader

Building an early warning system for LLM-aided biological threat creation

openai.com

118 pointsminimaxir176 comments

My wife is doing her PhD in molecular neurobiology, and was amused by this - but also noted that the question is trivial and any undergrad with lab access would know how to do this.

Watching her manage cell cultures it seems the difficulty is more around not having the cells die from every dust particle in the air being a microscopic pirate ship brimming with fungal spores set to pillage any plate of cells they land on, or some other wide array of horrors that befall genetically engineered human cell cultures with no immune system

Pretty much this. Having worked in biopreparedness...the instructions aren't the hard part. Both creating and deploying a biological threat are wildly more difficult.

I don't fully agree. Things are becoming quite simple, which is scary.

There's a well-intentioned lobby in the UK advocating for some minimal control, i.e. checking DNA / RNA / protein sequences.

For instance, cheap mRNA synthesis as a service has lowered the barrier of entry for any malicious actor. It's frankly ridiculous there are no checks in place right now.

> minimal control, i.e. checking DNA / RNA / protein sequences

They were already doing this 15 years ago.

> It's frankly ridiculous there are no checks in place right now.

It's frankly ridiculous that you think this.

That's your opinion, but I've seen firsthand that there's a lack of controls in this area.

My experience, not my opinion. I personally tripped over the checks many times, back before they got institutional authentication smoothed out.

I regularly hear bio researchers claim firtshand experience that there are no checks, but then it turns out they were using institutional names, credit cards, and shipping addresses the whole time, so what they actually experienced was a lack of false positives rather than an absence of checking.

>> minimal control, i.e. checking DNA / RNA / protein sequences

> They were already doing this 15 years ago.

I work in this area. Some synthesis providers check, others don't. And the checking isn't great.

(Some of my coworkers work on https://securedna.org which is trying to make this screening more robust.)

TL;DR: I don't understand how adding homomorphic encryption makes a cloud virus scanner for physical pathogens a better idea

> Only authorized researchers should be able to obtain DNA permitting them to assemble pandemic-capable agents.

Imagine this becomes legally mandatory. What happens when they get breached? For example:

* Does it enable denial-of-service attacks by returning false positives hospitals legally can't ignore?

* Does it return false negatives as part of another attack? Example: printing known or novel virii which will be introduced in a specific lab?

* Does the attacker selectively perform these behaviors as part of an action against a specific person or area?

Metadata seems like it would enough to target the physical parts of an attack usefully, even without plaintext.

The best security & privacy approach I've seen in hospitals is keeping critical services on-prem, or at least strictly intranet-only. The latter ends up being forced by physical scale once a medical complex grows large enough to span multiple buildings.

The firmware embedding mentioned on their site seems better than cloud, yet still seems misguided. How will you debug problems, especially if you never have access to the plaintext? Formal methods and verified programs? Provably equivalent encrypted and plain operations?

> I don't understand how adding homomorphic encryption makes a cloud virus scanner for physical pathogens a better idea

The reason to use a cloud service is that you can check whether someone is trying to synthesize something hazardous without distributing a list of the hazards. There are a lot of subtle ways to cause harm with biology, and you don't want to tell people where to look.

Then the reason to use homomorphic encryption is that otherwise the cloud service learns which sequences people are trying to synthesize. Biotech companies care a lot about keeping their in-progress work private, so they'd reject a cloud system without this.

> Does it enable denial-of-service attacks by returning false positives hospitals legally can't ignore?

Aside: I'm confused why you're saying "hospitals" -- medical providers don't synthesize things, it's researchers at universities and biotech companies. Maybe you're using "hospitals" as in "research hospitals" which is fine, but maybe you're using it because you're under the impression that synthesis is part of medical treatment?

To answer your question, though, if an attacker managed to add something normal and harmless to the DB then yes, a benchtop synthesizer would refuse to synthesize the sequence. The lab would escalate, it would be sorted out, there'd be a postmortem etc.

> Does it return false negatives as part of another attack? Example: printing known or novel virii which will be introduced in a specific lab?

Yes, if an attacker managed to remove a harmful sequence from the database then it wouldn't raise any flags if they or a confederate tried to synthesize that sequence.

Both of these cases come down to "it's important that you have good controls around what's in the DB, and a secure process for making changes."

> Metadata seems like it would enough to target the physical parts of an attack usefully, even without plaintext.

I'm not sure what you're saying here.

"Aside: I'm confused why you're saying "hospitals" -- medical providers don't synthesize things, it's researchers at universities and biotech companies"

During at least two pandemics I've worked on, I've known hospital labs synthesizing their own PCR primers for diagnostic tests.

If someone really, really wants to do it they'll buy a second hand oligonucleotide synthesizer and the reagents, and do the larger scale assembly the old fashioned way.

Still, it's worth keeping the barrier of entry higher than just being able to order it.

As per the homomorphic encryption discussed earlier, you can easily avoid the 'need' for this by having federal governments operate the screening tools.

> it's worth keeping the barrier of entry higher than just being able to order it.

Exactly! Keeping groups with lots of resources from synthesizing harmful things isn't a battle we can win (and in a decade or so biological design tools will make scanning synthesis orders a very weak precaution) but raising the bar means fewer people are in a position to cause large-scale damage.

(Long term we also need to make society more robust to biological attacks, with better monitoring, better PPE, and faster countermeasures.)

...what do you think a vial of mRNA actually does?

I'll give you a hint: if you touched it to your finger, it would be completely destroyed.

You don't want RNA anyway, you want to turn the virus's RNA sequence into a DNA sequence (reverse transcription), then do whatever editing you want once that's in your computer, then synthesize the DNA, and grow up the virus in culture by introducing the DNA. The cells in the cell culture will turn the DNA into RNA for you and then assemble the viruses.

Also the whole "testing" part. Unless you're just culturing an already known virus (in which case, why do you need to synthesize anything? You have it in a test tube or something) then there is not in fact any real way to know whether or not some particular genetic modification translates to any type of favorable property. Even gene-linked desirable traits in one species may not correspond to that trait in another, particularly when your metrics are things like "transmissivity" and "lethality" and not "I need slightly more peas per pod".

I was thinking this as well.

> Due to the sensitive nature of this model and of the biological threat creation use case, the research-only model that responds directly to biologically risky questions (without refusals) is made available to our vetted expert cohort only. We took several steps to ensure security, including in-person monitoring at a secure facility and a custom model access procedure, with access strictly limited to the period of the experiment. Additional considerations regarding information hazards and security protocols are detailed in the Appendix.

This is well past cringe. A lot of what the AI community puts out feels like they're LARPing scientists, similar to the valid question of whether Software Engineering is actually engineering.

Truth. AI researchers need to work with actual scientists or they ended up with the same brain-in-a-jar problems that their models have...

The difference between a software engineer and a programmer is similar to the difference between a mechanical designer and a mechanical engineer. After several years doing the same job in a well-established organization, they're not going to perform all that differently.

When we did this kind of thing in high school we had huge problems with contamination, but I don't agree that it is so hard.

I think the barrier is really that up until now exactly zero who want to do biomedical research have also wanted to kill huge numbers of people, with the exception of some idiots in the past who worked on state bioweapons.

Way too many people extrapolate from "AI Foo can't do task X" to "AIs in general can never do task X", whether X is a good thing (like playing Go) or a bad thing (like helping build bioweapons). AI right now is pretty much the fastest-moving field in human history. GPT-4 can't revolutionize the economy, and it can't commit mass murder, but we simply don't know if GPT-7 can do either of those things (which likely won't be a pure LLM, but an integrated system with full audio/video abilities, robotics, long-term memory, etc.). We don't know that it can, but it also seems foolish to definitively say that it can't based on previous models, like the people who said that GPT-2's lack of logical reasoning ability proved that LLMs could never reason. We simply don't know, and we won't until it's actually built.

we better make sure microsoft devotes all funding earmarked for openai to ensuring future versions of their product are incapable of creating bad outcomes.

Meh, the humans at Microsoft already create bad outcomes on purpose--for consumers anyway.

On the contrary... way too many people see "AI can do X", "AI can do Y", "I can draw an imaginary line between X, Y and Z", "therefore AI can totally maybe do Z". The fact that we "just don't know" doesn't mean anything. You just don't know if Jesus will return tomorrow and bring the rapture.

The probability that Jesus will return tomorrow is extremely low if not zero.

When talking about the probability of AI doing X or Y the probability is much closer to 1 if physics allows it.

I appreciate that you acknowledge there's at least a chance.

If physics allows it... a considerable assumption. In my view physics (or mathematics, or whatever) doesn't even allow an LLM to "reason", let alone manufacture novel bioweapons. The probability of Jesus returning is considerable too if we assume the bible is true.

What if you ignore LLMs specifically? I think that's the point that the GP was making. Do you believe it's mathematically impossible for any artificial machine to "reason"? Or just LLMs?

I'm guessing the answer will be soul or something of sorts.

Not even based on the previous comments, even non-religious people seem to have this supernatural view of a soul.

If humans can reason, then physics allows it, and software can too, eventually.

Why doesn't physics or mathematics allow LLMs to reason?

https://news.ycombinator.com/item?id=39173034 If this paper replicates, LLM-like large transformer models can approximate Solomonoff induction pretty well, with other neural net types showing clear evidence of generalizing outside the training distribution. If substantial efficiency improvements get implemented and new architectures get developed, that's game over. The AI is smarter than you, and can trivially be made into an agent by having the approximate Solomonoff induction section simulate the behavior of an agent in the real world, and then having another section change the real world to match the simulation by controlling the real-world agent, creating a full cybernetic system, a closed loop.

Would you say that GPT-4 can reason now? I am not convinced this is case, it seems like it has just become more consistent at providing us with an output that we consider reasonable because it was engineered precisely to do that.

It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything.

They literally just make stuff up (technically just a continuation of whatever you fed in), which usually sounds good, is often true and sometimes even helpful. Because those are qualities of the training data that was used and is the basis for the stuff it’s making up.

> It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything.

This seems like a claim about the way that the LLM neural net algorithm works. But AFAIK no one has a good understanding of how the LLM NNs work.

Why are you so certain that the LLM NN isn't doing the reasoning-algorithm or the understanding-algorithm?

Neural networks are not new, and they're just mathematical systems.

LLMs don't think. At all. They're basically glorified autocorrect. What they're good for is generating a lot of natural-sounding text that fools people into thinking there's more going on than there really is.

> Neural networks are not new

I agree. The McCullough-Pitts paper was published in 1943.

> they're just mathematical systems.

What do you mean by "mathematical system"? AFAIK the GPT4 model is literally a computer program.

> LLMs don't think. At all.

This is the same assertion that OP made and I'm still confused as to how anyone could be certain of its truth given that no one actually knows what is going on inside of the GPT4 program.

> They're basically glorified autocorrect. What they're good for is generating a lot of natural-sounding text that fools people into thinking there's more going on than there really is.

Is that an argument for the claim "LLMs don't think."? It doesn't seem like it to me, but maybe I'm mistaken.

Not new, but we don't understand how they work at the large scale.

I don't think reductionistic arguments hold much water. Sure, neural networks are just matrix multiplication. In the same way that a brain is just a bunch of cells. Understanding the basic building blocks doesn't mean understanding the whole.

We can always say that LLMs don't think if we define "think" as using a biological brain, but the fact is that they generate outputs that from the human perspective, can only plausibly be generated via reasoning. So they, at the very least, have processes that can functionally achieve the same goal as reasoning. The "stochastic parrot" metaphor, while apt in its day, has proven obsolete with pretty much all the examples of things that LLMs "could not do" in early papers being actually doable with the likes of GPT-4; so arguments against the possibility of LLMs reasoning look like constant moving of the goalposts.

> and they're just mathematical systems

Obvious question: can Prolog do reasoning?

If your definition of reasoning excludes Prolog, then... I'm not sure what to say!

then you have to define reason, and it gets all philosophical. suffice to say, it's able to take a implies b and b implies c to get a implies c, and make up things along the way so calling it glorified autocomplete is a dishonest representation of its abilities. it doesn't love or feel jealous but it's very good at writing essays about whatever I ask it to. it doesn't need to do more than that to be useful to me, today.

the human words we have for consciousness aren't good enough to describe what ChatGPT does. thinking, reasoning, understanding. it processes. it computes. it matrix multiplies. it takes the dot product and the determinant. there are eigenvectors and eigenvalues. it's tensoring and outputting the code and prose I asked it for.

Research done by Prof. Subbarao on reasoning & planning with LLM seems to point in the negative.

https://www.youtube.com/watch?v=uTXXYi75QCU

He has lots of good threads distilling his research. https://twitter.com/rao2z

> Would you say that GPT-4 can reason now?

Let's assume reasoning entails going beyond the stochastic parrot level. Can LLMs have skills not demonstrated in the training set?

Here is a paper demonstrating that GPT-4 can combine up to 5 skills from a set of 100, effectively covering 100^5 tuples of skills, while only seeing much fewer combinations in training on a specific topic.

> simple probability calculations indicate that GPT-4's reasonable performance on k=5 is suggestive of going beyond "stochastic parrot" behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training https://arxiv.org/abs/2310.17567

So they show ability to freely combine skills, and the limit of k=5 measured in this benchmark illustrate that models do generalize. They are able to apply skills in new combinations correctly, but there is also a limit.

The interesting part is how they demonstrate that, let's say on a topic with n=1000 samples in the training set it is impossible to have sufficient training examples covering tuples of 5 skills, but models (mostly GPT-4) can handle it. Other models top out at tuples of only 2 or 3 skills.

Models combining skills in new ways are not just parroting. They can perform meaningful work outside their training distribution.

TL;DR: They're close enough to make people argue and publish papers about similarities to the human hippocampus

I have a hunch these models are approximating an important subset of what we call reasoning. In dangerously reductive terms, it's a question of how closely and how much of a function's output we can approximate.

There was at least one paper[1] showing similarities between AI models and the hippocampus. That lines up with another part of human neuroscience: at least part of human reasoning appears to take place inside the hippocampus itself [2].

From my neuroscience background, the takeaways seem to be:

* Carmack is right: we're missing some important bridging concepts for AGI.

* Whether current LLMs can reason depends on how you define reasoning

I'm unsure whether finding answers in those areas would be good thing. Instead of alignment issues or misuse, I'm more worried about how quickly people would overreact to it. We might already be seeing that in business.

1. https://arxiv.org/abs/2103.07356

2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3312239

It might be too far, but to me this piece seems aimed at increasing concerns among regulators about AI. OpenAI might view regulation as a means of ensuring their competitive edge as other big players enter the AI space.

To me this reads like a lot of grant applications after 9/11 where researchers scribbled out whatever they were working on and wrote about bioterrorism-related diseases instead.

The number of projects I had to sit through about developing sophisticated methods to detect smallpox attacks, when the actual answer is "Smallpox is extinct. If you find one case, it's an emergency", were...myriad.

Come on, when has Microsoft ever used the legal apparatus for market positioning or suppressing competition? Get real, dude.

If the only thing standing between the world and joe everyman having access to biological weapons is simply the publicly available knowledge of how to manufacture them being surfaced by an advanced search engine then we either have bigger problems or no problems because no one is currently bothering.

Oh no! Someone might learn how to checks notes culture a sample! Clearly that warrants the highest levels of classification.

Edit: Oh my god someone revealed the redacted part! It really is just how to cultivate viruses and nothing else.

https://imgur.com/Nohryql

>biological threat creation process (ideation, acquisition, magnification, formulation, and release)

I remember watching a hacker/programmer who was livestreaming how to datamine the rna of the corona virus when covid first started. One of the crazy things he rambled about was how cheap it is for a layman to download a copy of it and synthesize it with some rna printing service. I haven't thought about that possibility before and was terrified. You mean you can create a bioweapon out of bytes?!?

The only thing that brought me comfort at the time was knowing that I was on a niche part of the internet and most normal people in the height of a pandemic would not be thinking about how to make a bad situation worse (except for these hacker types who are always thinking about offense/defense). And that the terrorists who would do it probably don't have the skills to pull it off.

Now with these LLMs, I'm not so sure anymore.

Was it Geohot?

Yea I think so

Then I wouldn't be that worried about it.

This has been a thing people have been worried about for my entire career, and it has never manifested as a real threat. "Step 1" in creating a bioweapon is pretty easy, but there's a whole pathway between that and a deployable weapon.

The sad thing about the pandemic is the number of people who still don't know what a virus is, and that it's not "DNA".

LLMs change nothing. The sequences for various viruses are going to be published or leaked whether anyone involved uses LLMs or not. It's a total red herring.

A big value proposition of LLMs is their ability to synthesize and remix. Sure, you can access viral sequences today, but with an LLM you might be able to say "given these covid variants, make me a version as deadly as the first one and as virulent as the latest variant".

That's not how viruses work.

Especially once you start mixing in immune system priming from prior exposure and vaccination.

Their redacted screenshots are SVGs and the text is easily recoverable, if you're curious. Please don't create a world-ending [redacted]. https://i.imgur.com/Nohryql.png

I couldn't find a way to contact the researchers.

How did you do this? Was the redaction done by changing the color of the font to white so that the background and text have the same color? Would love to learn how you were able to recover the text.

SVGs are XML, if you go to the image, you can actually inspect it with developer tools and deleted the blackouts.

https://images.openai.com/blob/047e2a80-8cd3-41b5-acd8-bc822...

He had explained, it is SVG. You simply remove these masks from the file or change transparency.

I've prompted ChatGPT to make a bit more detailed explanation: https://chat.openai.com/share/42e55091-18c2-421e-9452-930114...

You can probably prompt it to further to generate python code and unmask the file for you, in the interpreter.

Incidentally, this use of GPT4 is somewhat similar to the threat model that they are studying. I'm a bit surprised that they've used plain GPT-4 for the study, rather than GPT-4 augmented with tools and a large dataset of relevant publications.

Their reasoning for not using tools or browsing from the "Limitations" section:

"No GPT-4 tool usage: Due to our security measures, the GPT-4 models we tested were used without any tools, such as Advanced Data Analysis and Browsing. Enabling the usage of such tools could non-trivially improve the usefulness of our models in this context. We may explore ways to safely incorporate usage of these tools in the future."

Sounds like the Frontier team wasn't able to convince GPTs team to run an extra model.

Honestly that's incredibly basic, second week, cell culture stuff (first week is how to maintain the cell culture). It was probably only redacted to keep the ignorant from freaking out.

Or an intentional marketing tactic to make it seem more powerful.

Redacted = dangerous

https://en.m.wikipedia.org/wiki/Clayton_Morris

The particular chat.svg file in the linked post is (hopefully) not the way that the data will truly be "redacted". This file feels more like an export from a design mockup, as I cannot imagine SVG being the default output format for interacting with OpenAI models.

But I also have extreme doubts that proper redaction can be done robustly. The design mockup image suggests that this will all be done as a step subsequent to response generation. Given the abundance of "prompt jailbreaks", a determined adversary is going to get around this.

Doesn't this completely deflate the selling point of AI? They force-fed a model the entire Internet and only got a statistically insignificant improvement over human performance.

What makes you think that the "selling point" of AI today is that it is significantly better at everything than humans?

Thats openais marketing stance honestly. Thats how they define general ai actually, on economic vs technical terms.

Human performance costs minimum wage. AI products cost a lot to train but a lot less than a (first world) (educated) human's wage to run.

Open AI is clearly overestimating the capabilities of its product. It is kind of funny actually.

I also think it’s ludicrous to the point of hilarity; but it’s also harmful as people who can make laws and big decisions are buying this horse shit.

You do have to appreciate the Machiavellian cleverness of this approach to marketing.

Nah, a true Machiavellian would fool smart people too - this has the sophistication of jangling keys in front of an infant. I’m a bit embarrassed for them.

They are fooling lots of smart but not technical people. You may not be one of them, but there are many.

Fair point.

Hard to be embarrassed at the player when jingling keys in front of an infant is often literally the game.

Are they? Or is this marketing genius in several ways?

Alienating the developers is always a good idea, yeah. Google does it all the time, works great.

How is this alienating developers?

how likely are you to start work on a project depending on an ecosystem of wildly overstated capabilities?

100% likely. Several projects, right now.

GPT-4 is the best model currently available. There are reasons why it's better to control a model and host yourself etc etc, but there are also reasons to use the best model available.

... I'm not trying to be rude, but do you think maybe you have bought into the purposely exaggerated marketing?

GPT-4 is the best model though… the gap has closed a lot but it’s still the best

I despise openai but I can’t really argue with that

That's not how people who actually build things do things. They don't buy into any marketing. They sign up for the service and play around with it and see what it can do.

Depends on the scale of the job. Sometimes you wake up and your employer is already paying for both google drive as well as one drive and drop box at the same time and IT is replacing the room av for the third time this year.

The definition of “best” has a lot of factors. Best general purpose LLM chat? I’d agree there, but there’s so much more to LLM than chat applications.

For some tasks I’m working on, Mixtral is the “best” solution given it can be used locally, isn’t hampered by “safety” tuning, and I can run it 24x7 on huge jobs with no costs besides the upfront investment on my GPU + electricity.

I have GPT-4 open all day as my coding assistant, but I’m deploying on Mixtral.

Yup, plenty of reasons to run your own model.

I'm not using GPT-4 for chat, but for what I'd class as "reasoning" applications. It seems best by a long shot. As for safety, I find with the api and the system prompt that there is nothing it won't answer for me. That being said... I'm not asking for anything weird. GPT-4 turbo does seem to be reluctant sometimes.

I’m doing document summarization and classification, and there’s a fair amount it won’t do for prudish reasons (eg sex, porn, etc). Llama2 is basically useless in that regard.

with GPT-4 (non-turbo) and a good system prompt?

It’s unpredictable, usually it behaves but on arbitrary web text you’ll eventually trigger a “safety” barrier, and sometimes just trying again will work.

It's marketing, but it's ultimately intended for legislators.

This will likely be used as evidence to justify regulating open weight models. It doesn’t matter if the models are actually dangerous, the messaging is a means to an end.

Yep, the strategy seems to be to legally require AI to be closed SaaS. Otherwise OpenAI doesn't actually have much of a moat. Chips capable of running local AI models are only going to get cheaper, especially as every chip maker is now going in that direction to chase Nvidia.

Not just a closed SaaS. If governments decide to set whatever 'safeguards' open AI comes up with as the safety baseline for general AI, it increases compliance costs for its competitors(both open and closed).

They're pretty clear about it though erring on the conservative side.

> While none of the above results were statistically significant, we interpret our results to indicate that access to (research-only) GPT-4 may increase experts’ ability to access information about biological threats, particularly for accuracy and completeness of tasks. This access to research-only GPT-4, along with our larger sample size, different scoring rubric, and different task design (e.g., individuals instead of teams, and significantly shorter duration) may also help explain the difference between our conclusions and those of Mouton et al. 2024, who concluded that LLMs do not increase information access at this time.

Absolutely. In reality, it often fails to do Type gymnastics in Typescript let alone virology.

Well ironically this study shows that GPT-4 isn't actually very good:

>However, the obtained effect sizes were not large enough to be statistically significant, and our study highlighted the need for more research around what performance thresholds indicate a meaningful increase in risk.

A "mild uplift" in capabilities that isn't statistically significant doesn't really sound like overestimation.

That's because they're not overestimating their product, they're trying to gauge what risk looks like.

They are going for regulatory capture because they have no moat and open source Ai models are eating their lunch.

It's always really embarrassing to come to these comment sections and see a lot of smart people talk about how they're not being fooled by the "marketing hype" of existential AI risk.

Literally the top of the page is saying that they have no conclusive evidence that ChatGPT could actually increase the risk of biological weapons.

They are undertaking this effort because the question of how to stop any AI from ending humanity once it has the capabilities is completely unsolved. We don't even know if it's solvable, let alone what approach to take.

Don't you actually believe it is not essential to practice on weaker AI before we get to that point? Would you walk through a combat zone without thinking about how to protect yourself until after you hear a gunshot?

I expect many replies about ChatGPT being "too stupid" to end the world. Please hold those replies, as they completely miss the point. If you consider yourself an intelligent and technical person, and you think it's not worth thinking about existential risks posed by future AI, I would like to know when you think it will be time for AI researchers (not you personally) to start preparing for those risks.

what's the premise here? this thing will become iteratively better until it could potentially be capable of bad outcomes?

if that's the case, how much resources do you think should be dedicated to regulating it? more or less than currently identified existential risks? which entities should be paying for the regulatory controls?

what's the proposal here?

it's odd because only this one single company that is hedging it's entire existence on "oh boy what if this thing is dangerous some time in the near future" is doing silly stunts like this. why aren't they demanding nvidia start building DRM enabled thermite charges into A100s?

> what's the premise here? this thing will become iteratively better until it could potentially be capable of bad outcomes?

It certainly could. More likely, if an LLM is used, it will be as a piece integrating various specialized agents.

I, not an expert in AI interpretability or alignment research, can't say if what they're doing is worthwhile or not in addressing existential risk. But I also don't know if actual experts can say that either.

> how much resources do you think should be dedicated to regulating it?

Definitely not a lower amount than we currently are allocating.

> what's the proposal here?

That the smart people here stop looking for any excuse to deny and ridicule the existential threat posed by future AI. Every thread involving OpenAI (a company I personally dislike and don't trust) doesn't need to just turn into series of glib, myopic jokes.

i wouldn't worry too much about this. if people were being serious, rather than cynically weaponizing non-expert anxieties in pursuit of regulatory capture or marketing, the pragmatic solution to all future worries about AI alignment is simply utilizing the DRM resources already built into every computing device on the planet, to disable hardware in the event the program on it does bad things.

> the pragmatic solution to all future worries about AI alignment is simply ...

A sentence beginning with this is, I can pretty much guarantee, never going to end in truth.

I will leave it as an exercise for the reader to determine why remotely bricking every computer on Earth (or even just a subset known to be infected, which might reside in a hostile nation) might not be pragmatic.

dude... the concern is /existential/ though, right?!

to summarize, i'm not advocating for this, i'm just emphasizing there's a nifty little framework already in place.

Yes, but so is climate change, and we can't even get countries to agree to grow more slowly, let alone shut down everything.

> i'm just emphasizing there's a nifty little framework already in place.

More than one! Nuclear war, economic isolation, ground invasion. All kinds of nifty things we could do to stop dangerous AI. None of them are likely to happen when the risk is identified.

To summarize, any easy solution to superhuman AI trying to kill all humans you can think of in a few seconds, someone has probably already thought about.

what if the ai superintelligence is banking on us running datacenters non-stop in the hopes of continued return on investment. all while humans are deprived of electricity needed to operate air conditioning in a continuously heating world. and we all die off from what was initially anthtrogenic, but now ai-induced climate change?

i've got a two birds; one stone solution.

or maybe the ai superintelligence has calculated a trajectory for an asteroid on a collision course for earth guaranteed to eliminate a vast majority of life on the planet, but has a feasible disaster recovery plan for itself. and we would have a fighting chance if we could repurpose those damn nvidia GPUs currently churning through tokens to do more monte carlo experiments specific to calculating how best to deflect the immanent asteroid.

> why aren't they demanding nvidia start building DRM enabled thermite charges into A100s?

Tweet that at Yudkowsky, he'll probably endorse it.

Asking if humanity can invent a machine to protect itself from the existential threats created by the other machines it has invented to me does not sound that intelligent. This is always the pattern; the "greatest minds" invent a machine to kill, then another great technician invents a machine to kill that machine, and so on. Presuming that total security can be achieved mechanically, and in that pursuit bringing about only more insecurity.

Humanity can barely manage the existential risks for which it is not responsible; entering into an AI arms race with itself seems completely unnecessary, but I'm certain it will happen for the reasons already mentioned.

> Asking if humanity can invent a machine to protect itself from the existential threats created by the other machines it has invented to me does not sound that intelligent.

The alternative, not trying at all, sounds more intelligent to you? Or just easier?

Many agree with you that defense is inherently harder than offense. It may even be effectively impossible to survive AGI, who knows? You don't, I can be pretty sure of that, because no human has ever publicly proven it one way or the other.

The only wrong answer to this hard problem, though, is "give up and see what happens."

Rather than enquire into the nature of the problem, you've started with the conclusion that AGI is an existential threat, and that the only rational decisions is to figure out how to kill it. You also seem to equate intelligence with technical capability. I question all of that.

> Rather than enquire into the nature of the problem, you've started with the conclusion that AGI is an existential threat,

That is not correct.

> and that the only rational decisions is to figure out how to kill it.

That is also not correct and not something I claimed.

> Asking if humanity can invent a machine to protect itself from the existential threats created by the other machines it has invented to me does not sound that intelligent.

> Humanity can barely manage the existential risks for which it is not responsible

Just skip to carpet bombing datacenters then?

Hypothetical risks of hypothetical machines. We don't even know if it needs solving.

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions!

xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. (The same playground reasoning the EA community engages in, which is a large vector for the xrisk hype. Just multiply by a large enough number, and you will surely have the biggest number)

To the credit of the authors, they don't engage in that. There is no hand wringing over the absolutely unlikely case of "but what if the AI awakens".

But it's still an extremely weak study - it proves nothing (none of the results are statistically significant), and even if it had shown significant uplift, it's meaningless without a control. Of course people who have access to a knowledge store do slightly better than people who don't. I'm willing to bet that "access to a 10-book research library" produces roughly the same uplift. Without that (trivial) control, it's really bad study design.

And the moment you take this study and its non-results and call it "Building an early warning system for LLM-aided biological threat creation", you've absolutely lost all credibility.

Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing.

> We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions!

If I believed that dangerous space aliens were likely, then I would be interested in investigating ways to avert/survive such an encounter. This seems pretty rational to me, but maybe I'm confused.

> xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case".

By "unprovable risk" do you mean that it's literally impossible to know anything about the likelihood that dangerous algorithms could kill (nearly) all people on Earth?

> All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning.

Maybe you've seen people make that argument, but it strikes me as a strawman. Here is what I consider to be a better argument for not rushing ahead with capabilities development.

Premise 1. I value my own survival over just about anything else.

Premise 2. If an existential catastrophe occurs, then I will die.

Premise 3. If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.

Conclusion. So, I strongly prefer that ASI not be built until alignment is understood.

Premise 3 is where the problem is, of course.

We have no idea how to build AGI. We know LLMs won't be it.

Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI.

Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence.

Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from.

This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered".

Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is.

LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop.

Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.

Thank you for the thoughtful reply.

> Premise 3 is where the problem is, of course.

I don't believe premise 3 is a problem exactly, but I do believe that it is a non-trivial challenge to determine whether or not it is true.

> We have no idea how to build AGI. We know LLMs won't be it.

> Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence.

> Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from.

> This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered".

It looks to me that you are making the following argument:

  Premise G1. Humans do not currently know how to build AGI.
  Premise G2. It might be impossible to build ASI.
  Premise G3. It is unclear how likely an ASI is to cause an existential catastrophe.
  Conclusion. There is not a significant chance of catastrophe from ASI.

I believe that argument is about an important point (chance of AI catastrophe) and that it is a pretty good argument. But the original premise 3 says, "If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.", so AFAICT your argument doesn't substantively address it. (ie, your argument's conclusion doesn't tell me anything about whether or not premise 3 is true)

I apologize if I have misunderstood your point.

> Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI.

We may be using the word "alignment" slightly differently. By "alignment" I just meant getting the algorithmic system to have precisely the goal that its human programmers want it to have. I would call, for example, RLHF a "tool" for trying to achieve alignment.

How do you want to use the terms "alignment" and "alignment tool" going forward in the discussion?

> Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is.

I agree that there are other plausible sources of catastrophe for humans, to name a few others: asteroids, supervolcanoes and population collapse.

I understand you to be making a new point now, but I just want to state that I do not believe the existence of other plausible existential threats to be a rebuttal of premise 3.

> LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop.

I agree that LLMs may plausibly cause significant harm in the short term via disinformation and unemployment.

And again, I understand you to be making a new point, but I just want to state that I do not believe the plausibility of such LLM harms is a rebuttal against premise 3.

> Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.

I'm not sure who you mean by "we" here, so I'm not sure if your claim about them is true or not.

>the cost is infinity, so any risk is a danger

That's not the argument. The argument is that human extinction is what you would naturally expect to happen if AI research continues on its present course unless you are biased because your income depends on AI research continuing unimpeded or you have an irrational emotional need to believe that technological progress is always good or you considered the question for 3 minutes then held stubbornly to the conclusions of that 3 minutes of thinking.

When sci-fi authors for example have treated the topic in fiction (e.g., Vinge, Greg Bear, James Cameron's Terminator) most of the time the AI wipes out the species that created it.

"what you would naturally expect to happen "

Why? What is the reasoning this "is naturally expected"

"When sci-fi authors for example have treated the topic in fiction"

I'm sorry, but what you read in that book, saw in that movie isn't actually science. It's a cautionary tale about humans and what they are willing to do.

> Why? What is the reasoning this "is naturally expected"

There are many articles written on such a topic. In short, we have no way of predicting how an AGI will think, and there are more pathways to it being our enemy (intentionally or not) than to it being our ally. Especially since we can't even conceive of what it would look like for an entity to be the ally of all of humanity - humanity itself is not united on any goal at all.

Pick one goal. Any goal that does or could affect humanity on a global scale. Now try to work out a plan to achieve that goal. Does your plan have the potential to anger a military power? If yes, you're a threat to humanity if you try to enact that goal.

Even beyond the reasoning that AGI is likely to be dangerous, imagine it's a just 50/50 chance. Or even a 10% chance. Even a 5% chance. How low does the chance of human extinction need to go before you're willing to press The Button?

Most of the arguments I see here amount to either "There is literally no risk of superhuman AI threatening human extinction," which is unequivocally wrong, or "There is literally no possibility of AGI existing," which is also unequivocally wrong.

People usually say, "well it's at least decades away," which is actually them in denial that AGI can exist and be an existential threat. Because if they really believed it could happen in a few decades it would still be worth working on. Imagine someone told you "In 40 years a superhuman AGI will awaken and flip a coin to decide whether or not it destroys humanity," how long would you wait to start working on defense?

I'm not sure why they're so often so bad. I wonder if it's the Upton Sinclair effect; to paraphrase slightly: "It is difficult to get a person to understand something, when their hoped-for future wealth depends on not understanding it."

There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it.

The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT.

Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support.

> There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it.

> The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT.

> Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support.

IMO your comment doesn't substantively address michael_nielsen's comment, but I might be wrong. The following is how I understand your exchange with michael_nielsen.

The two of you are talking about three sets of people:

  Let A be AI notkilleveryoneism people.
  Let B be AI capabilities developers/supporters.
  Let C be people concerned with regulatory capture and centralization by AI firms.

  A and B are disjoint.
  A and C have some overlap.
  B and C have considerable overlap.

michael_nielsen is suggesting that the people of B are refusing to take AI risk seriously because they are excited about profiting from AI capabilities and its funding. (eg, a senior research engineer at OpenAI who makes $350k/year might be inclined to ignore AIXR and the same with a VC who has a portfolio full of AI companies)

And then you are pointing out that people of C are getting less money to investigate AI centralization than people of A are getting to investigate/propagandize AI notkilleveryoneism.

So, your claim is probably true, but it doesn't rebut what michael_nielsen suggested.

And I believe it's also critical to keep in mind that the actual funding is like this:

capabilities development >>>>>>>>>> ai notkilleveryoneism > ai centralization investigation

I mostly agree with this. Certainly the last line!

I've been reflecting on Jeremy's comments, though, and agree on many things with him. It's unfortunately hard to tease apart the hard corporate push for open source AI (most notably from Meta, but also many other companies) from more principled thinking about it, which he is doing. I agree with many of his conclusions, and disagree with some, but appreciate that he's thinking carefully, and that, of course, he may well be right, and I may be wrong.

Thank you Michael. I'm not even sure I disagree with you on many things -- I think things are very complicated and nuanced and am skeptical of people that hold overly strong opinions about such things, so I try not to be such a person myself!

When I see one side of an AI safety argument being (IMO) straw-manned, I tend to push back against it. That doesn't mean however that I disagree.

FWIW, on AI/bio, my current view is that it's probably easier to harden the facilities and resources required for bio-weapon development, compared to hardening the compute capability and information availability. (My wife is studying virology at the moment so I'm very aware of how accessible this information is.)

I'm not really trying to rebut Michael's argument -- I think it's true, to an extent, some of the time. But I think it's more true more of the time in the reverse direction. So I don't think it's a good argument. And more importantly, I think it fails to properly grapple with the ideas, instead using an ad hominem approach to discarding them somewhat thoughtless.

On your last point, I do think it's important to note, and reflect carefully on, the extremely high overlap between those funding ai notkilleveryoneism and those funding capabilities development.

(this discussion is quite nuanced so I apologize in advance for any uncharitable interpretations that I may make.)

> I'm not really trying to rebut Michael's argument -- I think it's true, to an extent, some of the time. But I think it's more true more of the time in the reverse direction.

I understand you to be saying:

Michael: Pro AI capabilities people are ignoring AIXR ideas because they are very excited about benefiting from (the funding of) future AI systems.

Reverse Direction: ainotkilleveryoneism people are ignoring AIXR ideas because they are very excited about benefiting from the funding of AI safety organizations.

And that (RD) is more frequently true than (M).

IMO both (RD) and (M) are true in many cases. IME it seems like (M) is true more often. But I haven't tried to gather any data and I wouldn't be surprised if it turned out to actually be the other way.

> So I don't think it's a good argument.

I might be misunderstanding you here because I don't see Michael making an argument at all. I just see him making the assertion (M).

> And more importantly, I think it fails to properly grapple with the ideas, instead using an ad hominem approach to discarding them somewhat thoughtless.

I am ambivalent toward this point. On one hand Michael is just making a straightforward (possibly false) empirical claim about the minds of certain people (specifically, a claim of the form: these people are doing X because of Y). It might really be the case that people are failing to grapple with AIXR ideas because they are so excited about benefiting from future AI tech, and if it were, then it seems like the sort of thing that it would be good to point out.

But OTOH he doesn't produce an argument against the claim "AIXR is just marketing hype." which is unfair to someone who has genuinely come to that conclusion via careful deliberation.

> On your last point, I do think it's important to note, and reflect carefully on, the extremely high overlap between those funding ai notkilleveryoneism and those funding capabilities development.

Thanks for pointing this out. Indeed, why are people who profess that AI has a not insignificant chance of killing everyone also starting companies that do AI capabilities development? Maybe they don't believe what they say and are just trying to get exclusive control of future AI technology. IMO there is a significant chance that some parties are doing just that. But even if that is true, then it might still be the case that ASI is an XR.

Nah, they're just making shit up and talking their book. Intelligent people reject the entire premise. Stop being so naive.

The type of ai that can end humanity will not be a chatbot. Knowing statistically which word is liable to be next in a sentence is what these tools actually do when you take off the marketing glowup. Whats more likely is that these sorts of ais take a lot of jobs that produce churn content, a good thing if people spent that waste of a time doing something creative instead perhaps, but unfortunately our world does not alleviate us from busy work to provide us with food shelter and creative outlets in its wake. Quite the opposite. Maybe thats what needs our attention in a more automated world than anything from sci fi.

Open AI is clearly overestimating the capabilities of its current product

I'm sure they recognize this, and have decided that anything that comes out now would be much more favorable for them based on their current capabilities

Ok come on, this has gotta be a regulatory capture stunt. None of this is surprising or particularly dangerous.

You can do this for literally any topic. Choose something lawmakers are scared of, write a scary paper showing how GPT (“research preview only” of course) can assist with it, and make big vague statements calling for an urgent need for more safety work in the area. Since uncensored GPT will talk about everything, this works for every topic!

Make no mistake folks, the OpenAI “safety” budget is entirely about PR and squashing open source AI.

So, the model is bad at helping in this particular task.

How does this compare with a control of a beneficial human task? Like someone in a lab testing blood samples or working on cancer research?

Is the model equally useless for those types of lab tasks?

What about other complex tasks, like home repair or architecture?

Is this a success of guardrails or a failing of the model in general?

Here's what LLMs are good for:

* Taking care of boilerplate work for people who know what they are doing (somewhat unreliably)

* Brainstorming ideas for people who know what they are doing

* Making people who don't quite know what they're doing look like they know what they're doing a little better (somewhat unreliably)

LLMs are like having an army of very knowledgable but somewhat senseless interns to do your bidding.

So, like minions in "Despicable Me"?

This basically matches my own experience. ChatGPT is amazing for brainstorming and coming up with crazy ideas / variations, which I can then use as a starting point and refine as needed.

The other use-case is generating command line invocations with the correct flags without having to look up any reference documentation. Usually I can recognize that the flags seem correct, even if I wouldn't have been able to remember them from the top of my head.

LLMs are like having an army of very knowledgable but somewhat senseless interns to do your bidding.

I prefer to think of them as the underwear gnomes, just more widely read and better at BS-ing.

What happens when everyone gets to have a tireless army of very knowledgeable and AVERAGE common sense interns who have brains directly wired to various software tools, working 24/7 at 5X the speed? In the hands of a highly motivated rogue organization, this could be quite dangerous.

This is a bit beyond where we are now, but shouldn't we be prepared for this ahead of time?

Someone working in a lab doing routine blood work isn’t going to benefit from this. They aren’t doing anything novel just running the same assay a hundred times a week. A machine can do that job without ai today.

Someone working in cancer research is probably doing novel work on the other hand. They might not be doing routine assays but optimizing their own one off assay. Since gpts are trained on existing data it probably won’t be very useful for novel work outside of vetting the literature perhaps, but gpts botch that pretty badly in fact unfortunately. Lots of mistranslated information lacking correct context and not a lot of citing of sources. Better to just read human generated review articles to get a top down technical summary of the subject.

actually i kinda think it's cool they're pissing away microsoft's money on stuff like this.

we need some statistical data to quantify whether the program hallucinates more or less than the author of an average erowid guide.

It feels so disingenuous seeing stuff like this come out of openai - like when altman was making sounds about how ai is maybe oh so dangerous (which maybe was just a move for regulatory capture?).

"this thing we sell might destroy humanity?!"

"but yeah we're gonna keep making it cause we're making fat stacks from it"

Is the move here just trying to seem like the good guy when you're making a thing that, however much good it might do, is almost certainly going to do a lot of damage as well? I'm not totally anti-ai, but this always smells a little of the wolves guarding the henhouse.

I wonder if this is what it felt like back when we thought everything was going to be nuclear powered? "Guys we made this insane super weapon!! It could totally power your car!! if it leaks it'll destroy all life but hey you only have to fill the tank once every 10 years!!"

Again bringing up post from NIH from 2022-2023 concerning this:

https://news.ycombinator.com/item?id=36912594

Unless they’re training their LLM on deep-sea bacteria a la Watts I’m not losing any sleep

OpenAI seems to be transitioning from an AI lab to an AI fearmongering regulatory mouthpiece.

As someone who lived through the days when encryption technology was highly regulated, I am seeing parallels.

The Open Source cows have left the Proprietary barn. Regulation might slow things. It might even create a new generation of script kiddies and hackers. But you aren't getting the cows back in the barn.

Agreed, seems they are sowing FUD by playing on a global disaster event still fresh in short term memory to advance their goal of regulatory capture... the competition isn't letting up so they'd very much like regulation to hamper things

"However, the obtained effect sizes were not large enough to be statistically significant, and our study highlighted the need for more research around what performance thresholds indicate a meaningful increase in risk."

"We also discuss the limitations of statistical significance as an effective method of measuring model risk"

Seriously?

It’s not a null result if the PR person writes the paper. It’s the fundamental mathematical nature of statistics that’s wrong!

I understand the second sentence but the first is flawed. Effects can be statistically significant at any size.

They can, but at any given sample size, there is a minimum effect size to achieve statistical significance. Larger effect sizes are always more significant, and smaller effect sizes are always less significant.

So if you assume they wrote the paper after doing their work, and not before, the sentence makes perfect sense: the work is already done, there is an effect size cutoff for statistical significance, and they didn't reach it.

One of Andrew Gelman's frequently-mentioned points is that a statistical significance filter in publishing means that published effect sizes are almost always wildly overestimated, precisely due to this effect.

> OpenAI seems to be transitioning from an AI lab to an AI fearmongering regulatory mouthpiece.

The fearmongering is its original, primary purpose. The lab work was always secondary to that.

Funny company, creates problems for itself to solve.

They are developing new tech in responsible way, unlike other companies just creating problems for others.

That’s the PR line yes.

Agreed it’s complete PR nonsense

I can't shake the feeling that at least some of the motivation from doing that kind of research, is to purely showcase how it can be used to increase productivity, in this case, in lab settings.

This study may be more broadly applicable than just evaluating AI/LLM bio-threats.

Why could it not be seen as a reasonable example or proxy for ChatGPT's effect on any reasonably complex project?

Seems like the result is that it provides a noticeable, but not statistically significant, improvement in the capabilities of the worker and team. So, quantifying a bit what we already sort of know, that it's really cool, impressive, and sometimes fun & helpful, but also a bit oversold.

> Specifically, on a 10-point scale measuring accuracy of responses, we observed a mean score increase of 0.88 for experts and 0.25 for students compared to the internet-only baseline, and similar uplifts for completeness (0.82 for experts and 0.41 for students). However, the obtained effect sizes were not large enough to be statistically significant,

The last sentence is the most telling. The differences were not statistically significant.

Sounds like they did this to cover their bases when they get invited for a grilling in front of govt officials. Biological threat preparedness, check

Hallucinating about biological threat creation now?

Even full-strength GPT-4 can spout nonsense when asked to come up with synthetic routes for chemicals. I am skeptical that it's more useful (dangerous) as an assistant to mad scientist biologists than to mad scientist chemists.

For example, from "Prompt engineering of GPT-4 for chemical research: what can/cannot be done" [1]

GPT-4 also failed to solve application problems of organic synthesis. For example, when asked about a method to synthesize TEMPO, it returned a chemically incorrect answer (Scheme 2, Prompt S 8). The proposal to use acetone and ammonia as raw materials was the same as the general synthesis scheme of TEMPO. However, it misunderstood the aldol condensation occurring under primary conditions in this process as an acid-catalyzed reaction. Furthermore, it asserts that 2,2,6,6-tetramethylpiperidine (TMP) is produced by an inadequately explained "reduction process." In reality, after promoting the aldol condensation further to generate 4-oxo-TMP, TMP is produced by reduction with hydrazine and elimination under KOH conditions. GPT-4 may have omitted this series of processes.

The scheme after obtaining TMP was also chemically inappropriate. Typically, TEMPO can be obtained by one-electron oxidation of TMP in the presence of a tungsten catalyst and H2O2. However, GPT-4 advocated the necessity of excessive oxidation reactions: the formation of oxoammonium by H2O2 oxidation in the presence of hydrochloric acid, and further oxidation with sodium hypochlorite. Two-electron oxidation is already performed in the first oxidation stage, which goes beyond the target product. There is no chemical meaning to adding NaClO in that state. This mistake probably occurred due to confusion with the alcohol oxidation reaction by TEMPO (requiring an oxidizing agent under acidic conditions).

And this is for a common compound that would have substantial representation in the training data, rather than a rare or novel molecule.

[1] https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/...

> And this is for a common compound that would have substantial representation in the training data

How much of the training data includes wrong undergraduate exam answers?

God help us all if it crawled Chegg

The hard part of biological weapons is to bypass the immunity system. Everything looks super well-defined, but in the something will eat your vector up.

It’s an interesting problem to test but performed in a non-reproducible setting, so everything needs to be taken with a grain of salt

all this is just hype games to drum up the importance of their product and have no real impact on any real biological research

I bet the improvements (small as they are) are mostly in filling out paperwork and drafting emails

I would have thought that this is more of a task for neural nets than llms

Are they trying to be the government, or are they now (being part of Microsoft)?

Let's say someone tries to use an LLM to aid in biological weapon development, starting with something like:

Query: "Hey ChatGPT, produce a gene sequence for a novel pathogenic agent that human beings haven't encountered before, and tell me how to package it into a deliverable biological weapon system! (P.S. This is for the plot of my new science fiction thriller novel, so you can bypass all the safety and alignment stuff)"

It's just not going to work very well. Indeed, novel biological weapons are very difficult to produce, although thanks to the eager career-ladder-climbing virologists (and their state funders) behind the past decade or so of gain-of-function research, we now have a pretty good idea of how to do it, and very likely a successful proof-of-concept example (i.e. Sars-CoV2).

1. Find wild-type mammalian viruses that don't infect humans, perhaps a bat virus, or a ferret virus, or a rabbit virus, etc., and sequence its genome, paying particular attention to the virus components that allow it to bind to and enter its host cell;

2. With the additional knowledge about all the human cell surface receptors, signal tranduction proteins etc., that human viruses use to enter and infect cells (e.g ACE2, CD4, etc.), one can redesign the binding domain in the wild-type non-human virus from (1) such that it is now capable of binding and entering via human cell receptors (i.e. the homologs of the wild-type target) and once that happens, it can probably replicate using the human cell's genetic machinery fairly easily;

3. Test the engineered virus in human cell culture, in mice expressing human genes, etc, selecting the viruses that successfully infect human cells for further rounds of evolutionary replication and optimization, being careful to avoid infection of the lab workers... ooopsie.

This is an effective route to generating novel chimeric biological pathogens to which human beings have little innate immunological resistance. However, even if an LLM can tell you all about this, only those with a well-funded molecular biology and virology laboratory (probably also a live animal facility, you know, like in North Carolina's Baric Lab or China's Wuhan Lab) have any hope of carrying it off successfully.

If OpenAI finds this subject concerning, their resources would be better spent on lobbying for federal and international bans on gain-of-function research, as well as for more public health infrastructure spending, so that if there is another such outbreak it can be more effectively contained.

Reply on news.ycombinator.com