OpenAI acknowledges new models increase risk of misuse to create bioweapons | Hacker News Reader

OpenAI acknowledges new models increase risk of misuse to create bioweapons

ft.com

25 pointsmostcallmeyt38 comments

This feels like a marketing move to me, claim your model is so powerful that it’s dangerous, and people will be more interested in using it.

Coincidentally, governments should prevent anyone from building an open source competitor, which would be too dangerous to be out in the wild.

The same sneaky ploy used to restrict nuclear weapons to a mere handful of "great powers". Even today, most people believe that nuclear weapons are dangerous.

Unfortunately we have tried the destruction power of nuclear weapons on muck up cities and actual cities. This bold claim is not in agreement with experiment.

The danger of nuclear is not the explosion, that's nothing special, really.

Maybe you can't have such a big explosion with conventional weapons but you can just use more.

The problem is contamination.

Invisible and deadly

> danger of nuclear is not the explosion, that's nothing special, really

It's absolutely the explosion. (And fires.) You can make a similar boom as historical (not modern) nukes with conventional weapons, but not in a form factor that fits on the tip of an ICBM.

Same schtick since at least gpt2

> feels like a marketing move to me

It's a fundraising move. OpenAI becomes concerned about how dangerous the things it's selling are when it comes time to fundraise. This nonsense doesn't work that well in America anymore, in part because our politicians bite the bait and then start regulating, which Altmen et al don't want, but you can absolutely get a term sheet in the Middle East and Asia by linking yourself to the national security budget.

Every time I read one of these from OpenAI I'm strongly reminded of Men Who Stare At Goats

No, prescient people were aware of these risks a long time ago, before any OpenAI company:

In 2022, a survey of AI researchers with a 17% response rate found that the majority believed there is a 10 percent or greater chance that human inability to control AI will cause an existential catastrophe.

Altman, 2015: https://blog.samaltman.com/machine-intelligence-part-1

Bostrom 2014: https://www.goodreads.com/book/show/20527133-superintelligen...

Stephen Hawking, 2014: https://www.bbc.com/news/technology-30290540

Take your pick.

Are Geoff Hinton, Yoshua Bengio, and Douglas Hofstadter also marketing?

Yes, they absolutely are! If this thing they are working on (or prognosticating about) is world endingly dangerous, it increases their importance.

Does your model of the world have anyone who genuinely believes that these things will someday develop levels of expertise in weapons (bio, cyber, or other) which, in the hands of terrorists or other malefactors, would be a real danger?

My model of the world does not recognize the possibility of a model that is simultaneously capable of designing new useful molecules while also not being able to create harmful ones.

Missile guidance PS2, remember that one? It's an old trick but media still fall for it.

Military grade... encryption

Government can try what it wants, we've done this before

Is the risk higher than with any chemistry or microbiology grad student?

Possibly. Imagine if the "school shooters" of tomorrow are depressed kids who release a bioweapon at their high school.

Is it just way easier/accessible to make a bioweapon then I think it is? Serious question. I've seen this pop up a few times.

One doesn’t need advance chemistry, perhaps what makes it harder to do it is the access to the materials and enough motivation.

Source: my limited understanding.

Depends what you're trying to do, light weight bio weapons are pretty easy to make, not an expert but there is ammonia and chlorine that can be used easily. I presume cytotoxins are among the easier of the "bad ones", but the precursors are monitored. that said, I think the risk here isn't about teaching people how to make the stuff we know about, it's about inventing new stuff we've not thought about?

Those are examples of chemical agents. Bioweapons (biological weapons) are weaponized pathogens like viruses or bacteria.

in theory you can print out any virus given the sequence which you can find just from googling (e.g. https://www.ncbi.nlm.nih.gov/nuccore/1798174254 attaaaggtt...)

Then you have to boot it up. Bio equivalent of putting the lightning in the rock (semiconductors). The DNA is like the lightning (like the code) the rock is the biological system you drop it into to run it. Booting viruses is not super easy.

In other news, Gutenberg's ghost acknowledges that the printing press increases the risk of misuse to create weapons, misinformation...

There's a bit of a difference between producing the laptop that can connect to the internet and creating a web site with information on how to kill someone.

Step 1. Buy a gun and bullets Step 2. Shoot someone

Oh no, now this website has information on how to kill someone.

I'm not a bioweapons expert, but I think there's a lot of equipment and materials required to build a bioweapon of sufficient lethality and efficacy.

Look at anthrax for an example of a bioweapon with only one of those.

Go to a beauty spa and take their supply of Botox. Boom incredibly toxic bioweapon. Or grow some Castor plants, the beans contain ricin.

You can get PTFE spray in any hardware store.

I think when most people talk about a bioweapon to be concerned about they mean something that can be deployed against a large number of people.

IDK much about Botox as a weapon, but can you use it to deploy broadly? With ricin, that's just another poison, so how's that better than rat poison?

It is more just that there are a wide variety of existing ways available to harm or kill the population. No need to invoke the specter that AI magic is going to engineer some super bug. Plenty of virulent pathogens already around.

You could trivially cause enormous harm by sabotaging electrical distribution, water treatment, hospitals, airports, etc. Go to any food manufacturer/distributor and contaminate the products with listeria or such.

> Oh no, now this website has information on how to kill someone.

That's just a bad faith response.

> I'm not a bioweapons expert

That's the whole point, innit? You don't know how to do, so you can't. Publishing functioning recipes using easily accessed materials would provide you and countless others with a bio-weapon. Thus the publication is the problem, not the tech stack (internet, pcs, servers, etc.).

Therefore, OpenAI publishing risky information is equally at fault.

Can't we make the crawler just a little smarter then to avoid chemistry websites?

Given how shit GPT is at programming and the amount of training data available in this domain - I highly doubt it would be more useful in this area over a Google search.

Your Google search will probably send you to Amazon or Pinterest.

Yeah, the chatbot's big draw for me is sidestepping the SEO wasteland.

Yeah but once you find the answer you can judge the source. Glue on a pizza equivalent in chemistry would be mustard gassing yourself ?

Can't read the article because of the paywall, so this may/not be related...

Is it fair to say that the real liability here is a dataset mapping protein/molecule structures to outcomes/effects? Hypothetically the govt could always require OpenAI to blur responses with malicious intent. But if the underlying corpus is available, what's stopping a bad actor from training another model to do the same thing?

I guess the question I'm asking here is what risk is unique to their model if not the data it was trained on?

https://archive.is/xmuVF

Reply on news.ycombinator.com