Dual use of artificial-intelligence-powered drug discovery
nature.com
nature.com
"Risk of misuse The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting."!!!
Operating in the rarefied air of 'science' they were unaware of the risk of misuse!
I don't think they can even conceive of the idea that the government might misuse this technology.
The thundering irony of this concept in a timeline where a (group of) scientist(s) discovered fission cannot be overstated.
I think we agree...??
Would it be better if nobody had access to easier ways to produce new and innovative nerve agents? Yes, of course it would be better.
Do I prefer a world where only state actors can build nerve agents, vs a world that includes state actors plus more? Yes, less sources of nerve agents is more good.
You might as well complain how pointless it is to block proliferation of nuclear weapons when malicious state actors already have them.
The world wont be safer by spreading nuclear weapons to more people, and similarly for nerve agents.
Fortunately, this sounds similar to nuclear weapons in another way, designing the weapon is not the hard part that limits them to state actors, its producing them that's hard.
Still, nuclear powers don't tend to publish their weapon plans publicly, for good reason, and the same logic should apply here.
I'm not using the comparison to imply this is as serious as nuclear proliferation btw, just that it's a useful example of knowledge that really should be kept limited.
The masses will use AI to do bioweapon design? That doesn't sound like something most people could do.
Knowledge and resources necessary to develop AI-proposed toxins: multiple PhDs, thousands of square feet (probably 10s to low 100s of thousands) of lab space, including nasty stuff, like bromides and heavy metals, and testing and evaluation of various delivery vehicles, ranging from canes with needles to air-launched warheads with atomizers.
> In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.
For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well.
Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated.
Someone else in the thread asked about next steps. Those would be good next steps.
The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
> We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them.
Is that enough?
Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
Your nearest research university library probably holds publications about the effects and synthesis of chemical warfare agents, like Some aspects of the chemistry and toxic action of organic compounds containing phosphorus and fluorine by Bernard Charles Saunders: https://catalog.lib.uchicago.edu/vufind/Record/587946
Which is also scanned and available online: https://archive.org/details/B-001-026-884-ALL
[1] https://en.wikipedia.org/wiki/Born_secret
[2] https://en.wikipedia.org/wiki/United_States_v._Progressive,_....
That applies to government employees, who willingly agree to be bound by classification. I think that if someone outside had the information, they could publish it (probably a very bad idea).
Also, the press has a right to publish classified information.
Certainly. However, here they have identified and published not the discovered molecules but only the identification process (and only in fairly broad strokes, at that), and shied away from working further on synthesis (we know enough to state that synthesis would be possible and practical for nearly any candidate molecule).
Sure, as a society we can expand ethical guidelines to some sort of "thou shall not optimize for toxicity", but enforcing it poses some serious challenges, not least of which is the fact that this has demonstrated that the process works even with fairly innocuous and public training data.
Expect to see a bunch of funding for research into technologies around toxin identification and detection, as well as the rapid creation and synthesis of antitoxins and other prophylactic measures (cheaper filtering, maybe). Perhaps even eventual (as in a decade out at least) "hardening" of organisms against toxicological weapons (possibly positioned as research into the mechanisms of acquiring pesticide resistance and similar).
I'm pretty sure that whatever a private individual figures out on his own is not "classified", since it's not a government secret. Is there actually a US law banning people from even talking about WMDs?
If you can actually afford your rights, be my guest!
And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it.
You don't even have to attack humans. Our society is dependent on a lot of assumptions.
And just as VWH states, I don't think it can be defended against. It's scary.
We should really be scared of the lone and bored grad student making a wheat killer and accidentally dropping the test tube.
On the topic of plants, instead of attacking crops directly, attack mycorrhiza. Plants can't get nutrients without them.
See the 2007 novel Ill Wind by Kevin Anderson and Doug Beason for an exploration of this scenario:
https://books.google.com/books/about/Ill_Wind.html?id=wWIgO7...
(Top of the front page, 56 comments.)
I'd advise against doing more search on these topics if you want to sleep well at night and/or if you don't think you're already on one "naughty list" or another ;)
You can already buy extremely potent toxins that also work against humans at the hardware store (some pesticides/insecticides/rat poisons).
New toxins might be interesting for covert operations, for example killing someone without being detected by a toxicological report, but this seem to have limited usage.
This seems inevitable because the rise of various threats is also inevitable via technology, and us humans like to be safe.
Being able to identify many such compounds very quickly implies some new threat models that are a bit disturbing.
Of the 40000 substances identified, perhaps one of them has another interesting characteristic we've never seen in a toxic compound before? Perhaps such characteristics could be idenfified and filtered for, and perhaps someone could pin down something entirely unique and outside of any threat-assessment anyone's produced.
>and not surprising at all it's obvious in retrospect, but they claim that the threat vector had not been seriously concidered before, and I see no reason to doubt their assessment. Many things aren't surprising once they've been pointed out, but were missed for far too long beforehand, and I personally had never concidered this before, despite seeing multiple articles about AI-generated substances for positive purposes.
But it is still interesting that the network generalized to these toxic nerve agents without having them in the training set
Unless you mean that, by definition, anyone who violates the Geneva Conventions doesn't care about them, you are overlooking that most major militaries observe the Geneva Conventions to a great extent. They are cleverly written to make it in everyone's interests.