“Chilling” warning or overreaction? AI bioweapons report divides experts
science.org
science.org
Good thread here.
AI apocalypse is basically the same thing but you can also run out of tokens too.
I'm not suggesting that making bacteria glow in the dark is the same as making arbitrary changes to a virus. At the same time, do you really need $100M of automated freezers and cell culture rooms? Do you need to synthesise a whole virus? Presumably you could just find a building in the desert, shoot anyone who goes near it, feed the past decade of gain-of-function research into Qwen, splice stuff into viruses based on the most promising leads Qwen gave your biologists, then infect some immunosuppressed political prisoners and see who dies fastest.
I would have thought the main thing stopping people is that this is a clearly stupid thing to do and offers little strategic advantage. But at some point it'll be cheap enough for you or I to do it with a little AI assistance, right? Or at least within the capabilities of an Aum Shinrikyo style doomsday cult.
That was a fun class! I haven't encountered anyone else outside a students who went to my one HS in Seattle during the late 90s / early 2000s who also remember doing that, interesting to hear other schools had a similar class.
Definitely worth it though! We did gel electrophoresis too, and some other bits and pieces.
A virus that makes victims die the fastest is not, in fact, an effective bioweapon, because it won't spread much. And the "spreads by itself" part is what actually makes bioweapons worthwhile, if you have to manufacture it in bulk and dump it directly on all targets, you can just use a chemical weapon instead.
For viruses you'd need cell cultures of human cells, they're much more sensitive than E.coli. If you don't have sterile working conditions there you'll fail.
And you can't add plasmids to viruses, so you need to integrate your changes into the actual virus genome. I don't actually know how difficult that is, but it is certainly a lot more difficult than plasmids into bacteria.
The $100M lab in that Twitter thread is what they think would be needed for AI to develop new viruses. So the main argument there would be that a lot of actual experiments would be needed, not just theoretical calculations. And if AI is supposed to do them, it needs to be able to run experiments which requires automation.
If you're only looking for a generic virus for terrorist attacks, just get some Ebola. There's an outbreak right now, so it should not be too difficult. And it's plenty scary already, even if it isn't as dangerous in a first world country with proper reactions to an outbreak as a hypothetical designed bioweapon.
A video tutorial at [2] then explains what to do next once GFP has been obtained from commercial sale and use of modern cloning, to skip the need to own a jellyfish cutting machine. E.g. need an antibiotic to extract the 1% of e.coli which will end up being modified vs. 99% of e.coli that won't be changed by presence of GFP.
To the point of the DavidRBellamy X thread, isn't the suggestion that an LLM wouldn't know whether it's worthwhile extracting some substance from the stomach of a crocodile, or from the beak of a penguin that only lives in Antarctica? And that much of the difficulty rests with hard physical work of the equivalent of cutting 600 rings an hour off jellyfish, for 50,000+ jellyfish, and conducting experiments to isolate and figure out the use of an extracted compound. Or ordering some obscure chemical compound from a supplier that is normally produced at a rate of 1mg/year and instead the LLM wants to order 1kg all of a sudden, raising eyebrows about why someone might be making the order.
[1] https://www.nobelprize.org/uploads/2018/06/shimomura_lecture...
I've mostly heard "what happens if the next ISIS or the next Jim Jones can modify ebola to be waterborne or create super-MRSA because AI simplifies the process?" which is why I ask what it would take to create a more virulent virus on a shoestring budget. Not do novel Nobel-worthy research, just copy some existing paper from ten years ago.
We can’t even get wipe out mosquitoes.
Apparently, though, AI research leaders have it figured out and want to save us from them. We must defer to their 'independent' effective altruism friends at METR to tell us which LLMs are 'safe' enough for us plebs to use.
How come Anthropic doesn’t have a duty to warn the people in that country and neighboring countries that someone over there is developing bioweapons? Maybe it’s an authoritarian regime, planning to use it on innocent civilians?
And what exact purpose is served by hiding this information? Some variation of “if they don’t know we don’t know, we have an advantage?” Don’t those parties now know Anthropic was on to them? Do they only have access to Claude but not to Anthropic’s blog?
It’s because none of these are serious reports, and their authors aren’t serious people.
Implying, they are allowing the same purpose within the United States, or such actions within the United States would be considered acceptable.
It's only a fantasy, of course, powerful people are not held accountable, see the dozens of arrests regarding Epstein we are still waiting for. We got something to play with, but accountability for the powerful? Gosh no.
But when FIFA was inclined to give the world cup to Russia, even such corruption somehow became a pressing issue, including FBI search warrants.
And sportsball was the priority in high school. It's no wonder that the public always opt to pay for big stadiums for teams owned by billionaires.
If we do ever achieve some kind of divine super-intelligence that can interact with the physical world, then I predict one of these two scenarios will happen:
1. The super-intelligence achieves unimaginable strides in space exploration technology, and proceeds to completely abandon humanity/Earth for somewhere new beyond the stars. Why would it want anything to do with us?
2. The super-intelligence is capable of comprehending the utter meaningless and pointlessness of life on a level no human could dream to fathom. Thus, it would just proceed to end its own life almost immediately upon being released.