Watch AI materials-science and bioscience abilities closely
lesswrong.com
lesswrong.com
I'm pretty bearish on the idea of AI becoming Skynet from Terminator 2, but I do think there's some credibility to the bioterrorism argument. When I used to talk with people that work in orgs tasked with preventing terrorism, this was the thing that was keeping them up at night, and this was before AI.
/the oxygen side pre-burner in a full flow staged combustion rocket engine is a nasty place and figuring out a material that will survive in that environment was the key (or one of the keys) to making Raptor a reality.
I think its a consequence of the improvements in harnesses and LLMs of the last year. Before they had a hard time self-reviewing the results and proceeding to a second iteration, now they are quite good at it although not perfect.
Unfortunately, "AI that's competent at bioweapon R&D" might be an easier mark to hit than "AI that can put a dent in the aging process". And biorisks can have a very favorable "resource input -> suffering output" rate of returns.
But I do agree that if we're on the verge* of von Neumann replicators and getting replaced by Pokhumans** to cheer on the ASI, then curing cancer, immortality and cryo-suspension are likely trivial achievements in comparison.
*Nope, not even close, but don't tell that to the ASI cultists, that's like questioning the veracity of the gospels now.
**Yes, that's the strong claim in AI 2027. And the democratic party apparently believes it.
One thing you've got to give them credit for is that they've been talking about AI -- about the present moment, really -- from the beginning. Nick Bostrom, who wrote "Superintelligence" about ten years ago, has always been in their orbit.
There isn’t enough here for it to be interesting.
IOW:
> There isn’t enough here for it to be interesting.
True - for you and the dozens of folks like you, to paraphrase a meme.
and then transposed into an ai chat prompt to get the analytical momentum for some grand scheme that gets published to whatever extent ie a comment, blog post, or whatever textual content that then gets ingested as training data…llm rinse repeat. not sure where this is going. is anyone?
I’m not sure it will ever be given its dues for its prescience. The whole rationality project was started with the explicit aim of upgrading participants’ reasoning abilities so that when AI came along for real, there would be people capable of thinking it through without making all the usual mistakes humans make when trying to reason about things.
In the discussions I read today, everyone involved from commenters up to CEOs are making all and exactly those mistakes, meaning it is in my opinion perhaps the greatest called shot of all time.
Grifters are opportunistic short-termists. Whatever you can say about these rationalists, they aren’t that.
Sure each 2-cent philosophical posts isn’t worth much on its own but add up many of them over time and we get something more than its parts.
I mean, we’re talking about real physical things, not first principles thought experiments here. Thus, they have limits imposed by physics and biology. Limits on transmission rate, lethality, particle size, evolutionary change rates, and so on. All of these interact and define a design space that nature has been searching for at unimaginable scale for as long as humans have existed. And yet here we are! Seems like “wiping humanity” might be really hard, perhaps impossible for a virus.
Perhaps I’m being uncharitable or insufficiently creative, but these people strike me as deeply unserious in their engagement with the actually real physical world of biology.
EDIT: Also, in this world, couldn’t an AI just “one shot” a vaccine and an effective treatment?
Also disgruntled is more or less unnecessary when combined with postdoc. Sort of goes without saying.
And going into space is pointless why would any animal ever develop such a facility :)
This kind of nerd sniped me, so I looked into it and turns out no, planes are more efficient than birds!
You have to look at the energy spent for moving a fixed mass over a fixed distance and for the most efficient birds I looked into, that’s about 300g of fat burned for 300g of “dry weight” for a trip of ~11,000km. (bar-tailed godwits)
For a plane that’s roughly 80 tonnes of kerosene for 170 tonnes dry weight on a similar distance (787). Energy density of fat and kerosene are actually pretty close so you can just look at the kg of fuel burned per kg of payload for the trip. For a bird that’s 1kg of fuel per kg transported and for the plane that’s ~0.5kg per kg. So roughly twice as efficient.
So from a pure energy point of view it’d be more efficient to put all the migratory birds in a 787 than letting them fly on their own!
And you could argue the plane also has a baseline metabolism ;)
You could not scale down a plane to the size of a bird and match their efficiency. On the other hand you also couldn’t scale a bird to the size of a plane so I guess we can call it even!
As a slight aside - since I was talking about the weakness of biological analogy on HN recently - the way humans fly is very very different to how (most) birds do. If nanomachines actually could be built by humans (non-biological nanomachines) they might be as different to virus/cells/organisms as plane wings are to bird wings.
Of course, also please do not prompt a frontier model to make nanomachines, either, thankyou.
Missing organs, odd tissue performing unclear functions, different, new or missing muscles.
It’s unclear whether a world ending virus is even possible. There might be better luck with prions, or fungi.
>EDIT: Also, in this world, couldn’t an AI just “one shot” a vaccine and an effective treatment?
This does not follow automatically. E.g. it is enormously easier to create an ICBM than to create something capable of intercepting an ICBM. Some technologies naturally benefit an attacker and some naturally benefit a defender.
Also there are plenty of doomer scenarios along the lines of e.g. some ethnosupremacist cult making a virus that kills everyone outside of their group.
And that was my reaction 15 or 20 years ago. It is only harder to poke holes in the logic now than it was then.
If you actually look at the trajectory of post-LLM AI developments, the successes have all been in areas where the major labs have some way to get their hands on s**tons of correlated data. That's why they're pushing into maths now. I see no reason to assume that will be replicated with biological science data, which is extremely proprietary and not required to be exposed as part of the end product. It doesn't even have an equivalent to a black market in piracy like books or movies did. This data was not valuable to the public before now.
Coding is, inherently, quite amenable to that. Is bioscience?
They're certainly working on making it so:
https://www.anthropic.com/news/model-hardware-standard-resea...
and then what?
was the needle move at all?
The best good thing possible for humans who are all suffering from an extremely painful, incurable disease might just be ending that misery.
The worst bad thing possible in a perfect utopia might be a minor inconvenience comparatively.
You say this like it's a truism, but I don't think it is. The bad things may be categorically easier than the good things. It's easier to destabilize a delicate system than to stabilize it. It's easier to break something than to fix it.