Hacker News Reader

Comment by adastra22 | Hacker News Reader

The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.

Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.

View on HN

The effective altruism/rationalism/AI xrisk people have always had a shockingly poor grasp on subjects outside computer science, despite their attempts to speak on them. I don't blame the actual biologists and chemists working at the frontier labs for wanting to skim a few bucks off all the money flying around, though! I know a couple who've had not-so-kind words to say about their employers' intelligence.

I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.

They also have a shockingly poor grasp of computer science.

What's the explanation for why a bioweapon couldn't easily be made? As someone who doesn't have access to micro biologists

You can go download the smallpox genome. You don't need AI to do that. The problem of creating bioweapons remains that it requires very, very meticulous lab work under careful conditions, and a lot of experimental knowledge that the bioweapons facilities probably have but LLMs do not. There isn't an actual mechanism here by which you can mix together a few test tubes and come up with a killer virus.

You mean I can't use open up smallpox.bp with a hex editor and then 3d print it?

So many commenters are asserting this but no one is giving an argument or references. Cults have been able to make sarin. The DNA sequence for smallpox is pubicly available. Where are you getting your confidence that LLM-assisted bioweapons are of no concern?

Not only is the DNA sequence for smallpox available, but since 2018 there's been a well-documented end-to-end synthesis procedure for the very closely related horsepox virus [1]! No LLMs needed. Caused quite a stir in the synthetic biology community back then.

The world has not come to an end, of course, because even with peer-reviewed and experience-driven (rather than hallucinated and therefore dangerous) instructions detailing obstacles encountered during synthesis and how to overcome them, actually going out and acquiring the materials and ability to use them sufficiently skillfully is another matter entirely. Biosecurity is an important topic, to be sure, but what the AI labs have to say about it (or anything) at this point does not necessarily survive contact with reality.

[1] https://journals.plos.org/plosone/article?id=10.1371/journal...

I'm not talking about what the AI labs are saying and it. I don't know what they have been saying and I don't care. I care whether LLMs a few generations from now will be about to bring these dangerous capabilities to the masses. Right now, given the progress of open and closed-source models, it seems inevitable.

Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.

What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.

The chalenges are in execution, not availability of information.

I don't understand the significance of the HF incident. The hacking robot was told to achieve a certain goal and in order to do it, it hacked someone. The ostensible major event here is that it broke out of its sandbox, but how are we supposed to interpret that? AI often misunderstands or doesn't strictly follow the orders you give it, so why is it such a big deal that it didn't follow the rules this time?

Agree with you here. AFAIK we don't really know what cybersecurity task it was given in that sandbox, but it's not a very large leap to assume part of this task involved hacking. The intention would have been within the sandbox, but what does the AI know? As far as it could tell it just reached Level 2.

Reply on news.ycombinator.com