That's specifically why OpenAI don't release weights, and why everyone who cares about safety talks about laws, and why Yud says the laws only matter if you're willing to enforce them internationally via air strikes.
> It’s going to be so satisfying
I won't be feeling Schadenfreude if a low budget group or individual takes an open weights model, does a white-box analysis to determine what it knows and to overcome any RLFH, in order to force it to work as an assistant helping walk them though the steps to make VX nerve agent.
Given how old VX is, it's fairly likely all the info is on the public internet already, but even just LLMs-as-a-better-search / knowledge synthesis from disparate sources, that makes a difference, especially for domain specific "common sense": You don't need to know what to ask for, you can ask a model to ask itself a better question first.
As you said the information is already out there - getting info on how to do this stuff is not the barrier you think it is.
If you think it's "laughable", what do you think tools are for? Every tool makes some difference, that's why they get used.
The better models are already at the level of a (free) everything-intern, and it's very easy to use them for high-level control of robotics.
> getting info on how to do this stuff is not the barrier you think it is.
Knowing what question you need to ask in order to not kill oneself in the process, however, is.
Secondary school chemistry lessons taught me two distinct ways to make chlorine using only things found in a normal kitchen; but the were taught in the context "don't do X or Y, that makes chlorine", not "here's some PPE, let's get to work".
Wonder if it's true?