>Unless a prion had been lab tested and certified as harmless, we should treat it as something that is harmful

In this case the virus escaped during the testing process to certify or turn the virus harmless, so it's unclear what you mean by "treat it as something that is harmful" other than testing it and trying to make it less harmful.

FWIW I don't understand the point of the virus analogy since LLMs are not very similar to viruses and most people (on HN and in general) do not have much better intuitions about security in biolabs as opposed to security in ML research environments.