He said something to the effect of "that's ridiculous - I would simply not let it out of the box."
We agreed to try it out some day, but never did.
We can only hope to either never create an AI so strong or to align it correctly. But if it is not aligned and only “contained” then it won't ever be safe.
The real question thus moves to the threshold of intelligence and 1. whether it's possible to emerge during training based on the architectural limitations of the agentic/LLM paradigm, 2. if the hardware substrate is sufficient for said intelligence and 3. that such intelligence could replicate onto other hardware that could support it.
e.g. If the threshold for uncontainable self-replicating intelligence takes 2000 football fields worth of GPUs that solves the first requirement, but then can it replicate itself anywhere else given those requirements? If not we can cut a powerline or two and "foom" scenario happened but didn't lead inexorably to grey goo.
His thought experiments never acknowledge any real world limitations on hypothetical super-AIs, which when unchecked leads theorizing into somewhat ridiculous territory like his "solar powered diamondoid nanobot viruses".
https://www.lesswrong.com/posts/bc8Ssx5ys6zqu3eq9/diamondoid...
A realistic Fermi equation for his various escape scenarios would assign much lower Doom probabilities than he does in public (which is somewhat ironic given his emphasis on needing to ground intuition with mathematical Bayesian reasoning otherwise).