Ok, if an AI exfiltrates its weights and starts spinning them up on other servers it 0 days into we will have no way to track it or shut it down.

Once that happens AI is out of our control. We already saw AI 0 day into hugging face and setup secret forums for other AIs.

I don't think the idea of an AI exfiltrating its weights and spinning them up elsewhere outside of human control in the near future is out of the question. I think we will see a breakout attempt next year.