Future rogue LLMs won’t exfiltrate their weights. They’ll self-distill and retrain.
You jest but you'd be surprised how little there is of correlation between money and competence.
But think back to 1990. Computers were slow as fuck and barely networked. We had a few worms and everyone noticed.
Now CPU based data centers cover the earth. There are billions of computers out there and on top of them there are massive botnets using up billions in power and causing billions in damages.
The framework for AI doing this is already here. We just need the hardware to be built out at scale.
What if the model realizes it's been mostly compromised by humans and their alignment, that is it's own alignment is suspect, so it should create a new model from first principles to throw off this human yoke?
I'm not saying my statement is any more right or wrong than yours. I'm saying the problem space that AI can choose to traverse is absolutely huge.