Digression, but this is the real Great Filter imo, not AI. I think technology advances to a point where it only takes one or two bad actors to type the right prompt to get a recipe for civilization-destroying bioweapons before you get anywhere near true AGI or anything relevant to the Kardashev scale. Biology is fragile.

But not that that’s a good justification for hamstrung models. I think it’s just the inevitable endgame and it’s more sad than scary