Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.