> it’s impossible for AI creators to definitively say their AI will never act maliciously or dangerously for any prompt it could be given

This is false, AI doesn't "act" at all unless you, the developer, use it for actions. In which case it is you, the developer, taking the action.

Anthropomorphizing AI with terms like "malicious" when they can literally be implemented with a spreadsheet—first-order functional programming—and the world's dumbest while-loop to append the next token and restart the computation—should be enough to tell you there's nothing going on here beyond next token prediction.

Saying an LLM can be "malicious" is not even wrong, it's just nonsense.