Sure, the chance of this happening is non-zero, but in my opinion a far cry from this doom scenarios that are propagated by people have a stake in making the public think that LLMs are the most important and dangerous thing in the world.

It's not like an LLM can accidentally hack the US government, trigger a nuke on Europe and then make all dams break and nuclear power plants explode in the US.