Safety standards were often the outcome of both accidents and insurance. For this to work on needs liability which is enforced. With so much money at stake regulatory capture now threatens this fundamental safeguard.
An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.
But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.