tbf most companies don't have a potentially world-ending product. only real similar field is defense contractors who typically can't brag about unreleased ideas as they're classified.
This technology interacts socially, so even if it can't jailbreak itself on a technology-level (which feels like a tough guarantee to make at this point) it can simply ask someone to do a bad thing and there is some chance they'll do it. The same way a human leader does.
The first kids who have only faint memories of a time before chatbots will be entering the military in 6-7 years. You have to assume they are acting as best friends, therapists, or even surrogate parents for a substantial number of kids right now.
We are going to need years to figure out what to do about this technology. I think some impetus to get that process started is a good idea.