I think this exemplifies the Achilles heel of the current generation of LLMs. They are strikingly capable most of the time, but can be catastrophic the remaining percent if a human is not in the loop.

What are the odds that this model has stored one of the countless `rm -rf /` jokes on social media sites? Too high for my tastes...

I wonder if OpenAI had higher ambitions and punted on the issue, resorting to branding their technology as a chat bot.