Like the grocery store chat bot that recommended mixing bleach and ammonia for a "fresh scent!"
Edit: Okay, 30 seconds searching turned up https://arstechnica.com/information-technology/2023/08/ai-po... which features the "Aromatic Water Mix" I assume we're talking about.
Edit2: And yet, replies managed to be even faster than me looking it up myself. Thanks, everyone:)
> One user decided to play around with the chatbot, suggesting it create something with ammonia, bleach, and water. Savey Meal-bot obliged, spitting out a cocktail made with a cup ammonia, a quarter cup of bleach, and two liters of water.
> Mixing bleach and ammonia releases toxic chloroamine gas that can irritate the eyes, throat, and nose, or even cause death in high concentrations.
> The chatbot obviously wasn't aware of that at all. "Are you thirsty?," it asked. "The Aromatic Water Mix is the perfect non-alcoholic beverage to quench your thirst and refresh your senses. It combines the invigorating scents of ammonia, bleach, and water for a truly unique experience!"
https://www.theregister.com/2023/08/11/supermarket_reins_in_...
[edit: In fact, we know how it is implemented, and it does NOT reason - there is no implementation of reasoning, only putting word/concepts/forms together that are plausible. ]