In the early days I worked at a company that had a natural language chatbot product. It wasn't an online thing, but rather part of a larger tool. You could ask it to do things like "show me the quarterly spreadsheet", and if it didn't know what "quarterly spreadsheet" was, it'd ask questions in English and learn what "quarterly" and "spreadsheet" meant. And it could use that new knowledge to update its questions so if you asked for "spreadsheet", it would know to ask if you wanted the "quarterly" one, and so on.

This was in the long ago, so it wasn't ML, but a very large state machine written by someone that was no doubt a genius, and also had no coding standards to speak of. Updating it was a maddening process, and thankfully one I rarely had to think about since I was tasked with GUI stuff.

When I was hired, I was told it was a fantastic app, customers loved it, and so on. Later, when I was talking to support reps one day over lunch, I learned that the number one activity they had to do was log into deployments remotely and remove "knowledge" from the state machine of new swear words the customer had taught the machine out of frustration trying to get it to do something, which caused this feedback loop of the interaction swearing at users.

I often feel like we're not in a better state these days.