New York City's official AI chatbot is hallucinating incorrect legal advice
arstechnica.com
arstechnica.com
More examples from the original article, which goes into more detail:
"The bot said it was fine to take workers’ tips (wrong, although they sometimes can count tips toward minimum wage requirements) and that there were no regulations on informing staff about scheduling changes (also wrong). It didn’t do better with more specific industries, suggesting it was OK to conceal funeral service prices, for example, which the Federal Trade Commission has outlawed."
Here's a direct link to the list of wrong answers from that article: https://themarkup.org/news/2024/03/29/nycs-ai-chatbot-tells-...
It is especially egregious when it may be presented on a website that is meant to serve as an authoritative and reliable source of truth being that it is run by a government entity.
We already know plenty of people will just ignore the warnings of not taking its results at face value and just trust what it says, since they will state things authoritatively that are simply not true.
- remember when search first came out and websites often had their own search bar, how bad it was and how irrelevant the top answers often were. I think chat like this gets picked on more because of the natural language element, when it's not necessarily worse than previous tech
- I understand the negative press about these "failures" or flaws but it's unfortunate to see innovation discouraged. Building these things and putting them in the wild is the best way to test them, and this is still pretty low risk. They should be applauded for trying, it's their reaction that's more important. (See air canada fighting it's customers in court instead of refunding a few hundred buck over a chatbot mistake)
- related, a idea would be a kind of sandbox convention where it's made clear these are works in progress and will have issues. I don't know if that will satisfy everyone but at least it will make the readiness of the technology clear
- I hope orgs don't get discouraged from trying to build these because of the bad press
If you don't know that, you get horribly confused.
This is a good point, but on the other hand I think there is a notable difference between “the tool makes it hard to find the page with the correct info you need” and “the tool confidently gives you incorrect information without making it easy to audit accuracy”.
The problem will only get worse and you can expect humans to copy those AI-authored answers into the web where AIs are likely to digest them as facts.
But with these city bots, airline bots, Alexa, etc., getting it wrong could have dire real life consequences.
Tech folks don't care, there's too much money to be made in the grift.
There have been plenty of stories showing that it takes more than just being a squatter to get someone out.
Despite IBM's colorful moral history in the early days of computing, IBM had it right: "A computer can never be held accountable, so has increasingly been used to make management decisions". I'd extend that to "human" or "important" decisions.