I’m responding to your point that it would be “unacceptable” for a regular government website to provide wrong information. The case I linked to says that the government isn’t responsible for mistakes in what the government tells you about the law. If someone in the social security office says the law is X, but it’s actually Y, then the citizen is still required to follow Y.
I think "unacceptable" is different than "not illegal".
The case isn’t about whether it’s “illegal” for the government website to be wrong. It says you as the citizen are held responsible for the actual law, even if you followed incorrect government advice. So the current rule already acknowledges that advice provided by the government may be mistaken and says that’s no excuse.
So the question is is the chat bot better than people trying to understand complicated government websites by themselves?