They know what they are doing. They provide something that sounds over-confident for anything it says, knowing full well that it can't actually know if what it generated is accurate because it is designed to generate plausible sentences using statistics and probabilities, not verified facts from a database. On top of it, they trained it on an uncontrolled set of texts (though IIUC even a set of verified text would not be enough, nothing guarantees that a LM would produce correct answers). And they provide it to the general population, which doesn't always understand very well how it works and, above all, its limitations. Including developers. Few people actually understand this technology, including myself.
Inevitably, it was going to end up causing issues.
This post factually presents a problematic situation for the authors of this post. How ChatGPT works or how it can end up producing wrong results is irrelevant to the post's authors problem. It just does, and it causes troubles because of the way OpenAI decided to handle things.
And it's not "fair enough, because this false stuff can be found on the internet".
The key thing is that it isn't a knowledge base. It doesn't claim to have correct information about the world. It has the ability to translate a question in natural language into what would be an answer in a natural language - but it isn't necessarily correct because its about the language rather than the knowledge.
People misusing the LLM as a knowledge base are at fault just as a person misusing a CD tray as a cup holder is at fault if it doesn't work correctly as a cup holder.
… are doing this because it's an intuitive thing to do and because they don't understand that
> The key thing is that it isn't a knowledge base. It doesn't claim to have correct information about the world
and that it's
> because its about the language rather than the knowledge.
or what are the implications of all these things.
And:
> is surprisingly correct in some situations as a knowledge base.
This is key. It seems to work well for this use case so people end up thinking it does.