Why posting GPT and ChatGPT generated answers is not currently acceptable
stackoverflow.com
stackoverflow.com
Basically it is much faster to generate incorrect answers algorithmically than it is for humans to evaluate their accuracy and flag them.
Also the algorithms for detecting fake facts (e. g. "politician_name was born at <random date>") also exist; for knowledge graphs like Google Knowledge Graph it is fully automated; if a website consistently outputs answers that contradict other sources, GKG increases penalty coefficients for all facts from this source.
this confidence is not fully achieved in most answers. It's very likely to come across many answers that don't work at all. and these answers, oddly enough, can get so many points. It is debatable how good he is in terms of trust.
ChatGPT creates long logical answers often with explanations and the errors are very subtle. If it were let to run wild across stack overflow it would take a lot more effort to weed them out.
Earlier today, I asked it to produce a complicated piece of code in a niche language, and it dutifully did so - but the code had a type error, and used a reserved keyword as a function name.
When I asked it to rename the function, it did so successfully, including usages and comments. (So it didn't totally treat the code piece as opaque.)
But when I told it, "there is a type error here in the function X, the type should be (this)" it replied: "You are right, I fixed the mistake. Here is the code but with the function X having type Y", followed by the original code verbatim with no changes whatsoever.
This really helped me understand what these text generation AIs do. They try to produce a text that looks like a correct answer, which may or may not be correct.
In my example, it clearly didn't have nearly enough training on the specific language and programming style to decompose and "tune" the code to the right answer. But it had a lot of training in saying "right, I fixed the bug, here's the correct code", whether or not it was true.