One of the things that worries me about ChatGPT-driven products is the issue of confidence. We’ve seen in other posts where it can generate convincing-sounding instructions from thin air. How can I trust that following the instructions for “How to set cookie in FastAPI?” is going to be legit?

We obviously also have this issue with the open web, but real authors are almost universally doing it for others’ benefit (as opposed to their detriment/trolling). ChatGPT generating instructions for processes isn’t “real” and so it has no notion of them being “correct” or “incorrect”; they just “look” the same as the training data.