Comment by nottorp | Hacker News Reader

Comment by nottorp | Hacker News Reader

Well, i've started to use ChatGPT instead of google when looking for quickie examples for something. Mainly because of how bad google has become.

It works fine for stuff like "give me a tutorial on how to initialize $THING and talk to it" or "how do i set $SPECIFIC_PARAMETER for $THING up".

Where it seems to fail is when you ask "how do i set $X" and the answer is "you can't set $X from code". I got some pretty hallucinations there. At least from the free ChatGPT.

So maybe add a trick question where the answer is "it can't be done"? If you get hallucinations back, it should be clear what is up.

Edit: not that I'm a fan of leetcode interviews. But then to get a government job in medieval China you had to be able to write essays based on Confucius. Seems similar to me.

The problem is its difficult to track what ChatGPT can and can't do. One day it'll give you junk and then an update or different prompt might fix that problem.

That's why you look at what it says with a critical eye.

I think of ChatGPT like a pretty smart co-worker. Just because they are smart doesn't mean they are always right.

I'm using it mostly instead of API documentation. And for stuff I already have an idea of.

Don't trust it further than that.

There's times when there's no good docs.... ChatGPT can give you a spot to start from, even if it is wrong.

Trusting it blindly is stupid. But, so is trusting any sources without verification.

Indeed. Most developers with enough experience have probably encountered documentation which was wrong or incorrect implementations against a spec which amounts to basically the same thing. A discrepancy between the documented reality and the actual reality.

Clearly, yes.

What I mean in relation to the post is that it's hard to even make a question that ChatGPT will fail destructively without doing a lot of research in advance. In many cases, even if it fails, it'll have a reasonable mistake that a normal human might make -- and a normal interviewer doesn't have the time to fine tune, craft, and research how to mess up an AI

There's also a lot of junk answers on Stack Overflow.

There’s your problem. GPT4 is an order of magnitude better than the free version, there’s no comparison.

Reply on news.ycombinator.com