It's quickly apparent that the people impressed with ChatGPT's code are generally solving pretty toy problems and want an alternative to SO for "how to do X".
So, you say, "Write me a heap for Java", and it spits out amazing, wonderful code that is almost right. Well, yeah, that's cool and all, though I would point out that its training data probably included a heap that was entirely correct, but, still, it's a common question.
But I'm not writing a heap for Java. I'm taking business object X and stuffing through system Y and feeding the result to system Z, all internal to the company I work for. Good luck with getting ChatGPT to do that.
But it deceives people, because the very questions they are most likely to ask as a test are the exact questions it knows the answer to, for the very reason that they are also the most likely questions to be in the training data.
(This article kind of doubles as great set of examples of just how quickly you get into confident nonsense if you stick a toe slightly out of line in your query. For instance, even if it knows "What is Y" and "What is Z", it can't necessarily combine them in any correct way. It'll spew out a lot of words that sound very confident, but it'll make mistakes without any indication they were mistakes.)
Exactly. Nearly every example I've seen is something that I could easily find an answer for on SO for anyways. ChatGPT customizes it as needed which is nice (though it often makes glaring and not so glaring mistakes).
ChatGPT will try to respond with exactly what you want, but with (multiple) errors that must be fixed.
I'll take the working example and adapt it myself.
There was a blog post on SO about this recently, they are banning accounts posting ChatGPT answers because they appear reasonable at first glance, but try and execute the code and it has syntax errors or is just plain wrong.
!remindme 10 years
I don’t dispute that it’s super cool technology made by some equally cool people, and honestly I would be perfectly fine with it putting me out of the job if it gets to that point.
This is a specific point about people being deceived by the very questions they ask, not a generalized comment about the AI.
I don't know why so many people are unimpressed with it because it isn't some kind of all knowing hyperintelligent god. This guy is literally unimpressed with it because it can't solve Fermat's last theorem! Wtf?
It’s impressive tech, but giving a false-but-believable proof is substantially worse than saying “I don’t know how to do that”. If it had come back with “I don’t know how to give a proof of Fermat’s Last Theorem in Isabelle”, I wouldn’t have been unimpressed; there are plenty of very cool things it can do. I just think that it being confidently incorrect is a bad thing.
EDIT:
I should point out, it also was wrong with substantially simpler proofs, like the proof of infinite primes.
If we could automate away engineers that’s fine with me, but i just don’t think that ChatGPT is there yet. I actually think Wolfram is kind of onto something with having ChatGPT work with Alpha to be used for stuff that has “objective” answers.
The thing is, the hiring managers reacting like that to ChatGPT are also the same folks who would've considered outsourcing engineering anyways to one of those offshore "consultancies". Tools like ChatGPT will ultimately be a replacement for use cases where you've already fully scoped out what to build and how to build it and the implementation is straightforward.