ChatGPT's odds of getting code questions correct are worse than a coin flip | Hacker News Reader

ChatGPT's odds of getting code questions correct are worse than a coin flip

theregister.com

34 pointsmikece21 comments

From the paper [1]:

> For each of the 517 SO questions, the first two authors manually used the SO question's title, body, and tags to form one question prompt and fed that to the Chat Interface of ChatGPT. The generated answers by ChatGPT are then stored in CSV files. For the additional 2000 SO questions, ChatGPT 3.5 Turbo API is used.

I would be interested in seeing the results for GPT-4. For a paper released in August 2023, it’s strange that GPT-4 isn’t even mentioned.

[1] https://arxiv.org/abs/2308.02312

>> it’s strange that GPT-4 isn’t even mentioned

Except for when you look at the results. ;)

Isn't this a bit silly since chat gpt _was trained_ on SO questions, or are these all questions asked since the training cutoff...

If you told me in 2005 there would be something non-human that can get a coding question right with near-coin-flip accuracy I'd say you're crazy.

Wouldn’t a coin be able to do that pretty well in 2005?

I would expect a coin to do it 0% of the time.

Unless the question was "write a program that simulates a coin flip"

True/false answers to programming questions don't build apps.

Only if it was choosing between one of two answers to begin with. The analogy is a bit weird here because there's far more than two potential answers it could give but it still manages to pick the right one almost half the time.

The free version of ChatGPT, which is now calling itself 3.5, has a seemingly consistent type of answer whenever I ask it how to code something I don't know how to do.

It outputs some boilerplate code, and inserts a method with comments saying "insert logic here" - entirely sidestepping the request.

You can say that's remarkably human-like, but if it was a human cow-orker, you would not keep them on the payroll. It is a cheat, like ELIZA, like SHRDLU, like a useless employee.

I asked ChatGPT-3.5 for the usual rejoinder to complainers:

"If you're looking for an enhanced experience, you might want to consider trying out the paid version called ChatGPT-4. It offers even more capabilities and improvements. Feel free to explore the options available to find the best fit for your needs!""

Two thoughts: (1) ChatGPT answers questions immediately (I'm paying for premium service so I don't get shut out), which is a big win over Stack Overflow where I recently waited for two or three days before getting a partially-correct answer. (2) I often am using ChatGPT (in my Ruby world) to find gems (ruby packages) or language features that address my issue. Even if the whole answer is wrong, it often provides me with very helpful tidbits of information at a huge time saving.

On more than one occasion, the hallucinated solution has had better dev ux than the real solution and I've convinced whatever package to add support for a similar syntax.

I think 30% is still impressive. Imagine if we treated search engines this way.. “30% of the time someone googled a software problem and clicked the first link it was the wrong answer”. I don’t think anyone would be shocked by such a groundbreaking finding.

Well...as it happens, I have a recent example of Google getting something trivial wrong and ChatGPT getting it right.

But oddly that doesn't change my opinion of ChatGPT.

The only reason I know that ChatGPT was right is because I blindly followed the first Googled suggestion.

If I started with ChatGPT, then I would have to verify the answer somehow. If I use Google, I should verify the answer. But ChatGPT cannot do that job.

Wikipedia is starting to feel as useless to me as ChatGPT. I don't know if anyone is actually replacing the articles with AI generated fiction at scale yet, or whether it's still an artisan, human-centered craft but the more I click on references, the more I find they are uncheckable, even if it's ambiguous whether they are fraudulent. People write rants on a topic and cite a single book without page numbers five times. Or, of course, dead links to websites without archive.org.

This has definitely not been my experience, nor my colleagues’ experiences.

It also says that programmer essentially prefer wrong answers.

I prefer easy to fix answers

ChatGPT has helped me install plugins on my Minecraft server and given me configuring commands with near 100% accuracy, and the only mistake it made was using an old command that is one letter different now. This was ChatGPT-4, running the Code Interpreter.

It really helps me get a better idea of what I should be doing. Like any answer from anyone, you have to understand and peer review. I usually end up rewriting ~50% of anything I get out of chatgpt.

I can remember a few time when a computer program getting 48% of coding questions right was science fiction.

This is really more about social factors anyways… what’s the study actually trying to hypothesize and test?

Reply on news.ycombinator.com