ChatGPT Isn't as Good at Coding as We Thought | Hacker News Reader

ChatGPT Isn't as Good at Coding as We Thought

pcmag.com

55 pointsthunderbong45 comments

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4.

It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”.

I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines.

It brings shame on not only the institution that produced it, but the journalists that failed to notice and published the clickbait headline without even reading the paper.

It seems like it's only important to bring out a paper to fill a quota somewhere. The researchers probably don't care because no one (e.g. a perspective employer) is going to be actually reading the paper, just the amount of papers they published. That's why linking to something on Arxiv trying to undeniably prove a point ("Look, researchers confirm my point!") most of the time makes me cringe.

Plus, I've mostly given up on journalists. I get clickbait, that's why titles are there for. But clickbait doesn't have to equal badly researched. They could have just as well gotten clicks if they wrote an article "Researchers fail to prove ChatGPT is bad at coding because of this lazy mistake". But they mostly either have no idea of the field they are reporting about or are pushing an agenda. Research? Ain't nobody got time for that.

I'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me.

Maybe the logic is to study the most popular product? Would make sense to me.

Many more people drink tap water than eat dinner at a MICHELIN Starred restaurant, but if you're a food critic writing about the latest fancy restaurant, you had better pay for some food and write about that.

This is especially painful considering the ludicrous pace of advancement in AI. You really have to aim for where the puck will be, not where the puck is. Within a year GPT4 will be out-of-date.

Fundamentally, the entire point of this "research paper" is to compare ChatGPT with Stack Overflow answers, coming to the conclusion that people prefer SO.

Yeah, well, meanwhile Stack Overflow usage has dropped off a cliff since ChatGPT become generally available: https://observablehq.com/@ayhanfuat/the-fall-of-stack-overfl...

I certainly prefer to ask ChatGPT basic coding questions because I get an answer immediately with no argument.

This analogy should be in a gallery, good day to you sir.

But if all that existed at the restaurant was water? The food or better drinks werent released yet.

If that were the case, I would expect to see an explicit reference to the CGPT version in the introduction, and an explanation as to why the authors chose it for the study.

Instead, the only version information appears to be buried in section 3.1.2 ("ChatGPT 3.5 Turbo API is used").

They used 3.5 because that is all that was available at the time.

It takes time to create and publish a study.

Jesus Christ, dude. To know they used 3.5, you must have read the methods section, in which case you can also see they conducted the data collection in March 2023, the same month GPT-4 was released. Bad timing, I guess, but your rant reduces to telling researchers to just scrap whatever work they did and not bother if an update happens to be released to whatever they were studying before they publish.

GPT-4 has gotten so much worse over the last few months, and GPT-3.5 has gotten better.

I still think GPT-4 has an edge but its frustratingly dumb and loses context so often that I'm not sure how much of an edge it really has any more.

I don't understand why generating language, or code, would be a good thing. Language, and code, is a tool for communication. If you treat it as fodder or merchandise, you loose the main use: transport meaning between different people.

I fail to understand why we would want automatically generated language at all, except for the fact that you can sometimes make money off of it.

Considering we're where we are as a civilization because people could make money off of poisoning the Earth and causing catastrophic ecosystem and climate change for profit, I think we should examine our motives more closely.

You can't understand why software that can do in a fraction of a section what a human would do in hours would be valuable?

The software indeed has to do what I do, which is understanding my environment and mentating solutions that are somehow appropriate for either fixing or alleviating any one of the uncountable challenges said environment faces. I cannot even describe what I am doing, only that relatively small parts of the end-result are encoded into symbols called code.

This reminds me of what Bohr said to Oppenheimer - in the movie - when some doubt arose around his mathematical ability: "Algebra is like sheet music. The important thing isn't 'can you read music', it's 'can you hear it'. Can you hear the music, Robert?"

I'm very much wondering if GPT can hear the music, but time will tell if and to what degree this even matters.

But the real question is whether or not an LLM can let someone who can hear the music (they’re an expert in the context, the project, and the business environment) but who can’t read or write the sheet music (can’t code) will be able to make the computer do what it needs to do well enough using the LLM as a sort of person to machine translator to eliminate the need for a person to listen to the subject matter expert and write the code.

Well, in that example the musician makes music and a machine could transcribe it to sheet music. I can see that working and maybe it already does.

What would be the equivalent of “making music” in this case? Talking haphazardly about your business?

I can see this working if something like “making music” is possible without knowledge of said music. Which in actual music is quite possible because it is an intuitive art. Programming.. less so, but I never say never.

ChrisRR--

I would counter this with:

- What value is it to your company to know that the developers who churn into and out of your company... while they're there, they actually know a section of your codebase quite well?

versus the alternative: the degree to which your devs know where your code lives or what it does is low, since they do not engage with it enough.

Sure, one could say: "AI will navigate our codebase for them".

But I think it becomes a bit of a slippery-slope, regarding the question of: "When/to what degree can really we take the developer out of the picture?"

Imagine being a developer, and no one on your team really knows how your product works.

Is AI building your product at that point then? Is that product actually going to exist & be funcitonal?

You're describing automation. I'm talking about a very specific subset of it: automatically generating human or (human readable) language.

So if the job is to generate articles, technical reports, presentations, emails, marketing materials are the task, then generating human-like text is automating that task.

Would you be so kind as to demonstrate how generated language and code does not transport meaning between different people.

I'd say the symbols themselves are meaningless without their ground truths, whatever that is. We can call this "experience" or "thinking". The results of these opaque processes are somehow amenable to be encoded into symbols which are themselves deprived of any significance. Which I guess makes them usable as such in the first place.

Extreme example: if a random algorithm picked out a few words from a dictionary and somehow cobbled a sentence together using those words, would you say "meaning" has been transported?

The "generation" of these symbols should be the result of a process equivalent to whatever we are doing when we "experience" or "cognate" or whatever, otherwise the results will be only very superficially useful.

> I'd say the symbols themselves are meaningless without their ground truths, whatever that is. We can call this "experience" or "thinking".

So you're basically dismissing the whole field of genetic algorithms?

If said genetic algorithms were used to directly spew forth symbols that represent or point towards human experience (“language”) as if the underlying processes that ultimately form the foundation of said symbols could be approximated by slowly changing random walks, then, yes, I’d question the usefulness of that approach.

If not, what do you mean?

I mean, just because an array of symbols is generated mechanically doesn't inherently make it meaningless, because meaning arises from the execution of those symbols (in case of computer programs; or interpretation, in case of human language).

If you read a sentence (whose source is unknown to you) and it has meaning to you and affects you, what difference does it make if the sentence was written by a human or a machine?

That’s an excellent observation.

I struggled with this too and because I am fond of defective analogies this made me think of the meaning of a personal note to, say, a lover.

Does it matter by whom or indeed what it was written? The experience of reading occurs solely in the recipient’s mind.

Surely it does not affect him/her differently if said note was produced by a romantic slot machine.

Before I meander towards my confused and shaky conclusion, let me clear one thing out of the way: I did not mean to imply that mechanically generated sequences are meaningless.

They absolutely carry meaning, because meaning is ultimately created by us. We are meaning-creation machines. We can find meaning in tea-leaves. Surely tea-leaves carry meaning, but sadly not the will-of-the-gods-kind. Although that doesn’t stop us from thinking they do.

What I am tentatively suggesting is that the meaning of the lover’s note is as much found in the experience of the relationship they share as it is found in the symbols on the paper. The symbols reference that experience.

Because it is ultimately a human that reads and interprets I think failure to take into account the origin of the material will be the source of a subtle and highly impactful, but inexorable error not dissimilar to that of the tasseographer: what you see doesn’t mean what you think it does.

Language used to be a domain dominated by humans only. We are wired for it and biased by it. If it sounds intelligent, it is intelligent, right? This bias could make us defer to the machine quicker than we should.

What’s the difference between GPT4 generating a good piece of code and a human?

Maybe I should shut up now. I am open to interesting reading suggestions by the way.

The difference is in the temporal dimension of the message. Generated text and code are simply statistically likely to convey enough meaning for the purpose at hand. That's all good and dandy until you think about the repercussions of communication. A person facing a problem and finding a solution encodes a message. This message ideally conveys context and purpose. The situation changes over time and we need to adapt our systems and behaviors. So we can go back to the message and re-work it.

If there is no inherent message, if it is just a byte array likely to get a desired result in a specific context, you are losing traction of mind over reality and letting an awful lot of noise into our delicate systems.

Isn't letting accidental complexity aka noise into our systems bad? If you don't know the purpose and the context of the message, you already let that happen, and it will only get worse as the subsequent modifications to the systems described by the code or behaviors enacted due to the message starts to become an opaque box due to the fact that there is no meaningful blueprint behind the code or the texts underlying it.

Yes, the repercussions of this vacuousness are what interest me most. The underlying “systems” and “behaviors” that gave birth to the message are prior to any language model, but there, pre-language, is exactly where all changes happen. These changes then percolate to the top, so to speak, as language. I believe you captured this beautifully.

Then you get a bunch of humans that interpret this soulless message, but fail to comprehend the inherent emptiness sufficiently.

I’d say that’s not good, but I have no solution either.

I see very little difference between letting LLMs create language that people and computer systems are expected to act upon and using oracles/'revealed' wisdom/sacred texts which dictate what we ought to want, decide and do. Letting inscrutable systems write the rules is just bad. We have 300 years of humanism building a base for our shared reasoning skills to enlighten the way and giving that up for the convenience of letting noise and chance guide us seems like going backwards.

The purpose of code is first and foremost to make a computer do things. Legibility is, strictly speaking, secondary.

I see you've never had to maintain code.

Precisely my point. If you see code, or written language, just as something you dump in a box to get paid, creating code or text with LLMs makes perfect sense.

If you realize code and language are ways to communicate asynchronously and conserve information across time and space to enable developing and maintaining systems, the apparent value of generic concatenation of words that looks good and functions but has not meaningful message conveyed drops off precipitously.

Code that doesn't run is just dead bytes. It's not useful for anything. Code that does run, even if impossible to comprehend without considerable research (like, say, DNA), is useful for something. You just cannot change its function.

I have maintained code, and even wrote some terrible stuff myself. All I'm saying is that once you strip all the lofty ideas about what code should be (which are all very good ideas!), then you are left with the very simple core idea: code is supposed to run and do stuff.

52% correct seems awfully high - I am guessing it's inflated because of the easiness of the questions on StackOverflow. In my experience, at least in my domain, I will be lucky it it can generate an answer that is more than 5 lines of code without a single error. Still incredibly useful - just have to handhold it and watch it very carefully when going through a solution.

And it does not need to be. Personally i do not use it to generate novel code that will solve my problems, but when i point it at a github repo with the github plugin enabled, it will answer the questions i have about the code pretty much on the nose.

its not your new AI Friend steve, its a LLM. And one with a knowledge cutoff in 2021 too

> but when i point it at a github repo with the github plugin enabled, it will answer the questions i have about the code pretty much on the nose.

Holy crap, you can do that? I need to learn about ChatGPT plugins.

yes its in the "plugin store" and named "ask the code"

Note that at time of writing the “ask the code” plug-in does well with describing repo structure, but hallucinates file contents (I tested it on one of my own repos)

maybe i have random successes or my questions are simpler than yours, but it works for me:

https://chat.openai.com/share/fbd14b6c-0ba0-473f-9e94-db97c6...

Lately it’s even worse. And any AI agent that relies on open AI has gone down in quality too (phind). I find it’s more of a struggle to have the AI write code in the first pass, even when I explicitly say something like “I want you to write a function that does this {action}” and on average it takes about 3 messages back and forth before it actually writes it. Whereas before it would start writing it immediately, even though it was almost always wrong, I could tweak the result after. Now though, the stupid context memory is filled up with useless junk words not related to the actual function I want to write, since we spent multiple messages going back and forth.

A tip for your context window problem: prefer ‘editing’ a previous message to add whatever clarification is required - rather than having a ‘chat.’ Definitely helps keep it on the rails.

Fair. But garbage in garbage out. The prompts are in human language. Of which the models use to derive output. Human language is obviously not a rigid mode of instruction. Therefore; the models behave exactly as you'd guess. The output is not purely predictable nor rigid. This is by design.

Providing a prompt, and/or know how to properly prompt is key.

LLM are autocomplete with context, I don't know what they were expecting

This article shines a light on exactly why LLMs /arent/ just autocomplete with context. There’s real computation going on in there. https://arstechnica.com/science/2023/07/a-jargon-free-explan...

GPT4 can run code, so you can ask it to run the code it generated and iterate until it works. Does that make it one shot? Either way, its limitations are becoming apparent, but it's still quite the leap forwards for writing code.

I don’t think it runs code, it’s an illusion and it’s actually inferring what the output would be but not actually executing any statements

Reply on news.ycombinator.com