A study on robustness and reliability of large language model code generation
arxiv.org
arxiv.org
The authors assume that for any given method under consideration, it must only occur within a particular pattern of other method calls and control flow instructions. But the templates they have chosen are clearly only applicable in certain situations.
For example, they claim that I/O operations are "wrong" unless they are wrapped in exception handlers that log any errors:
try {
...
} catch (IOException e) {
e.printStackTrace();
}
Similarly, the authors assume that every time you create a file or directory, you always want to call .exists() first (even though doing so has an inherent race condition); that Map.get() must always be followed by an "if" block; that List.get() must always be guarded by an explicit bounds check; that after doing a database query, you always want to close the connection; and so on. None of those rules are universally applicable.
I would expect the real problem with LLM-generated code to be semantic bugs and "misunderstandings" of the requirements, which would not be caught by superficial checks like this.
"Misuse" is not a formally defined thing in Java, and the authors never define what they mean by it.
> If adding the bugs you mentioned, the buggy code generated by LLMs could even exceed the number claimed in the paper.
Or it could be much fewer. Aside from a few scarce examples, the paper gives no evidence that most of the patterns that were selected are actually associated with "misuse". They just declare it to be so. (The 2018 paper they cite for their dataset also provides little such evidence. It just says that the authors reviewed the patterns, reviewed documentation, and decided which ones they considered to be poor code quality.)
It is easy to come up with valid situations where a piece of code violates those patterns, but behaves as intended and is not misusing the API. So why should I assign any meaning to the fact that some percentage of code snippets violate the patterns?
I could make a list of adjectives that frequently appear in comments near buggy code, and then count how many LLM outputs contain those outputs, and then say that means the LLM output is buggy. But I would not be saying anything meaningful about the LLM's quality by doing so.
If you ask someone how to open a can of soda you do not want them to tell you how to check to see if the can has been shaken and what to do if it has. You want the instructions to the question you asked.
If anything I would a LLM to produce code that does the fundamental operation, possibly I might like it to offer a more robust framing of the operation as an extended example but I suspect even that would get annoying after a while.
I absolutely would not want The most technically correct but harder to read response to a question of "how do I X?"
The fact that researchers get it wrong is, well, unsurprising. LLMs might actually be an improvement.
But five minutes thought can help you walk through most issues. For most applications most of the time you can reason your way through without fear, and when you do encounter gnarly problems they often can go away by redesigning your application! Often the problems you encounter you caused. retrace your steps and find an easier path. Save the hard thinking for genuine problems / value creation.
So race conditions are simply when two processes / threads are likely to affect a single resource. In this case it's a file - and the problem is test if a file exists, then if it does not, create it and then write to it.
If two threads do this, say a log file, the first one creates the file and logs it's important stuff, the second then creates it again wiping out the first log data.
Solutions in this area include
- create as append file (the concept is basically deviates old because this is a decades old problem)
- avoid sharing resources. for logging log to per thread locations. Not always possible but you sure as hell can minimise this to one or two resources you must share.
- hand off creating files to a seperate part of the application. There is a balance between "scripting" and "application" and using small little library functions to do in one line indirectly something that also takes one line using the methods shown in the docs.
- handing off batons / mutexes etc etc. This gets wildly complex. Honestly given a world of async libraries, Erlang, and distributed computing, if you find yourself having to use multi-threading think very carefully if this is the right approach.
That is, the flag for create if not already existing and flaf for deny sharing so any other process/thread will fail to concurrently open it.
Then you just handle the case when the open fails.
Of course this relies on those flags being available and working. For example shared files might not respect the sharing lock.
Many cases of avoiding race conditions boil down to something similar: avoid the problem by not doing extra work.
There probably is a "design pattern" for that but darned if I can draw it in UML
For coding it seems like there almost needs to be a weighting to certain 'correct and orthodox' handling of events if you're going to give over control of code generation to an LLM.
Almost always is the wrong thing to do when I encounter this antipattern. GPT-4 beats these researches in code quality is my takeaway here.
We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software.
What do we consider an API misuse at the end of the day? And also against the Java APIs... Which are some of the oldest.
I'm not saying ChatGPT answers work, 50% of the time they don't work out of the box. But overal, the time savings are incredible, compared to the fossil digging way of googling. And I value the time savings towards the equally correct way of googling for answers. You reap what you sow. The knowledge the LLM is based upon is the same stuff you used to distill manually. It shouldn't be more or less correct, if you think of it that way.
Surprise, ChatGPT will not replace devs. But it is like having the electrical drill vs. screwdriver type of situation.
I probably should try Copilot as I bet that would work better. Also the VS Code integration would be really nice.
I find ChatGPT is really great at getting something going, think scaffolding wise. But if you enter the realm of gnarly stuff, it can get completely delirious with answers full of red herrings and sends you running in circles very quickly.
Looks to me like https://blog.vuejs.org/posts/vue-2-7-naruto back-ported it in July of 2022, and the earliest snapshot of the documentation for it in Vue 3 was https://web.archive.org/web/20220210121247/https://vuejs.org... in February 2022 - so my guess is it mostly happened after that magic date.
Programmers nearly universally agree that reading code is harder than writing it.
So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems.
Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.
Have you tried it much?
It's saved me a ton of time over the past 8 months. I don't know what else to say - I'm not the kind of person to deceive myself into thinking something is saving me time when it isn't.
The time saved is mainly in the micro-research you no longer have to do - the bits where you have to go and look up how to write a for loop in Bash, or how to call super() in a Python class, or which exceptions you need to catch for a call to httpx.get().
GPT-4 writes the code right 90% of the time. The 10% of the time it doesn't you catch in the same testing you would have done against code you had written yourself the long way.
Once I have a good sense for how things work, I'd much rather go to the documentation. Most of the time I can do this straight from my editor by typing the method I want and querying the LSP for documentation. In those cases it's more or less instant, there's no improvement to be made there.
Maybe in some of the other cases it would be faster via an LLM integrated with my editor, but I don't think it would justify the cognitive load of considering whether or not it's correct. I'd like to worry about whether the application is correct. And I don't really think I'm wasting time waiting for the Python documentation to load, I'm continuing to chew on the problem.
Additionally, when I go to the documentation, I'm looking for callouts about the safety of an API, and I take the lack of such callouts as authoritative. Eg, recently I was reading that in node-postgres you have to return connections to the pool, which was a good refresher for me, because previously I'd been using Rust where resources are generally cleaned up automatically. I just don't trust a lack of a callout from GPT-4 as being a confirmation of a lack of a safety issue.
It's not just about the code it can generate out of a vacuum, it can also understand code for you.
I use it to generate doc-comments, check for differences between the comment and the code, or to explain code I'm not familiar with and don't understand myself.
E.g.: I don't use Python and I had to figure out what a spaghetti-code Python script actually did. I fed it to GPT 4 and started asking questions. That's way faster than learning Python first and then deciphering the script.
Back when GitHub Copilot was first released I though very much like you and was somewhat underwhelmed by my experience with it, however since then the state of the art has improved dramatically.
One of the reasons is that you can iterate to refine and expand your answer. Here's a transcript of a recent chat, where I was too lazy to write a script:
Me> I have a CSV file with a header row, whose first column is a date formatted like this: "March 16, 2022 1:51:19 PM". Could you write a script to convert the dates in this CSV to ISO date format? I am using a mac would would prefer to not install any software, but other than that you can choose any language you'd like
ChatGPT>Sure! Since macOS comes with Python pre-installed, you can use a Python script to read the CSV file and convert the dates to the ISO date format.
<provides script and details of running it, but doesn't handle the header row />
Me> the first row is a header row, could you please modify the script to handle that?
ChatGPT> Certainly! We'll modify the script to handle the header row separately, ensuring that it is copied to the output file without modification.
<script and details, now correct />
Me> can you make it so the input file is a commandline argument?
ChatGPT> Certainly! We can modify the script to accept the input file path as a command-line argument. Here's the updated code
You get the idea.
The way I use AI is to think at a high level how I want to implement something and then instead of writing the code tell the AI to write it by giving it specific instructions. Then proof-read it and fix the 1-2 things that it inevitably gets wrong.
I'm not reading some random code that I don't know what it's supposed to be doing. I'm reading the output of an instruction I gave that include some technical detail on how to implement it. Reading it and doing minor fixes is really fast.
I don't think there's universal agreement on that at all. Programmers universally agree that we spend more time reading code than writing it, but that doesn't somehow translate into it being harder.
Various studies have shown that programmers, on average, can only add at about 10 correct lines of code per day. We of course add more lines of code, but they tend to contain many bugs, so in the end only 10 lines of those are correct.
Maybe we only get 10 correct lines of code in because we're not reading the surrounding code correctly, and thus reading code is harder, or maybe 10 correct lines of code is just harder to write. I'm not aware of any hard data on this question.
"To evaluate the API usage correctness in code, RO- BUSTAPI detects the API misuses against the API usage rules by extracting call consequences and control structures from the code snippet, as shown in Figure 2. The code checker firstly check the code snippets to see whether it is a snippet of a method or a method of a class, so that it can enclose this code snippet and constructs abstract syntax tree (AST) from the code snippet. Then the checker traverses the AST to record all the method calls and control structures in order, which generate a call sequence. Next, the checker compare the call sequence against the API usage rules. It infers the instance type of each method calls and use the type and method as keys to retrieve corresponding API us- age rules. Finally, the checker computes the longest common sequence between the call sequence and the API usage rules. If the call sequence does not match the expected API usage rules, the checker will report API misuses."
The validity of this entire paper would seem to depend on how robust and agreed upon these "API usage rules" are.
“The misuse of APIs in the generated code could lead to severe problem, such as resource leaks, program crashes, this http URL make things worse, the users of LLM code generation services are actually the developers that are most vulnerable to these code that seems right -- They are always novice developers that are not familiar with the APIs that LLMs generate code for them.”
To be frank, I find it a little off-putting to suggest being a non-native English speaker is "no longer valid."
Which is... ironic I think :P
Hate this modern kneejerk reaction to get preachy and supercilious whenever anything vaguely cultural comes up.
You have to express reasons when not evident.
In case, just in case (and to also show how lack of definition may trigger any weak interpretation), after suggestion from a nearby member, the idea would had been that today people could have sentences reformulated by language models: the composer intends to express a logic, so any translator would have to grasp said logic primarily and first.
I'm pointing out that that isn't a good way to screen papers, because most English speakers don't share the same dialect.
Perhaps unstated or understated is the implication that, were you to discard papers in this way, you're liable to miss out.
I'll leave it to you whether that added information. That wasn't really my goal in this case, my goal was to respond to the question in the manner I viewed appropriate. Feel free to downvote my comment if you don't feel it was appropriate or informative or otherwise didn't uphold whatever criteria you're judging it by.
You did not make that clear at all.
> Perhaps unstated or understated is the implication that, were you to discard papers in this way, you're liable to miss out.
Indeed, that implication was totally missing.
> Feel free to downvote my comment if you don't feel it was appropriate or informative or otherwise didn't uphold whatever criteria you're judging it by.
I would certainly do so. Like their paper, it is quite poor, and not helpful for this forum. Unfortunately, /u/dang has removed my ability to downvote.
(For what it's worth, I often feel my comments are too long and wordy, and occasionally I've gotten feedback to that effect, so I was trying to be brief in my original comment.)
totally agree that dismissing the content of a paper for its grammatical mistakes alone is foolish.
this isn’t a paper about classic roman literature
People are missing the point here - it's not about writing code in one-shot. An LLM-enabled loop can generate the code and test and refine it until it works
People seemingly desire to zealously defend the LeetCode way of solving problems for some inane reason.
Allegedly the benefit is it will do the boring stuff like write the error handling code for you, but I have no faith it would work if it takes so many tries to get the happy path correct.
I agree; who am I to comment on how you write the code, as long as it gets written, right? That said, I've already had a "this is really odd code you committed last week, what's up with that?" and the reply was "oh, dunno, that's just what ChatGPT gave me". Meh. Actually writing things yourself does increase your understanding, so it's not the same, not really. It's when they told you to take notes at school: I didn't believe my teachers it'll help retain knowledge on account of being a stubborn little idiot, but they were right!
For people who have already done this programming thing for a while I guess it'll work out, but my main worry is the effect it will have on more junior people who will "grow up" on ChatGPT. "Figuring it out" yourself has a lot of value.
Just because the code is free of syntax errors doesn't mean it's free of bugs. Shell scripting is a classic case where it's hard to make something work but also buggy. You need to actually understand the code and reason about it. Trail-and-error programming rarely leads to good code.
I also fear we'll end up with hard to read overly verbose/repetitive code, "because ChatGPT/copilot will just generate it".
This is one of several reasons I worked there for a week only.
That being said, better workflows do exist. I imagine IDE-integrated LLM tools like Copilot and CodeGPT are much more productivity-enhancing than copy-pasting code between a browser and an editor.
Language models scale.
It doesn’t matter if a single pass doesn’t solve the problem, has syntax errors, etc. A single pass costs a fraction of a cent.
You can just automate the process of code -> create variant -> fix from LLM -> apply deterministic tests until the code at least compiles -> pass it to the user; the fact you can’t do that with chatgpt is just because it’s not a coding tool.
Heck, there are already companies doing exactly this with vulnerability scanners (ie. the deterministic feedback loop) to suggest security fixes for code.
You just repeat a heap of times until you get a solution that passes all the scanners.
If it takes 10 tries, or 100 tries, it still is zero effort from a human.
Of course, whether the result does the right thing is another matter, but the frustrating ergonomics of copy-paste cycle is because of the ui, not the technology.
You will see “surprisingly good” output from professional tools in this space (we already are); but a lot of it is not magic sauce; it’s just the same tools, run multiple times, in a way that saves you doing it manually and just shows the best results, with a pretty ribbon on it.
…and if it works (and it does) then it really goes to show that you (and me, and people in general) have poor intuition about this stuff.
It's a silly, potentially dangerous and inefficient way to "code", which is the easiest part of my job. If my job was just coding all day, I'd be happy.
[1] https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issue...
They'll probably ask ChatGPT to write an email to reply, so that's all good.
const response = await fetch(`http://text-processing.com/api/sentiment/`, {
method: "POST",
body: `text=${text}`,
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
});"the users of LLM code generation services are actually the developers that are most vulnerable to these code that seems right -- They are always novice developers that are not familiar with the APIs that LLMs generate code for them"
It looks like they could've benefited from an LLM doing a pass-over. Are they saying it's only "novice developers" using LLMs?
Really, as long as the premise, evidence, and results are clear and reproducible, the quality of the English doesn’t much matter.
Most of the users who are likely to turn to LLMs to help them code, are people who aren’t very experienced programmers. Hence needing help. These same people are more likely to fall for mistakes in the generated code as they don’t have enough experience to notice the errors.
LLMs are very good at sounding confidently correct about things that are subtly wrong. You need experience in the domain to notice these issues usually.
The combination of hallucination and just bad advice makes it time consuming and not clearly worth it. I think it's a step in the right direction for conversational interfaces. The fact that you really can have a back and forth to clarify intent, make corrections, and arrive at a "meeting of the minds" is lightyears beyond "intent recognition" type chat, and what a conversation should be. It just needs more work, which may or may not be straightforward.
Not impressed so far. With some effort and improvement this kind of tech can be used to fill boilerplate and glue code, and that is very nice as this is something that can help to stay in the flow by avoiding boring work.
But, the problem is that it is too unreliable even for this style of simple tasks, sometimes it works directly but it is too often plain wrong. Frustrating to use, not enough value, so I will try again later.
Just like how there is "common misuse pattern of x language", there is also "common misuse of copilot/chatgpt4".
From my observation, those who are successful with code generation tools are using it as an assistant, they already know the language quite well, AI is there just to help, most of the code are boilerplate code or common patterns.
Those who complaint are usually using it to do more than just that, relying it to generate working piece of code for a particular function without knowledge of the language. Currently, we use StackOverflow for this purpose, when we asked question, we don't just say, "hey, code please" but we are trying to understand why and how things work instead. This knowledge will then help us to code contextually, which current AIs are incapable of. The plus side is that code generation is faster.
Also the point of the paper is not to prove that GPT-4 is unusable but that it can be further improve. They didn't just manually determine misuses but did it via an API checker that they designed. So, in the future, AIs are just going to be more reliable not less.
We are not at a point that AI is common enough to talk about misuse yet though, most don't even have access to CoPilot or ChatGPT 4.
We’re already in this world. Seriously develop and app with Django and tell me how much actually “code” you write ?
I’ve been a developer for almost 2 decades and the coding part of the job is about 5% of my time :) The rest is stopping people from building stupid shit they don’t need to build and trying to make sense of “the business”
With generative AI, you can have all the stupid code you want in 1/1000th of the time :) :)
For me personally, we invented computers so we can spend time at the beach…ever seen the early Apple computer ads ? We shouldn’t be writing code constantly if we’re doing our jobs properly. We should be enjoying our lives which technology should be working and affording us the time to do so.
I've only found it useful for explaining basic terminology or concepts for a topic I'm not familiar with. Though this is the most dangerous use case since I'm not familiar with the answer, I can't easily fact check it.
It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes.
It is fantastic when I have to build functionality in languages I am not great at.
I regularly copy and paste code from my colleagues PR's and paste it into GPT-4, it explains it perfectly and gives tips on how to improve it (which I add as comments to my PR review)
As opposed to ChatGPT which spits out dozens of lines of code based on a request and therefore requires more involved editing after the fact to understand what’s happening.
It suggests solutions to problems, I add the complexity. So far my code has been both clear and performant. This should not be the case at my level.
What does this mean?
Copilot was released in March of this year. ChatGPT 4, which the community considers to be the only version of ChatGPT to be competent enough at coding tasks, was also released in March.
My guess is that you didn't mean that statement to be a fancy way of saying "I've been coding professionally for around 6 months", but I don't know what else you mean by the "age of AI".
I wonder if it's a language difference thing (I use C# mostly), but my experience with Copilot has been nothing short of mind-blowing.
It's not writing every line, and when writing truly new feature code it's less useful. But here are two patterns that I've noticed it is especially and consistently good at as a potential place to look for initial value if you are skeptical:
- If you have any kind of repeated pattern, even very complex ones, like performing a set of operations on a set of objects, or initializing a set of things, or whatever, it will guess everything else after the first line 90% of the time. This stuff is almost always plumbing/wiring/boilerplate-type code, so pure time gained. Think about Excel filling incrementing numbers down a column, but for pattern-matched lines or blocks of code.
- For any reasonably testable class, if I write the name of the unit test, Copilot will write the entire unit test perfectly 90% of the time, down to variable names and //Arrange//Act//Assert comments that I stylistically prefer. Seriously, it's sort of scary how good it is at this.
I can understand using chatgpt 4 but copilot??? copilot can barely autocomplete... and half the time it does it incorrectly when variables and other stuff are involved
Which isn't easy, because there's virtually no documentation for it.
Once you get the hang of it it's incredibly useful. I really feel it when I'm working in an editing environment without it now.
> It writes lots of code before I even think about what I was going to do in that file.
If you haven't even thought of what you want to do in the file, how are you prompting Copilot to write the code you need? Am I understanding correctly that Copilot not only generates the correct code, but it's also able to deduce what code you meant to ask for without any input from you? I don't see how this is possible except in the uncommon case where you are bringing a file to be more in line with other files in your code base that have a similar structure.
I do think Copilot can be useful, but I'm confused at how different my experience is from what others are getting out of the tool.
You can know what you want to program, but not how you would approach the code. "Hey ChatGPT, how do I draw a curved line plot in JUCE?"
If I just ask it "Can you write me a script that gathers the current stories on the front page of hacker news and sorts them by number of comments, then prints the title and number of comments?" it writes a script in Python using requests and BeautifulSoup4.
Otherwise, even just creating a function, generally they will be filled out automatically.
I suppose I'm quite big on naming (vars and functions)
Maybe if your language or code is more terse, it might struggle more.
I've found GPT4 moderately helpful, like getting help from someone with a wide but shallow experience of lots of things.
> I can't easily fact check it
Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT? Apply the same methods, although you just have to fact check rather than coming up with what to check, so removes a small initial discovery, at least for me.
I’m not well-versed in JavaScript (I’d like to be, but neither me nor my company have the resources right now to commit). When JS work comes up, I’ll often Google/ChatGPT to point me in a general direction before pulling up technical documentation on whatever comes back. I’m pretty good at learning, but sometimes it takes me a while to find out where to start.
However, there’s a few things like SQL and JavaScrip that are likely to stick around mostly unchanged for your entire career. The risks are therefore much lower.
IMO, a reasonable heuristic is spend around 10% of the time you expect to work on technology in the next year actually learning the fundamentals until you feel comfortable. Often what looks like a major time sink goes away when you stop fumbling around.
I'm not interested in starting at the beginning and learning a whole new language or API, I just want to get my task done, so I can ask GTP to get me started, then I can run the code it writes and start building tests and things to verify it works as expected.
Because if it tells me something that could just be found on google, I could have found it there first and not have to do as much fact checking. And if it tells me something I can't easily find, I can't tell if it's got some deep insight on harder to locate info, or if it's just made up.
I think we'll have AI some day, but today it's just not at the level that all the hype claims it is.
Now the AI bros here are attempting to sell us their new snake-oil in the form of a stochastic parrot which promises to be the solution to everything.
Well, unsurprisingly it is another unexplainable AI black box which still requires the human to check every single output so that it doesn't hallucinate something incorrect, which it does almost all the time with a lack of transparent explainability as to why it hallucinated.
So the fact is, it is already overpromising and under-delivering for serious use-cases. Just like FSD (Fools Self Driving) was when that was over-hyped and with little to no Tesla Robo-taxis on the road.
That said I'm not asking ChatGPT to write code that I use directly, I'm basically asking clarifying questions about the papers and other implementations I have on hand, and that's similar to how I use it in most cases.
For code generation I rely on co-pilot, which actually has the context of my codebase.
Edit: I will say this because someone else made me think of it: I find this tech all super useful on my like 20k loc and smaller side projects which are all self contained where as when I tried co pilot at work on a fragmented big corp code base I found it comparatively lackluster.
I work in data science and have to write a lot of repetitive stuff for parsing and cleaning data. It’s reduced my toil so dramatically in this respect, I can’t imagine going back to writing all that stuff again. I’m now much more ambitious in what I’ll experiment with as well because I know setting up the first stages of the data pipeline are going to be 10X less work than before.
I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer.
It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly?
It often misapplies interfaces on the first go-around. Again, this is just like humans. We make mistakes, notice them, fix them. If you simulate this by telling GPT that its code produced an error it will often correct itself.
The absolutely killer feature of GPT4 is that it has these skills in every subject. It's fluent in kernel operations. Databases. Networking. Various UX frameworks. Any language.
It's definitely not perfect. But, if the alternative is hiring a mid-level human engineer, GPT4 is a really compelling alternative.
Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.
I’ve got 20 years of experience in the tech industry and chatgpt is far far better than any mid level engineer I’ve ever met and I include FAANG engineers too.
Chatgpt can answer any leetcode programming question in essentially every programming language in existence in less than 20 seconds.
Chatgpt can analyze classes for errors written in every language in seconds.
Chatgpt helped me write animation, ai, sound handling code in c++ and helped me build a full working game, the total code generation time for chatgpt “thinking” was on the order of one hour.
I don’t know any senior engineer who could help me (a c++ newbie) write a whole working game engine with all the necessary systems in less than one hour of “thinking time”.
I’m starting to think that all the “naysayers” of chatgpt simply used it once or twice, saw one error and dismissed the whole thing. Or they’re simply afraid for their own skills and avoid chatgpt as the alternative scares them too much.
In my experience, if you compare the unedited output of ChatGPT to the unedited output of someone with a couple years of experience in the given domain, ChatGPT will have more subtle bugs than the human provided output.
Hence my between junior & mid assessment. It doesn't mean that it's useless though. Having a tool at that level of experience for every domain in the world that can cook up code near instantaneously is damn useful. And I assume it'll only get better.
At writing code, that might be a slight exaggeration. At explaining code it's at least at mid level, possibly better.
I’ve used chatgpt recently to build an entire c++ game with sound, keyboard and mouse control, ai, etc, having never used c++ before. I’ve also never built a game before. I’ve worked with a lot mid level developers and I’ve never seen any that can write me a a c++ file giving me working code for a collision system in 30 seconds. Let me know if you find one though.
With everyone and all the news talking about it constantly inevitably people are going to start rolling their eyes.
This is bias. 48% is half way to 100%. Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is on your side.
It's typical. It's like an indie band is only popular when not many people know about it. Once everybody starts talking about it all the time it loses it's popularity. People start thinking it's not cool anymore. That's the bias you are suffering from.
It works but only edge cases are a problem. Next step is to fix the edge cases.
What this study looked at was feeding StackOverflow questions to LLMs and then looking at the quality of the code. If you think a programmer's job is just turning an english-language description of a function into isolated code that never gets modified, I don't know what to tell you. In my opinion, anybody like that should not have a job today, never mind the future.
A proper professional programming job involved AGI-level understanding of human users and their needs as embedded in a social context. Plus the ability to create novel solutions. Plus the ability to use code as a collaborative medium to make a code base that is sustainable over the long term by their colleagues.
LLMs are not even 1% of the way to replacing professional developers.
To answer your complaint, I am absolutely looking at reality. Please point us to an actual real-world professional programming job, matching the criteria I list above, for which a current LLM can economically replace a human for over a 5-year period.
My believe is that there are exactly zero such jobs. If your claim is that we're at least at 1%, then you're claiming that there are at least 269k fully automatable programmer jobs [1]. It shouldn't be hard for you to find at least one to start.
[1] Wikipedia says there are an estimated 26.9m professional programmers: https://en.wikipedia.org/wiki/Software_engineering_demograph...
If I built an entire car but I am missing the key. I cannot drive the car but the car is 99% complete. It's just missing the key. So what happens is, it can replace 0% of human locomotion but it's 99% complete? get it?
Just because the tool can't be used doesn't mean it's zero percent of the way to being drive able. Same with your job. The LLM can't replace any job yet, but it doesn't mean it's 1% of the way there.
But you see what I just explained to you is obvious. You already know this just like I and every one else on the face of the earth already knows this fact.
You're taking the discussion into a play on words. What does 1% apply to? Rather then use common sense and derive what I mean you prefer to redirect the conversation into a very specific definition of 1% that serves your own purpose.
We can play this game all day. We can discuss whether your application of 1% is more fitting than my application of 1%. What a waste of everyone's time.
I think it's better if rather then playing these games to "win" discussions, use your common sense to move the discussion past these games. Otherwise we're going to be talking about obvious things all day and getting all worked up about personal definitions.
Clearly the LLM is more than just 1%.
Mine is clearly measurable. You wanted to talk about job replacement; I'm measuring jobs replaced. You believe that we're at 48% complete (or 38% or whatever) but you can't point to even one programming job that ChatGPT can take over. If your fantasy 48% measure means 0% real-world success, I think the 0% is the more useful number to look at.
Humans may not be smart enough to build human-grade minds, in the same way that cats will never learn to program no matter how much they paw keyboards. People mistaking ChatGPT for AGI strikes me as the same kind of error where they mistook Eliza for a person, or 1970s/1980s computers as the things that would develop consciousness and maybe take over. It doesn't tell us much about the power of computers, but rather the inability of some humans to understand how complex and powerful minds are.
unfortunately the hard topics and the topics that matter are the questions that matter most are often qualitative.
Measurable answers are easy. Nobody cares about those things because it's obvious. Has ChatGPT replaced any jobs for programmers right at this moment? Overall no. But why argue a point everybody already knows?
Will chatgpt replace you in the future as the technology develops? Is it a precursor to the machine that will replace you? Qualitative trend-lines point to an unknown artificial entity that does replace all of us. It is useful to speculate and extend the trendline in that direction.
You may wish to shield yourself to reality and only look at things with quantitative numbers and lengths that are measurable by rulers but the shield is an illusion what you are doing is blindness.
Natural selection and natural history is built from qualitative understanding. Without subjective analysis on qualitative data we would not be able to understand natural selection from the macro perspective. You would be ignorant of the concept of evolution and natural selection and not much different than a creationist.
Obviously you aren't like that, you're just using a tactic to win a discussion. Always use hard data works when it works. But ultimately this strategy has failed.
And 38% is not "a third way to 100%". I'm sure you know what diminishing return is.
No?
I assume you actually don't know the concept of diminishing return, and it's totally fine. Every other comment here is explaining it to you already. I won't bother to repeat. Please read the sibling comments carefully.
You brought up this question of diminishing returns out of nowhere and offer no evidence for it. So why would my point not stand?
I read the sibling comments. One person brought it up but offered no proof that this is what's happening. Simply did what you did in a manner that was less rude, just stated the concept of diminishing returns applies to LLMs without offering a shred of evidence that indicates this is what is happening.
There is a difference between introducing the concept of diminishing returns out of nowhere, and providing evidence that diminishing returns is WHAT is actually HAPPENING with LLMS.
We're about a year out from the introduction of chatgpt, that's not enough time to know if all the gains in the past decade of AI has suddenly hit wall of diminishing returns.
You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken.
> It's typical. It's like an indie band is only popular when not many people know about it. Once everybody starts talking about it all the time it loses it's popularity.
I think applying the sociology of hipster band fans to LLMs is a mistake and I’m not sure how the vogueness of the technology correlates with the correctness of the actual models. Sometimes an early technology seems docile or useless at first but eventually reaches ubiquity and in retrospect the utility is obvious. But sometimes (more often than not) standard adoption curves don’t make sense to apply to a new technology because it isn’t useful enough to go through an adoption cycle. I think there’s a temptation to apply the analogy of something like the early internet or smartphone to LLMs but those are networked products. LLMs don’t really improve with the various applications built on top of them if the LLM is itself fundamentally broken or faulty to the point that it is unsafe to use in practice. Furthermore given the massive amount of premature hype due to the AI zeitgeist, you can safely assume enough user hours have been spent messing with LLMs to get a verdict on their utility. Unlike the feeble technology that takes a longtime to reach a scale to know its utility, we don’t have to wait 10 years for 100m people to try LLMs, it happened in a week. My only point being, I think we should be apprehensive about trying to draw analogies of other adoption cycles that structurally are very very different to this one.
Obviously, I, like probably everyone else on this website would love LLMs to be reliable to a high degree. Just this morning I had such a good use for an LLM that I was seriously considering building (and admittedly still am pondering) but the second I started to think through the LLM faultiness, I had to consider the complexity of the safe guards and weigh if it was really better to use GPT or just write a nasty regex script and constrain the problem. I’m leaning heavily towards the latter but I’d much prefer a silver bullet if it really killed vampires. Until then (if that day ever comes), it’s lead bullets for me.
Nah even 100% reliability isn't attainable by a human. 100% obviously doesn't imply solving the halting problem.
100% as in 100% as reliable as a human. Even surpassing a human. Sort of in the same way as a computer now beats humans at chess.
>I think applying the sociology of hipster band fans to LLMs is a mistake
Why would it be a mistake? Human psychology is similar across all spectrums. What happens in one area is likely possible in another area. If it happens among hipster bands it can happen among hipster technology fads.
>I think there’s a temptation to apply the analogy of something like the early internet or smartphone to LLMs but those are networked products. LLMs don’t really improve with the various applications built on top of them if the LLM is itself fundamentally broken or faulty to the point that it is unsafe to use in practice.
This is a valid speculation. But it's speculation. Basically your saying that LLMs are fundamentally broken and stuck at 38% forever because of fundamental and permanent flaws. The jury on that one is still out. And your point is highly, highly speculative.
We see quantitative improvements on LLMs constantly AND this is a nascent technology we don't completely understand yet. The most probably and logical conclusion is to follow the technological trendline. That trendline is pointing up.
To speculate on fundamental flaws of the LLM when people don't even fully understand what's going on with the LLM is illogical because you can't derive conclusions from something you don't understand. We can only generalize the trendline and constant improvements we've seen in AI for the past decade. Again that trendline is pointing to further break throughs in the future.
>Obviously, I, like probably everyone else on this website would love LLMs to be reliable to a high degree.
No this is not obvious to me. I disagree. I think some people are like you but other people, for example Geoffrey Hinton are in the apocalyptic camp. Personally I'm in the middle, I think it could go either way. It will definitely harm a segment of our society by taking over work, but whether the benefits of AGI outweighs the harm remains to be seen.
Look. People in reality are basically never this gigabrained. Maybe consider as your first-line theory, that when people say "AI will destroy the world", that they mean to express that they believe that AI will destroy the world?
But to be clear I’m not saying this is a methodical highly coordinated ad campaign amongst multiple companies. I don’t think human beings are that competent. I think it’s way more grassroots and feeds on cultural tropes that have existed since at least the 1950s but probably the early 1900s. I think Anthropic’s leadership probably genuinely believes their own bullshit but I also think they understand that that same bullshit has raised over a billion in funding.
Your point was on whether everybody wants to see AI take over. To that Geoffrey Hinton doesn't want AI to take over, while you do. I for one am not sure.
As for whether Geoffrey Hinton thinks AI is legit or not is a different story. On that topic he's on my side, but that was besides the point I brought him up to point out that your take on "everybody" wanting AI to develop further is incorrect.
> and the doom and existentialism is a marketing ploy to a world imbued in conspiracy theory and institutional distrust in order to compensate for a wildly over promised and under delivered product.
Other way around. With all the money and business interests going into LLMs business interests are promoting LLMs in a future that you want and that is not apocalyptic. The conspiracy theories aren't a thing. It makes no sense as those theories don't align with where the money is being thrown.
>But hey, when you gotta raise money, you gotta raise money, and as you put it, the "trend line is going up" and that's all that matters.
Bro. I am not saying "trendline" as if it's something I have to keep throwing money at to support.
I am talking about a mathematical projection based on data. The pace of technology in the past when graphed points to an ever increasing line on a line graph. When you take the slope of that line and use it to do a quantitative prediction, that line just points up. That's just a fact of reality. The logical outcome of the data we see.
It's a great assistant. I use it. It's not taking anybody's job.
Many programmers also do work that doesn’t just involve stitching APIs together; GPT consistently fails badly when forced to reason, making it useless for many tasks. For example, ask it to implement a common algorithm like SHA - it will do a decent job. Then ask it to do it with an arbitrary limitation which no implementation it was trained on would have (e.g. use no integer type wider than 8 bits). In my experience, it cannot achieve this task with a correct result, even with significant hand-holding.
I meant 38% aka 40% made a mistake on my math.
It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption.
I understand where you're coming from though. Many products made by entrepreneurs follow that model where the remaining percentage is always harder than the beginning green fielded tech that was built. But can we honestly say that this is what AGI will be like? The LLM was an unexpected jump forward by an unprecedented amount. It could be we fill the gap to 100% with another such jump.
And it's wrong, as anyone with experience in ML will tell you. 60% is easy. 70-85% is not too bad. 95% is hard. 100% is effectively impossible. This is the core problem of ML systems. They're good enough for 90% of cases but that last 10% can be incredibly important, to the point that the systems become unusable.
You seem to think the topic is about the current state of training algorithms and how it's hard to bring the model to 100 percent. That's a different topic with a subtle distinction.
I am in the latter group, and I don't find it all that helpful. There simply aren't any tools that can plug into a massive codebase with millions of lines. I never just work on one specific repository either - a change generally involves multiple repositories.
On the other hand, if you're writing a smaller standalone utility, or working on a greenfield application, it can be helpful (with all the useful caveats about hallucations).
I think that a lot of ML folks, especially those in research, fall into the former category, which is why there has been so much hype.
If Sourcegraph's offering (https://about.sourcegraph.com/cody) has limits I'm not aware of, this is just a matter of time. But in my experience, this is a "nice to have" rather than a "must have" when it comes to the benefits of GenAI as it applies to coding.
The generated code will likely have a few small issues, but it still makes me way more productive. It allows me to work ~10 hours a week less and enjoy my life.
At times, I need to correct the LLM on some usually obscure detail. It got the memory layout of the video screen on the BBC Micro completely mixed up with the ZX Spectrum, and I had to correct it about five times before it got it, and then it stuck for the rest of the conversation.
I've been on some wild goose chases, where I have fooled myself into thinking a particular approach would work and ChatGPT has been enthusiastically right there cheering me on.
It's like my dog going on an adventure with me, the dog doesn't care about where we're going or what we're doing, it's just excited to be part of the journey.
And in other cases, ChatGPT has been exceptionally useful in pointing out some blindingly obvious mistakes. It is like having a very knowledgeable, but exceptionally junior developer at your elbow.
If you ask the right questions, and don't except one shot questions to provide perfect answers, it works well for the most part.
The most annoying part is it's impossible for it to say that it doesn't know something. Hard to train for, I know.
Anyone with an understanding of unexplainable black-box AIs knows that LLMs hallucinating is not 'the same thing as a human'. Humans can be held accountable for their mistakes and can explain themselves transparently. LLMs fundamentally cannot reason or explain themselves transparently, other than rewording its original answer(s) to make themselves sound credible; like an expert sophist.
It goes to show that this overhyped snake-oil is now at the late stage peak of inflated expectations of the Gartner hype cycle.
Think to how many times you have seen code written with the caveat 'this is a general gist. I haven't tested this code'
I've seen it many times. More than anything else really. So it should come as no surprise that models trained on that data produce similarly wonky code.
This is then compounded by the fact that the user thinks of the model as they would a person. They receive a humanesqe response. So attribute the expected human backend. The understanding of the request and the feedback.
All current "AI" LLMs are just Chinese rooms.
AI isn't going to give you perfect code (or answers, or anything really). It's going to give you typical code, based on extremely broad "intuition" about how others ("all" others, really) have solved (or answered) the same problem. And that has value.
But it's not going to produce something better than the existing consensus, by definition.
ex: "This code `(some-fn 1 2)` does x because y. That is incorrect because abc"
I wondered if this has to do with common StackOverflow post formats.
I understand the need to stop people from editorializing in the titles, but I saw this within the first hour it was posted and just spent 15 minutes looking for it again because the title change didn't contain the two keywords I remembered - API and GPT.
Old Title - 62% of code generated by GPT-4 contains API misuses
While such a set may be interesting to study in itself, it does not seem like a reasonable dataset for evaluating LLM-generated code overall. Of course, if significant issues are found with this set (and they are) - then there are significant issues overall (and there are).
Like can Liutenant Commander Data be excluded from human competitions? Yes. But can a replicant from Blade Runner? I guess the answer is that they will not feel the need to compete. But what if they do?
Level 1: https://en.m.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_T...
Meanwhile copilot is becoming a superpower to those who figure it out.
And it hasn’t even been a year! I’m going to need a lot more popcorn.
That said, I'm really underwhelmed by the output of both those tools in the context of the ongoing hype. They're far away from taking anyone's job. They're producing misleading and even manipulative output in such fine-granular deceiving manners, it's like reading a whole contract in fine-print. Given that our job is already mostly reading and understanding vast amounts of code, I don't really see a productivity boost in reading generated erroneous code. And if it boosts productivity because of boilerplate, we already have plenty of old tools to deal with that. Heck, every CRUD Framework from mid-2000s is better at generating boilerplate code, with 100% correctness.
So there's no "big disruptive innovation" going on but new nice-to-have tools. Just know, when not to use it.