Comment by keyle | Hacker News Reader

Comment by keyle | Hacker News Reader

Parent Full thread

Ok but what are we calling API misuses?

    We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software.

Should have used ChatGPT to correct those sentences. /s

What do we consider an API misuse at the end of the day? And also against the Java APIs... Which are some of the oldest.

I'm not saying ChatGPT answers work, 50% of the time they don't work out of the box. But overal, the time savings are incredible, compared to the fossil digging way of googling. And I value the time savings towards the equally correct way of googling for answers. You reap what you sow. The knowledge the LLM is based upon is the same stuff you used to distill manually. It shouldn't be more or less correct, if you think of it that way.

Surprise, ChatGPT will not replace devs. But it is like having the electrical drill vs. screwdriver type of situation.

Evidently, THEY are calling the following API misuses:

"To evaluate the API usage correctness in code, RO- BUSTAPI detects the API misuses against the API usage rules by extracting call consequences and control structures from the code snippet, as shown in Figure 2. The code checker firstly check the code snippets to see whether it is a snippet of a method or a method of a class, so that it can enclose this code snippet and constructs abstract syntax tree (AST) from the code snippet. Then the checker traverses the AST to record all the method calls and control structures in order, which generate a call sequence. Next, the checker compare the call sequence against the API usage rules. It infers the instance type of each method calls and use the type and method as keys to retrieve corresponding API us- age rules. Finally, the checker computes the longest common sequence between the call sequence and the API usage rules. If the call sequence does not match the expected API usage rules, the checker will report API misuses."

"compare the call sequence against the API usage rules"

The validity of this entire paper would seem to depend on how robust and agreed upon these "API usage rules" are.

Everything depends on something else. That's life.

I just don’t see how it could save time.

Programmers nearly universally agree that reading code is harder than writing it.

So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems.

Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.

"I just don’t see how it could save time."

Have you tried it much?

It's saved me a ton of time over the past 8 months. I don't know what else to say - I'm not the kind of person to deceive myself into thinking something is saving me time when it isn't.

The time saved is mainly in the micro-research you no longer have to do - the bits where you have to go and look up how to write a for loop in Bash, or how to call super() in a Python class, or which exceptions you need to catch for a call to httpx.get().

GPT-4 writes the code right 90% of the time. The 10% of the time it doesn't you catch in the same testing you would have done against code you had written yourself the long way.

I have found it to save time when I'm learning a new, widely used framework. (Presumably this would hold for a language as well.) I used it to learn Keras and React + Redux, and it was helpful there. For instance I was able to ask it about what sort of messages would be emitted by Redux in a given circumstance, and the precise values were wrong but it didn't really matter. That was very cool.

Once I have a good sense for how things work, I'd much rather go to the documentation. Most of the time I can do this straight from my editor by typing the method I want and querying the LSP for documentation. In those cases it's more or less instant, there's no improvement to be made there.

Maybe in some of the other cases it would be faster via an LLM integrated with my editor, but I don't think it would justify the cognitive load of considering whether or not it's correct. I'd like to worry about whether the application is correct. And I don't really think I'm wasting time waiting for the Python documentation to load, I'm continuing to chew on the problem.

Additionally, when I go to the documentation, I'm looking for callouts about the safety of an API, and I take the lack of such callouts as authoritative. Eg, recently I was reading that in node-postgres you have to return connections to the pool, which was a good refresher for me, because previously I'd been using Rust where resources are generally cleaned up automatically. I just don't trust a lack of a callout from GPT-4 as being a confirmation of a lack of a safety issue.

GPT can read and understand both code and the comments about 1000x faster than a human.

It's not just about the code it can generate out of a vacuum, it can also understand code for you.

I use it to generate doc-comments, check for differences between the comment and the code, or to explain code I'm not familiar with and don't understand myself.

E.g.: I don't use Python and I had to figure out what a spaghetti-code Python script actually did. I fed it to GPT 4 and started asking questions. That's way faster than learning Python first and then deciphering the script.

GPT won't only writes code, but also documentation and unit tests. It can also read code for you and explain what is going on. Furthermore, even as an academic researcher, most of the code I write is simple boilerplate code that GPT excels at. This means I get to spend more time on the actual code that is hard and novel and that GPT cannot really help with.

Back when GitHub Copilot was first released I though very much like you and was somewhat underwhelmed by my experience with it, however since then the state of the art has improved dramatically.

It depends on your baseline. If you're comparing it with "I know how to do this already but don't want to type" I think ChatGPT is slower. If you're comparing it to "I can't remember how to use this API so I'm going to read a bunch of Stack Overflow posts" than ChatGPT is seriously faster.

One of the reasons is that you can iterate to refine and expand your answer. Here's a transcript of a recent chat, where I was too lazy to write a script:

Me> I have a CSV file with a header row, whose first column is a date formatted like this: "March 16, 2022 1:51:19 PM". Could you write a script to convert the dates in this CSV to ISO date format? I am using a mac would would prefer to not install any software, but other than that you can choose any language you'd like

ChatGPT>Sure! Since macOS comes with Python pre-installed, you can use a Python script to read the CSV file and convert the dates to the ISO date format.

<provides script and details of running it, but doesn't handle the header row />

Me> the first row is a header row, could you please modify the script to handle that?

ChatGPT> Certainly! We'll modify the script to handle the header row separately, ensuring that it is copied to the output file without modification.

<script and details, now correct />

Me> can you make it so the input file is a commandline argument?

ChatGPT> Certainly! We can modify the script to accept the input file path as a command-line argument. Here's the updated code

You get the idea.

> Programmers nearly universally agree that reading code is harder than writing it.

I don't think there's universal agreement on that at all. Programmers universally agree that we spend more time reading code than writing it, but that doesn't somehow translate into it being harder.

Various studies have shown that programmers, on average, can only add at about 10 correct lines of code per day. We of course add more lines of code, but they tend to contain many bugs, so in the end only 10 lines of those are correct.

Maybe we only get 10 correct lines of code in because we're not reading the surrounding code correctly, and thus reading code is harder, or maybe 10 correct lines of code is just harder to write. I'm not aware of any hard data on this question.

Sounds like those programmers are spending a decent amount of time stuck in meetings :P

ChatGPT is pretty good about telling you what a line of code does. So I just copy it in and ask it.

Not all reading is the same. If I tell an AI to write code that does X then checking if a provided code actually does X is a lot faster than writing it myself.

The way I use AI is to think at a high level how I want to implement something and then instead of writing the code tell the AI to write it by giving it specific instructions. Then proof-read it and fix the 1-2 things that it inevitably gets wrong.

I'm not reading some random code that I don't know what it's supposed to be doing. I'm reading the output of an instruction I gave that include some technical detail on how to implement it. Reading it and doing minor fixes is really fast.

To write the code you need to know what to write. In many cases I find it easier to read proposed solution to my problem and then figure out from there how I actually want to implement a solution. It's great starting point.

Depends what you're doing. If you tend to hop between languages, software stacks, platforms etc. then you probably (as I do) spend a lot of time looking up what the options are to achieve any given task, finding and interpreting API calls, etc. Having an LLM spit out code that's in the right direction would be a huge time saver.

I have been trying to use ChatGPT to make a VueJS website (I’m a backend dev). What I have noticed with VueJS specifically is that it has little to no knowledge of the composition API, and when I ask it to help build a component it is pretty good at the scaffolding. The component will generally work; however, it is pretty useless for CSS styling.

I probably should try Copilot as I bet that would work better. Also the VS Code integration would be really nice.

Copilot probably wins because it holds more of _your_ code against the desired output.

I find ChatGPT is really great at getting something going, think scaffolding wise. But if you enter the realm of gnarly stuff, it can get completely delirious with answers full of red herrings and sends you running in circles very quickly.

The only truly helpful use-case I’ve found is in generating some well defined text in a different form. Say - here’s a JSON object , please write a MD file describing it or turn it to X language struct/object.

How stable was the Vue composition API prior to the September 2021 training cut-off date for the OpenAI models?

Looks to me like https://blog.vuejs.org/posts/vue-2-7-naruto back-ported it in July of 2022, and the earliest snapshot of the documentation for it in Vue 3 was https://web.archive.org/web/20220210121247/https://vuejs.org... in February 2022 - so my guess is it mostly happened after that magic date.

ChatGPT is very bad at CSS. The only good part is you can test it and see that it’s not working very quickly. Unfortunately if you go back to ChatGPT it will waste your time with endless apologies and more broken CSS.

Can you confirm if you used gpt-4? This was not my experience

It was gpt-3.5 (the free version offered). I will give get-4 a try though that is good to know.

Use Bing chat to get access to GPT-4 for free.

Reply on news.ycombinator.com