Don't fully trust AI in dev work /yet | Hacker News Reader

Don't fully trust AI in dev work /yet

blog.akom.me

9 pointskaumnen20 comments

I don't personally use Github Copilot but this article almost makes me want to. I can see how mindlessly pressing tab can get you into situations like this but if you already know exactly what code you want to write then Copilot probably saves a lot of keystrokes. You just need to make sure it's not writing the code for you.

Keystrokes do not take detectable part of time it takes to author code. Debugging does.

Copilot lets me think about the code without being distracted by the boring parts. And from my experience, blunders like in the article happen rarely.

Copilot feels a lot safer when combined with static analysis output that immediately tells you when it is doing something wrong. TypeScript + ESLint together give me a lot of confidence in being able to determine whether or not the output is any good. I also review the doc comments of any library function calls it generates that I'm unfamiliar with to make sure it's not doing something that only looks okay. In the end, it's not so different from needing to do a code review of code written by any of your collaborators.

It's true that you shouldn't "fully trust" AI in dev work yet (or probably ever). But I think it's also missing the point. After all I wouldn't blindly trust my own code that makes a request to an API endpoint either, even if it's basically just copied from the docs.

I will post this to the next guy who tells me that github copilot saves suuuuuch a lot of time which I, old fashioned, vim-using as I am "waste with all that typing". :D

I happily use neovim with Copilot today. You can have both. :)

https://github.com/github/copilot.vim

Don't need it :-) I just write my code the old fashioned way.

Try writing unit tests with Copilot and you won't want to go back. When it comes to simple repetitive code like that, it excels.

I was playing with copilot last week and was managing to auto generate trivial code by just by writing comments. I found out that "# test function1" also wrote some trivial tests. All my examples were along the lines of "# get all even numbers in a list", but it did really impress me.

I've seen some push back at work against Copilot. Obviously, you the developer is ultimately responsible for the code you commit.

The problems I see here are:

1. The parameter was named accessSecret, but the environment variable was (...)ACESS_TOKEN_SECRET. This type of inconsistency can easily confuse a human programmer as well.

2. The programming language (JS) allowed an extraneous parameter to be included. Languages with stronger typing would at least allow the IDE to mark the parameter as unused/invalid.

3. The library didn't complain about a missing important parameter.

You have to carefully read the code before pressing tab as it sometimes might introduce things that look the same almost but are not. But besides that it is pretty good at saving from typing the most boring drivel (functions for which should be in the language or stdlib really).

I’ve had stories like this too, after the third time, I began triple checking the var names and syntax. When copilot works it’s really cool, when it doesn’t it can be frustrating to debug bec you didn’t write the code.

Why didnt intellisense pick up on this, i dont use github pilot, but does that override intellisense or wasnt it installed correctly ?

This sort of thing can happen only in a language with a type system where including an extra parameter is fine. So JS or Python, and you have to be using an IDE that's not too smart or it will detect that the parameter is unused anyway. In most other languages (and I'm pretty sure even in JS/Python if you use JetBrains IDEs) the Copilot's code would immediately be underlined.

Could someone explain this to the layperson?

The TwitterAPI constructor expects an object with an "accessSecret" key (among others) as parameter. The Github Copilot suggested "accessTokenSecret" instead. Even though the suggested value was correct, it seems that having the wrong key name means the secret is not sent, which causes the 401.

Thanks!

Which is a weird behavior by the library. It should complain when it is supplied no secret, and not just blindly send the request and hope for the best.

How does that even compile... /s

Reply on news.ycombinator.com