Hacker News Reader

Comment by fourseventy | Hacker News Reader

There is no way 33% of code is written with Copilot. I want to see the data backing that claim up.

I work for GitHub (although writing in an entirely personal capacity) and the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help.

I don’t know about how the data was gathered, I was not involved, but the segment is 4 minutes in so you can see what the actual announcement claims here: https://youtu.be/etMvd9IKPH4

I have used copilot and watched a colleague writing comments and frequently getting great go code suggestions without loads of edits. I personally am sincerely very impressed.

Thanks for sharing. So in other words it's the journalism that is wrong.

So 30% of code written by users with the extension installed was written with some extension assistance?

How exactly does that translate into the claim being made? Is github asserting the 100% of users have the extension installed?

My guess is that some small number of trial users did their homework with it then ran to stack overflow when it did something other than what they hoped.

> the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help.

The gap between this statement and what's in the OP title and text is too big. It's just false misleading bad journalism.

I need to write an article titled "People who install Copilot only use it for 30% of the code".

That video says: "Since we started, 30% of all code written by early adopters come from copilot suggestions."

That is a far different headline than the one on this post. WTH is wrong with journalists now days? FFS.

People writing code outside of using GitHub are likely more skeptical towards Copilot. We shouldn't forget that other, FOSS Git forges exist.

Martin from GitHub here. While Copilot is good it's not quite up to those numbers yet I'm afraid. It’s early days, but we’re already seeing that for some languages when people have it installed, we see about 30% of newly written code being suggested by Copilot. Vary’s between languages and the numbers of users in the technical preview is a LOT less than the 73m folks we have on GitHub, so a long way to go before it makes up a third of everything..

For more information, the interview with one of the creators of Copilot has been posted here https://www.githubuniverse.com/2021/session/707714/inside-gi...

That doesn't seem to be the claim. They say one third was written with AI help.

Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it.

edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.

Even if you do the "% of users using Copilot" approach, Copilot was limited to both Visual Studio Code (until today) and an invitation-only program.

The fact that the combo reached 30% of newly generated code doesn't make sense.

Ya, it is clearly a lie or the journalist was clueless

The specific claim in the video was % of code being written by early adopters. I can't tell from the video how "early adopters" was defined or how they measured % of code so it's ambiguous. But story should have made it clear this isn't a percentage of all new code.

I hadn't seen the video quote until it was posted above: I assume "early adopters" is those who were accepted into the program and installed it, but yes it's still ambigious.

I mean, Google search uses AI, and developers use Google to find things, so technically you could say the vast majority of code is written with "AI help", but that's not exactly a noteworthy observation to make...

Yep! I was just watching how clang-tidy can now automatically apply code fixes in realtime. Is this AI?

https://www.youtube.com/watch?v=EfYMKOMc738

Maybe they're just measuring lines committed (total guess)?

If you count white space, comments, variable and function declarations, and boiler plate code like creating loops, then maybe I could see how they got there.

Kind of related, but if I think about autocomplete, it probably writes about 10%-20% of the individual characters in my code. While it is totally misleading, it wouldn't be factually incorrect to say that autocomplete writes X percentage of my code. I wouldn't be surprised if GitHub is using some "misleading, but technically true" metric for this.

+1.

>Nearly a third of new code on GitHub is written with AI help

May be true if you count VSCode's IntelliCode, JetBrains IntelliSense, CoPilot, and other similar tools together. But no way copilot has that much marketshare yet.

Maybe it's bots doing it? Someone might be trying to fill up Github with autocompleted code.

Reply on news.ycombinator.com