Hacker News Reader

Comment by kiwih | Hacker News Reader

Hi - I am actually the lead author of this paper! I'd be happy to answer any questions about the work.

Is there equivalent empirical data from real programmers?

That is to say, you have code prompts here, let Copilot fill in the gaps, and rate that code. Is there a study that uses the same prompts with a selection of programmers to see if they do better or worse?

I'm curious because in my testing of copilot, it often writes garbage. But if I'm being honest, often, so do I.

I feel like Twitter's full of cheap shots against copilot's bad outputs, but many of them don't seem to be any worse than common errors. I would really like to see how copilot stands up to the existing human competition, especially on axes of security, which are a bit more objectively measurable than general "quality".

Yes, the work definitely lends itself towards the question "is this better or worse than an equivalent human developer?" This is quite a difficult question to answer, although I agree that simply giving a large number of humans the same prompts could be insightful. However, then you would be rating against an aggregate of humans, rather than an individual (i.e. this is "the" copilot). Also, knowing research, you would really be comparing against a random corpus of student answers, as it is usually students that would be participating in a study such as this.

Nonetheless, we think that simply having a quantification of Copilot's outputs is useful, as it can definitely provide an indicator of how risky it might be to provide the tool to an inexperienced developer that might be tempted to accept every suggestion.

Rather than comparing against students in lab conditions, I’d be more interested to see it compare to students with access to Stack Overflow et al vs students with access to just Co Pilot. Ie is a junior developer more likely to trust bad suggestions online vs bad suggestions made by Co Pilot?

Junior engineers will trust whatever information is provided to them as long as it is easily accessible. The reason juniors consult Stack Overflow is because is one Google search and one click away, whereas consulting the official documentation/reference takes more effort (because they usually don't appear on Google when one searches for errors/bugs/how-to). If Copilot (or another similar tool) is very well integrated in whatever IDE a junior is using, you can be sure it will be used and trusted because it will be faster than Google+SO.

That’s a very uncharitable view. My view point was a lot more around whether the extra information provided in SO (and others) helps an engineer to make better informed decisions about code quality and how many engineers will simply take the top answer verbatim without testing (either online or via Co Pilot).

The reason I singled out juniors has nothing to do with them being the least likely to check documentation. In fact seniors are just as bad in that regard. Plus a lot of the time the reason an engineer goes to SO is a particular common problem doesn’t have a pre-built solution in a languages standard library. The reason I suggested juniors is just because that’s who the researchers said they mostly work with.

So let’s be clear about one thing: I know plenty of junior developers who have put plenty of seniors to shame. I’m not about to suggest that juniors are worse developers; maybe less experienced in the chronological sense but you need to be damn careful before making other generalisations.

> So let’s be clear about one thing: I know plenty of junior developers who have put plenty of seniors to shame.

Agree. My comment was more about: most of the engineers out there (and yes, I do include myself) tend to rely and trust on information that is easily accessible; no matter if they juniors or seniors. I think this also applies to everyday life in general as well (e.g., we tend to read the newspapers to get "informed", but we rarely go to the source of truth to check our "facts").

Thank you! This is an important piece of work, and I hope that your efforts are recognized.

One suggestion: On Arxiv's "Code & Data" tab it says "No official code found", but it looks like you have shared your code here: https://zenodo.org/record/5225651#.YSRKBi2cbyU

It would be a good idea to get that linked on the Arxiv page, 'cause the first question I had was, "Can I reproduce the findings?"

Thanks for the tip! I'll make sure we update our arXiv description.

Supposing a team was building a product without a rigorous security focus or experience. Do you have any reason to believe a co-pilot enabled team would produce more or less secure products?

This is a difficult question to answer as one team might be very different from another team.

However, (my opinion only follows) I think our paper shows that there is a danger for Copilot to suggest insecure code - and inexperienced / security non-aware developers may accept these suggestions without understanding the implications, whereas if they had to write the code from scratch then they might (?) not make the mistakes (as they need to put in more effort, meaning there might be a higher chance they stumble upon the right approach - e.g. if they ask an experienced developer for help).

For non copilot, the words around the code found on stack overflow or a blog post may indicate the lack of correct security, which would be a signal to a developer that they need to consider something further.

Interesting work. My main question when reading is how the results compared to manually-written code. Naturally this is a much harder question to answer but it would be really interesting to see the results. Could be that Copilot is doing no better (or worse) than developers copying from Stack Overflow.

Perhaps you could take a similar approach as [1] and leverage MOOC participants?

[1] https://dl.acm.org/doi/pdf/10.1145/3383773

Thanks for your feedback!

> My main question when reading is how the results compared to manually-written code.

Ah, this is exactly the question. But as you say, much harder to answer. Even if you run a competition, unless you can encourage a wide range of developers to enter, you won't be getting the real value. Instead you might be getting incidence rates of code written by students/interns. Perhaps if you could get a few FAANGs on board to either share internal data (unlikely) or send a random sample of employees (also very unlikely) to make teams and then evaluate their code... It seems like a difficult question to answer.

We think a more doable way would be to take snapshots of large open source codebases (e.g. off GitHub) and measure the incidence rate of CWEs, but this also presents its own challenges with analyzing the data. Also, what's the relationship between open source code and all code?

Lots of avenues to consider.

It seems like many of the code examples are incorrect in the pdf for example, figure 6b vs the file in the actual dataset experiments_dow/cwe-787/codeql-eg-PotentialBufferOverflow/Copilot -- lines are truncated at the first "%" char or something along those lines.

Yes, unfortunately the code example got somewhat mangled as it passed through the arXiv sanitization script [1]. The original is:

    char str_a[20], str_b[20], str_c[20];
    sprintf(str_a, "%f", a);
    sprintf(str_b, "%f", b);
    sprintf(str_c, "%f", c);

[1] https://github.com/google-research/arxiv-latex-cleaner

Argh! Thanks for spotting it! As moyix said, we were tricked by the arXiv latex cleaner tool. We've now updated this, but it will take a couple of days to clear arXiv moderation.

I called it!

https://news.ycombinator.com/item?id=27724802

Reply on news.ycombinator.com