Is GitHub Copilot a blessing, or a curse?

There's an interesting section that talks about what you want from a pair programmer - questioning your assumptions, spotting errors, debating design with you, etc, and how Copilot doesn't do this, but instead just spits out the code it thinks you want. That made me think that, when using Copilot, the human is actually the copilot. You're the one spotting bugs, picking different designs, and so on.

I also think the discussion around searching versus autocomplete is interesting. What we need is something that autogenerates a Google search based on your code context and shows you some promising results. I'm imagining a pane in VS Code getting populated with Stack Overflow answers whenever I slow down.

Copilot was made from stealing code on Github, ignoring the licenses set on repos such as e.g GPLv2, using AI as a trick to license-launder code.

Copilot has announced their plans to become a paid service.

So this product that would not be possible without public, open-source code will itself be non-public, closed-source, closed-data. It is extracting value from the commons and funneling it to a private company.

> The code Copilot writes is not very good code.

This has been my experience as well. Some of the early promo material for Copilot showed someone writing a function signature with docs, and having Copilot write the entire body. This rarely works for me, except for fairly trivial functions. This is also not how I normally write code.

However, where Copilot has been rather good is offering intelligent "tab completion", where it makes a suggestion for the next line of code. In particular, it seems very good at figuring out the structure of my code and making sensible suggestions.

I suspect that if anyone ends up using Copilot for the long term, they will (as a human operator) need to learn and gain intuition about how to use Copilot's strengths and avoid its weaknesses.

> In this case, I felt like the Copilot approach would be worse for both experienced and beginner programmers. Experienced programmers would need to spend time studying the various options proposed, recognize that they don’t correctly solve the problem, and then would have to search online for solutions anyway. Beginner programmers would likely feel like they’ve solved the problem, wouldn’t actually learn what they need to understand about limitations and capabilities of regular expressions, and would end up with broken code without even realizing it.

This is similar to what I thought—replace "experienced" and "beginner" with "curious" and "lazy"—when I first read the tagline of GitHub Copilot on its landing page:

> Skip the docs and stop searching for examples. GitHub Copilot helps you stay focused right in your editor.

We need docs and examples for things that we aren't familiar with. Skipping them means that we blindly trust Copilot to generate correct code. And even if the generated code is correct, we still need to understand why and how it's correct.

One use case I see the best fit for Copilot is for generating boilerplates that we are already familiar with and understand.

Having used Copilot for a couple of hours on a Typescript/React codebase, I've encountered a very wide range of results.

Sometimes, the model correctly guesses exactly what I'm trying to write, and gives me non-trivial contextually correct suggestions. More than once I've been impressed enough to pause and take a screen recording of what it generated.

Other times, it's a complete mess that competes with the local language server for autocomplete corrections.

Among the misses, the most glaring issues have been when it's suggested code that violates static typing constraints. The model outputs the right contextual code, but uses variables with slightly different spelling/wording/capitalization.

This is to be expected of course, as Copilot doesn't integrate with the language server in VSCode. However it does show the limits of building the model without a native understanding of the AST it's working with.

Whenever a new generative language model is released and discussed on Hacker News, there's always the comment that "it's worthless because it doesn't always generate coherent text" which I've always disliked because it's unfairly reductive as progress in this field is iterative.

However, in the case of Copilot, there has to be a much higher signal-to-noise ratio in terms of generated text than creative language models like GPT-2/GPT-3 to use it as a pair programmer where the overhead in working with the model is worth the time saved via the model.

Now, imagine you're sitting with your boss, the average typically incompetent project manager or lead programmer ...

Them: type type type ... accept accept ... save (unaware or choosing not to understand the meaningless uncompilable garbage that's been generated)

Them: "There, that's the xxxx function. You'll have the rest done in 2 days, won't you!"

How does copilot fair in a language where correctness or at least well soundness is more transparent, for example Haskell (eg at least making sure all the types are correct, actual logic on the other hand seems harder, but maybe the declarative style helps)? I imagine in a less strict language you can generate garbage more easily.

Writing code is, uh, not remotely the hardest part of software development.

Maintaining it is.

So far it's been worse than useless for me, the suggestions it offers don't even make sense. The existing code completion suggestions in VSCode for TypeScript are better.

The article mentions TabNine as an alternative. I think TabNine’s approach of providing one-line autocomplete suggestions is a nice middle ground where you’re still in charge of writing code but can do it faster with AI assistance.

Unlike Copilot you don’t have to give it instructions - it guesses what you’re trying to do from context. And since it works on a single line of code at a time, providing multiple completions which you can glance at and decide whether to use or ignore, you the programmer are still in charge of planning the function and making choices.

I have found that the cognitive load required to evaluate its suggestions is more of a distraction from what I'm doing--in sharp contrast to conventional autosuggest which is predictable.

That said, I haven't turned it off yet. It's very fun on those occasions when it suggests correctly.

It's mentioned towards the end of the blogpost, but I think the real blessing in disguise might be the ability to identify similar code from elsewhere on Github and linking to it. Too many libraries have you wading through pages of documentation in search of a single usage example.

Some other ideas:

- Look for libraries with similar code and suggest them

- Deobfuscating minified variable names? Or just suggesting better ones for your source code

- Speculatively running things. If the model predicts importing a certain library, start indexing it before I even type it

- Estimate how long it thinks a block of code will take to write, based on the predicted code + past usage patterns (like the kindle reading progress estimate)

Just spitballing but all of these things feel more helpful and pair programming-y than trying to fabricate code out of the blue.

Feels like its a magical thing for me honestly. although I'm never waiting for suggestion while coding, sometimes the suggestions just slipped through and there is the OMG moment when it completes your thought.

1 thing though, tab for auto complete still does not work for me.. ever

> blessing or a curse?

It's just a footnote.

I think it's getting talked about more than it deserves to be.

Copilot is going to really rock the numbers on lucrative unhose-your-sucky-codebase consulting gigs. I'd feel gleeful about that but those gigs are the definition of soul-sucking so I guess still no free lunch.

Well if they ever get it right I guess I could always become an electrician (this is not a dig on electricians, it seems like a fun trade, that or under water welding)

Hi all, Jeremy Howard here, author of the article. I'm happy to answer any questions you have about language models and program synthesis, Copilot, etc.

I do not know. I think best copilot are... - services - tested software libraries

All the time I wrote the code in the last years, I ended up discovering someone ALREADY have done it, often in a better way (not always).

There is a nice quote who says "Who did not know Unix is doomed to rewrite it..." I think the same is with Java/Python/... code.

The value of copilot is in ideating solutions not code correctness.

Given it's far from getting syntax and semantics right, it should probably be best to position this product as a tool that promotes creativity not productivity.

> Given this, why does Copilot write such crappy code?

> The reason is because of how language models work. They show how, on average, most people write. They don’t have any sense of what’s correct or what’s good. Most code on GitHub is (by software standards) pretty old, and (by definition) written by average programmers.

Does Copilot apply any project quality filters before training? A small minority of public repos on GitHub are awesome projects that are popular and maintained, while the rest are forgotten experiments, side projects and interview answers.

I signed up for Copilot’s waiting list yesterday and I am looking forward to trying it, with some skepticism.

That said, I find OpenAI’s beta APIs based on GPT-3 to be very useful (so useful that I immediately released new versions of my Common Lisp and Clojure books with examples). These APIs are so inexpensive to use and useful, that they really should be part of many developers’ standard stack: different than Copilot that seems like a gimmick (but to be fair, I haven’t tried it yet).

what i don't get, is how a copy of (parts of) whatever's code, (or worse, a concoction of them), without any context, would solve anything but import the (unfound yet) errors as well, and introduce new errors.

Hah, just make it autocommit, and will be much like with self-driving cars - human takes the blame for computer's mistakes (= algorithm/ programmer/ knowledgebase), as well as hir own, and can't/won't do anything about it..

I’m not into copyleft, but is there already an analogue to AGPL that explicitly prohibits the usage of the code to train AI models? Shouldn’t be that hard to do, right?

There's a need to answer the question (precedent, new technology, huge potential / market interest, and ethical extremes in one way or another) but there's an alternative to giving in to that urge. Perhaps we're better off leaving it as-is. We can evaluate about this in some later stage of the process.

Neither of both. It's completely irrelevant, as you cannot and may not use it. You cannot use it for production code as it's merely a toy for managers.

And you may not use it as the license of the generated code is unclear at best, GPL at worst.

His conclusion to the question in title : "I don't know."

I hope they release a fine tuner for Codex. Jeremy is right that whole problem solving is likely overreach right now. But can I train the model to do a Rails version upgrade?

It will depend on the companies and the price of Copilot. Then they will replace expensive experienced programmers with cheaper inexperienced ones and Copilot.

I enjoyed the appropriately new phrase mentioned in the article “… gives a start of the art result…”

In all literature there are strong themes of mistaking the blessing for a curse, and vice versa.

I like the idea, it's like a manual vs automatic clutch in cars. You are still the driver.

[Warning: long comment ahead]

I haven't used Copilot yet (I'm not coding in any of the languages it covers) but I've tried to understand it from the few articles (most critical) posted about it on the internet and from my understanding of program synthesis and languae modelling. I don't fully trust lay users to be able to exercise a machine learning model comprehensively enough to precisely understand its pros and cons, but I think I've formed some intuition about how Copilot works from the demonstrations I've seen. Here's my thoughts:

a) As TFA points out, Copilot was trained on _all_ code on github (for select languages). "All" includes good code and bad code, code with bugs and code with, er, fewer bugs, code that does what it should and code that doesn't. The result is that Copilot models _all_ that code, regardless of whether it's got bugs or not.

b) Copilot is (based on) a language model that predicts the next sequence of tokens, not a discriminator of good/bad code. That means it can generate code but it can't generate "correct" code. It doesn't have a concept of correct/incorrect code.

b.1) More generally, it's impossible to discriminate between "correct" and "incorrect" code from features (tokens, in the case of language models) alone. The correctness of code depends on the programer's intent. In program synthesis, the programmer's intent is encoded as a specification and the correctness of a synthesised program must be considered in the context of that specification. A language model could be trained to include a specification as a feature, or as an additional modality (similar to training on images and text together to generate captions for images, or images from captions). However, this is not how Copilot was trained. The upshot of all this is that Copilot can't generate "correct programs" because it doesn't know what "correct programs" are.

b.2) But wait! Copilot can generate code from a specification! From a natural language specification, in the form of inline comments given as part of a prompt. This seems to be a favourite mode of testing Copilot and its results are impressive, but there are two problems: i) natural language specifications are imprecise and get more so the longer they grow; and, ii) Copilot still has no way to determine whether the code it generates satisfies a natural language specification, or not.

b.3) For program synthesis with guarantees of correctness, a specification is not enough: a verification procedure is also necessary -and those tend to be very tricky because Halting Problem.

c) Copilot has memorised at least part of its training set. This is evident in the demonstrations of verbatim generation of known code, e.g. the Inverse Square Root function (from Quake... III Arena I think?) etc.

c.1) This observation is helpful in understanding language models in general. The assumption is often stated that a model can't possibly be memorising its training set because models are smaller on disk than their training sets. And yet it seems that large language models are memorising parts of their training sets- most likely, the ones that are more common, and therefore most likely.

c.2) This in turn means that the programs that Copilot is most likely to generate are variations of programs included in its training set and that have been memorised. Why is this most likely? Because those programs are themselves "most likely" - they are programs with the highest prior probability in the space of programs represented by Copilot's model.

c.3.1) That said, Copilot doesn't always generate memorised programs verbatim. It clearly has the ability to "mix and match" and generate programs that are not included in its training set. How does Copilot (or language models in general) achieve that? A language model is essentially a representation of a dense region of Cartesian space bounded by the elements (tokens, expressions, whole programs) of the training set. Anything within that dense geometric region, a language model can recognise and generate. Anything outside it, is down to luck.

c.3.2) To generate programs not included in its training set, Copilot interpolates between the programs in its training set and other programs in its model ("interpolates" as in "finds a point on a gradient between two programs' representations"). However, this interpolation is stochastic. Fudging very, very much, we can imagine Copilot traversing the AST of a memorised program and applying slight peturbations to that AST, at random, though according to its learned prior probabilities. This is why Copilot will generate different code each time it's run even with the same prompt, and the generated code is closer or farther away from memorised code according to its "temperature" parameter (that controls the randomness of "perturbations").

c.4) Suppose you take some program P and walk down its AST, perturbing the nodes at random, without having any way to know whether a perturbation results in correct code, incorrect code, or pure garbage. What is the expected result? The expected result is correct code, incorrect code, or garbage. Remember: Copilot can't tell which is which. So we don't know which we'll get.

d) All that said, I think that i) Copilot can make a great boilerplate generator but ii) only if people learn to use it right. So far I think most demonstrations I've seen are "holding it wrong".

d.1) As to ii) above, I think the best use-case of Copilot is to generate short code snippets, spanning at most a couple of lines, rather than entire functions. I think it's also going to work a lot better if it's allowed to complete code rather than comments (with code). See point b.2 above. The demonstrations I've seen so far seem to try to get Copilot to act as an Autopilot (sorry, couldn't resist). In that, it fails in the ways it should be expected to fail . I suspect that as users get more familiar with Copilot's abilities the best way to use it will become evident and Copilot will take its rightful place as a useful tool in the programmer's toolbox that can simplify, but not replace, day-to-day programming work.

e) Rumours of the imminent demise of the programming profession have, of course, been greately exaggerated. Wait another couple of generations, maybe. Human generations.

f) If you're interested in program synthesis and the progress that has been achieved so far in this old, but still active field of research, the following is a good recent survey of the field:

https://www.microsoft.com/en-us/research/wp-content/uploads/...

It is tempting to apply Betteridge's Law and answer no. But, perhaps the answer is yes. It probably can be a blessing, once you're used to it. And it certainly can be a curse, writing code that is incorrect, buggy, or just not understood by the programmer. It would be nice if it better could explain its reasons and add references relating to the snippets. Maybe one day. Until then, I think I'll just google stackexchange as I've been doing for so long...

Does Copilot use a lotta RAM?

A brief tangent - the only useful bit of voice recognition for me has been telling Siri to remind me about something at a day and time. Calendar events don't work well (gets inserted on the wrong calendar, can't invite attendees properly), and every other use case for voice control has been a gimmick at best that I have shut off or not used. Whenever I call any phone number, I slam the 0 button and say "operator" to bypass the horrible menus, which usually accept digits anyway as the voice control is a piece of shit. The number of "Chat Bot AI" startups that have been funded, found no usage, pivoted to something else has been astronomical, and chat bots should seemingly be one of the first use cases of useful AI.

With that said, I have found all "creative" types of AI to be similarly gimmicky and shallow. Co-pilot is like IntelliJ code-complete in more generic circumstances. It can't architect systems, interview and recruit good engineers, or do any of the things that separate true success from failure. It's yet another cool demo but ultimately useless for me.