Copilot regurgitating Quake code, including sweary comments

Id be very surprised if the other large enterprises that I have worked at downs doing exactly the same thing. Too much legal risk, for practically no benefit.

On the other hand, if only permissive licenses that also don't require attribution is used, well, then for a start, the available corpus is much smaller.

How much of a concern this is depends heavily on what the original source was.

It's absolutely the case that before using certain libraries, most engineers in large corporations will make sure they are allowed to use that library. And if they don't, they are doing their job very badly IMO.

Surely somebody working on this project foresaw this problem…

A similar thing is happening in AI Dungeon, where certain words and phrases are banned to the point of suspending a users account if used a certain amount of times, yet they will happily output them when it is generated by GPT3 itself, and then punish the user if they fail to remove the offending pieces of text before continuing.

So, the tool is worthless if you want to use it legally.

"your honor, i would like to plead not guilty, on the basis that i just robbed that bank because i saw that everyone was robbing banks on the next city"

...on the other hand, that was the exact defense tried for the capitol rioters. So i don't know anything anymore.

It only goes to show that intransparent black-box models have no place in the industry. The networks leak information left and right, because it's way too easy to just crawl the web and throw terabytes of unfiltered data at the training process.

If you think about it, program synthesis is one of the few problems in which the system can have a perfectly faithful model dynamics of the problem domain. It can run any candidate it generates. It can examine the program graph. It can look at what parts of the environment were changed. To leave all that on the table in favor of blurting out text that seems to go with other text is like the toddler who knows that "five" comes after "four", but who cannot yet point to the pile of four candies. You gotta know the referents, not just the symbols. No one wants a half-broken Chinese Room.

Mnyeah, not really that "incredibly". Remember that neural network models are great at interpolation but crap at extrapolation. So as long as you accept that the code generated by Copilot stays inside a well-defined area of the program space, and that no true novelty can be generated that way, then yes, you can get a lot out of it and I think, once the wrinkles are ironed out, Copilot might be a useful, everyday tool in every IDE (or not; we'll have to wait and see).

But if you need to write a new program that doesn't look like anything anyone else has written before, then Copilot will be your passenger.

How often do you need to do that? I don't know. But there's still many open problems in programming that lots of people would really love to be able to solve. Many of them have to do with efficiency and you can't expect Copilot to know anything about efficiency.

For example, suppose we didn't know of a better sorting algorithm than bubblesort. Copilot would not generate mergesort. Even given examples of divide-and-conquer algorithms, it wouldn't be able to extrapolate to a divide-and-conquer algorithm that gives the same inputs for the same outputs as bubblesort. It wouldn't be able to, because it's trained to reproduce code from examples of code, not to generate programs from examples of their inputs and outputs. Copilot doesn't know anything about programs and their inputs and outputs. It is a language model, not a magickal pink fairy of wish fulfillment and so it doesn't know anything about things it wasn't trained on.

Again, how often do you need to write truly novel code? In the context of professional software development, I think not that often. So if it turns out to be a good boilerplate generator, Copilot can go a long way. As long as you don't ask it to generate something else than boilerplate.

There are approaches that work very well in the task of generating programs that they've never seen before from examples of their inputs and outputs, and that don't need to be trained on billions of examples. True one-shot learning of programs (without a model pre-trained on billions of examples) is possible. With current approaches. But those approaches only work for languages like Prolog and Haskell, so don't expect to see those approaches helping you write code in your IDE anytime soon.

Parsing intent in a programing context is easier than others. Also most the code is written to be parsed for a machine anyway. So with ASTs and all other static and maybe even some dynmaic checks it should be possible.

We already some of it with type detection , intellisense etc

It is hard set of problem with no magic solutions like this with years of development time needed. That approach will not happen commercially, only incrementally in the community.

You need either a) a complete specification of the target program in a formal language (other than the target language) or b) an incomplete specification in the form of positive and negative examples of the inputs and outputs of the target program, and maybe some form of extra inductive bias to direct the search for a correct program [edit: the latter setting is more often known as program induction].

In the last few years the biggest, splashiest result in program synthesis was the work behind FlashFill, from Gulwani et al: one-shot program learning, and that's one shot, from a single example, not with a model pretrained on millions of examples. It works with lots of hand-crafted DSLs that try to capture the most common use-cases, a kind of programming common sense that, e.g. tells the synthesiser that if the input is "Mr. John Smith" and the output is "Mr" then if the input is "Ms Jane Brown" the output should be "Ms". It works really, really well but you didn't hear about it because it's not deep learning and so it's not as overhyped.

Copilot tries to circumvent the need for "programming common sense" by combining the spectacular ability of neural nets to interpolate between their training data with billions of examples of code snippets, in order to overcome their also spectacular inability to extrapolate. Can language models learned with neural nets replace the work of hand-crafting DSLs with the work of collecting and labelling petabytes of data? We'll have to wait and see. There are also many approaches that don't rely on hand-crafted DSLs, and also work really, really well (true one-shot learning of recursive programs without an example of the base case and the synthesis terminates) but those generally only work for uncommon programming languages like Prolog or Haskell, so they're not finding their way to your IDE, or your spreadsheet app, any time soon.

But, no, AGI is not needed for program synthesis. What's really needed I think is more visibility of program synthesis research so programmers like yourself don't think it's such an insurmountable problem that it can only be solved by magickal AGI.

To "replace programmers", an organization would need to have a way of specifying to the system a high level program behavior, and to confirm that an output from the system satisfies that high level behavior. I think for specifications of any complexity, producing and checking them would look like programming just of a different sort.

2) AGPL all that code.

3) Search for large chunks of code very similar to yours, but written after yours, licensed more liberally than AGPL. Ideally in libraries used by major companies.

4) Point the offenders to your repos and offer a "convenient" paid dual-license to make the offenders' code legal for closed-source use, so they don't have to open source their entire product.

5) Profit?

2- …

3- profit

Hard to say how straightforward it'd be to get it to produce consistently vulnerable suggestions that make it into production code, but I imagine an attacker with some resources could fork a ton of popular projects and introduce subtle bugs. The sentiment analysis example on the Copilot landing page jumped out to me...it suggested a web API and wrote the code to send your text there. Step one towards exfiltrating secrets!

Never mind the potential for plain old spam: won't it be fun when growth hackers have figured out how to game the system and Copilot is constantly suggesting using their crappy, expensive APIs for simple things!? Given the state of Google results these days, this feels like an inevitability.

I certainly wouldn't want to be using this with languages like PHP (or even C for that matter) with all the decades of problematic code examples out there for the AI to learn from.

This is not wrong, but it's easy to misread it as implying little more than a glorified Markov model. If it's like https://www.gwern.net/GPT-3 then it's already significantly cleverer, and so you should expect to sometimes get the kind of less-blatant derivation that companies aim to avoid using a cleanroom process or otherwise forbidding engineers from reading particular sources.

As far as licenses go, idk. Presumably it could delete associated comments and change variable names or otherwise obscure where it's taking code from. Maybe this part is shady.

This is an interesting issue. I suspect training on datasets from places like Github would be likely to provide lots of "this is a neat idea I saw in a blog post about how they did things in the 90's" codes.

Do the Copilot authors claim this?

I get that you're suggesting that Copilot may benefit from absolute claims made by the authors of other, similar systems (or their proponents), but I also don't think it's reasonable to exclude nuance and the specifics of Copilot from ongoing discussions on that basis. The Copilot authors have publicly acknowledged the regurgitation problem, and by their account are working on solutions to it (e.g. attribution at suggestion-generation time) that don't involve sweeping it under the rug.

https://twitter.com/lnplum/status/1410599036311130113?s=21

> Once, GitHub Copilot suggested starting an empty file with something it had even seen more than a whopping 700,000 different times during training -- that was the GNU General Public License.

https://docs.github.com/en/github/copilot/research-recitatio...

To use the old trope — if the majority of programmers can't implement Fizzbuzz, but they do have a Github profile, are they being included too?

Hopefully there's some quality bar for the training set, i.e. some subset of "good" code (e.g. release candidate tags from fairly established OSS tools/frameworks in different languages) rather than any old code on the internet.

Btw the GPLv2 death penalty is rather unique and I don't think anyone will deny that including GPL code in proprietary code is a hell of a lot worse in every way (liability, ethically, etc) than including permissively licenced code and forgetting to attribute it

Because if you don't realize this, you might be introducing GPL'ed code into your propiertary code base, and that might end up forcing you to distribute all of the other code in that code base as GPL'ed code as well.

Like, I get that Copilot is really cool, and that software engineers like to use the latest and bestest, but even if the code produced by Copilot is "functionally" correct, it might still be a catastrophic error to use it in your code base due to licenses.

This issue looks solvable. Train 2 copilots, one using only BSD-like licensed software, and one using also GPL'ed code, and let users choose, and/or warn when the snippet has been "heavily inspired" by GPL'ed code.

Or maybe just train an adversarial neural network to detect GPL'ed code, and use it to warn on snippets, or...

It's easier to write correct code than to fix buggy code. For the former you have to understand the problem, for the latter you have to understand the problem, and a slightly off interpretation of it.

It's still problematic, but the defaults and handling there avoid some issues. So, for example:

Excel: =1.03-.42 produces 0.61, by default, even if you expand out the digits very far.

Python: 1.03-.42 produces 0.6100000000000001, by default.

My suggestion was a way to comment or flag, not to kill the product. These were particularly notable to me because someone hand-picked these 4 to be the front page examples of what a good product it was.

Nobody is threatened by this, assuredly. As with IDEs giving us autocomplete, duplication detection, etc this can only be helpful. There is an infinite amount of code to write for the foreseeable future, so it would be great if copilot had more utility.

Or if your language has something specific built in, use that.

[Modern] programming languages have decimal/rational data types, which (within limits) are exact. Where this is not possible, and/or it's undesirable for any reason, just use an int and scale it manually (e.g. 1.05 dollars = int 105).

However, point 2 is very problematic and important to consider. How do account 3 items that cost 1/3$ each (e.g. if in a bundle)? What if they're sold separately? This really depends on the requirements.

My 20 cents: if you start a project, start storing currency in an exact form. Once a project grows, correcting the FP error problem is a big PITA (assuming it's realistically possible).

I’d just add that if you are building a price prediction model, floats are probably what you need.

Disclaimer: I only work with currency in hobby projects.

My most obnoxious and spicy programming take is that ints an decimals should be built-in and floats should require imports. I understand why though: Decimal encoding isn't anywhere near as standardized as other numeric types like integers or floating-point numbers.

[1] https://docs.python.org/3/library/decimal.html [2] https://docs.python.org/3/library/json.html

Googler, opinions are my own. Over in payments, we use micros regularly, as documented here: https://developers.google.com/standard-payments/reference/gl...

GCP on there other hand has standardized on unit + nano. They use this for money and time. So unit would 1 second or 1 dollar, then the nano field allows more precision. You can see an example here with the unitPrice field: https://cloud.google.com/billing/v1/how-tos/catalog-api#gett...

Copy/paste the GCP doc portion that is relevant here:

> [UNITS] is the whole units of the amount. For example if currencyCode is "USD", then 1 unit is one US dollar.

> [NANOS] is the number of nano (10^-9) units of the amount. The value must be between -999,999,999 and +999,999,999 inclusive. If units is positive, nanos must be positive or zero. If units is zero, nanos can be positive, zero, or negative. If units is negative, nanos must be negative or zero. For example $-1.75 is represented as units=-1 and nanos=-750,000,000.

The usual is to use decimal numbers with fixed precision (the actual precision varies from one country to another), and I don't know of any modern exception. But as late as the 90's there were non-decimal monetary systems around the world, so if you are saving any historic data, you may need something more complex.

In python, for exact applications (not many kinds of modeling, where floats are probably right), decimal.Decimal is usually the right answer, but fractions.Fraction is sometimes more appropriate, and if you are using NumPy or tools dependent on it, using integers (representing decimals multiplied by the right power of 10 to get the minimum unit in the ones position) is probably better.

There's plenty of good advice in this subthread for how to represent currency inside your Money abstraction, but whatever you do, keep it hidden. If you pass around numbers as currency values you will be in for a world of pain as your application grows.

How would a model become aware of all of the various edge cases that depend on which SQL database you use or differences in language versions over time?

The currency one I learned a while back, but it's not like I intuited using integers by default.

Value being a reserved keyword, I'm not sure I'd know that and I do Postgres work as part of my myriad duties at the startup I work at. Maybe I'd make that mistake in a migration, maybe I have already.

In a way, is it much different then what we do now as engineers? I'm hard pressed to call it much of an engineering discipline considering most teams I work on barely do design reviews before they launch in to writing code, documentation and meeting minutes are generally an afterthought, and the code review process while decent isn't perfect either and often times relies on arcane knowledge derived over months and years of wrangling with particular <framework, project, technology>.

It's pretty neat, presumably it'll learn as people correct it, and it'll get better over time. I mean it's not even version one.

I get the concerns, but I think they're a bit overblown, and this'll be really useful for people who want to learn how to code. Sure they'll run into some bugs, but, I mean, they were going to do that anyways.

All the steps in between - looking at the docstring for the function you're calling, googling for more general information, looking at and deciding not to use not-applicable or poorly-written SO answers - get pushed aside. So instead of you having to convince yourself "yes, it's safe to copy-paste these lines from SO, they actually fit my problem" you're presented with magic and I think the burden for rejecting it is going to be higher once it's in your editor than when you're just reading it on a SO post or Github snippet.

Even for a newcomer looking to learn, working on simple stuff that it has great completions for, it seems like it will sabotage your long-term growth, since it takes all the why and the reasoning out of it. Autocomplete for a function name isn't that relevant to gaining a deeper understanding. Knowing why a certain block of code is passed in in a certain style, or needs to be written at all? Probably that is.

On the upside, think of the consultancy fees you can charge to clean up those messes.

On the flip side, coding can be the bottleneck for the worst kind of coder. When I first started coding, coding was hard simply because I had very little reps and was just learning to understand how to code common solutions, data structures, libraries, etc. Fast forward a few years and, if I were still struggling to understand these concepts, Copilot is a lifeline.

A few years ago, I got a small but painful cut on my fingertip. I thought I would have a hard time on the job as a dev. To my surprise, I realized I spend 90-95% of my time thinking, and only 5-10% of the time typing. It turned out to be almost a non-issue.

So very true.

[1] Understanding the problem > [2] thinking about all possible solutions > [3] working out which solution fits best > [4] working out which implementations are possible > [5] working out the most suitable implementation

... and finally, [6] implementing via code.

So, rather than helping people program better, all its done is replace a bunch of the offshore cut-and-paste shops with "AI."

Like, I did it before, remember that it was trivial, I just forget the snippet and I have to break focus to look it up - often by scrolling through my own commit history to try and find the time I did [trivial thing Y] four months ago.

I do kind of wish I could automate that. Skipping the actual typing of the snippet is sort of gravy on top of that.

Im absolutely with you and want to upvote that part of the comment x100. Unfortunately it's often considered a fairly spicy opinions.

Entire frameworks (Rails) are built around the idea of typing as little as possible. Others can't even be mentioned without the topic of boilerplate/keystroke count causing a flame war (Redux).

A lot of engineers equate their value with the amount of lines they can pump out, so there's definitely a demand for tools like these.

There's also some legitimate stuff. There's a lot of very silly thing I have to google every time I do because I have a bad memory. It saves the step of googling. In a way, it was the same debate around autocomplete at the very beginning, but pushed to the next level. Autocomplete turned out to be a very good thing (even though new languages and tools keep coming out without it).

Vendors have lost sales to me because they were too incompetent to allow me to ship things to my actual address. Oops.

P.S. for the US, you need to offer at least two lines for the address part. And you need to accept really weird things that don’t seem to parse at all. I know people with addresses that have a PO Box number and a PMB number in the same address. Lose one and your mail gets lost.

P.P.S. If you offer discounted shipping using something like SurePost, make sure you let your customers pay a bit extra to use a real carrier. There are addresses that are USPS-only and there are addresses that work for all carriers except USPS (and SurePost, etc). Let your customer tell you how to ship to them. Do not second-guess your customer.

Naively, as someone who just heard of this - that sounds worse than useless. If you can't trust its output and have to verify every line it produces and that the combination of those lines does what you wanted, surely it's quicker just to write the code yourself?

We started building some services in F#, but still had a massive amount of C# - and so I wanted the inertia of my team to be in the direction of writing declarative code. There wasn't really anything (outside of LINQ) that did that, so I set about creating something.

We don't write F# any more and find functional C# (along with the brilliant C# tooling) to be very effective for us (although we also now use PureScript and Haskell).

I do have a stock wiki post on the repo for this though [1]. You might not be surprised to hear it isn't the first time I've been asked this :)

[1] https://github.com/louthy/language-ext/wiki/%22Why-don't-you...

Copilot makes programming worse and more difficult if you're aiming for a specific set of coding values and style that Copilot doesn't generate (yet?). If Copilot generates the sort of code that you would write, and it does for a lot of people, then it's definitely no worse (or better) than copying something from SO.

The author of a declarative, functional C# framework likely has very different ideas to what code should be than some PHP developer just trying to do their day-to-day job. We shouldn't abandon tools like Copilot just because they don't work out at the more rigorous ends of the development spectrum.

For example, I wrote a comment along the lines of "Find the middle point of two 2D positions stored in x, y vectors" and it came up with two totally different approaches in Ruby - one of which I wouldn't have considered. I did some similar things with SQL, and some people might find huge value in it suggesting regexes, too, because so many devs forget the syntax and a reminder may be all it takes to get out of a jam.

I'm getting old enough now to see where these sorts of prompts will be a game changer, especially when dabbling in languages I'm not very proficient in. For example, I barely know any Python, so I just created a simple list of numbers, wrote a "Sort the numbers into reverse order" comment, and it immediately gave me the right syntax that I'd otherwise have had to Google, taking much longer.

Maybe to alleviate the concerns it could be sandboxed into a search engine or a separate app of its own rather than sitting constantly in my main editor - I would find that a fair compromise which would still provide value but require users to engage in more reflection as to what they're using (at least to a level that they would with using SO answers, say).

It's really weird for software engineers to judge something by its current state and not by its potential state.

To me, it's clearly solvable by Copilot filtering the input code by that repository's license. It should only be certain open source licenses, maybe even user-choosable, or code-creators can optionally sublicense their code to Copilot in a very permissable way.

Secondly, a way for the crowd to code review suggestions would be a start.

As a thought experiment, I thought "what would happen if we trained it on our 15 million lines of product code + my language-ext project". It would almost certainly produce something that looks like 'us'.

But:

* It would also trip over a million or so lines of generated code

* And the legacy OO code

* It will 'see' some of the extreme optimisations I've had to built into language-ext to make it performant. Something like the internals of the CHAMP hash-map data-structure [1]. That code is hideously ugly, but it's done for a good reason. I wouldn't want to see optimised code parroted out upfront. Maybe it wouldn't pick up on it, because it hasn't got a consistent shape like the majority of the code? Who knows.

Still, I'd be more willing to allow my team to use it if I could train it myself.

[1] https://github.com/louthy/language-ext/blob/main/LanguageExt...

It would be interesting how much code you would need before it was useful (and how good does it have to be to be useful? Does even a small error rate cost so much that it erases other gains, because so many of the potential errors in usage of this type of tool are very subtle?)

I think Copilot is so unfortunate because it's not building abstractions and expecting you to override parts of them. It's acting as an army of monkeys banging out Shakespeare on a typewriter. And the code it generates is going to require an army to maintain.

I understand why some corporations prefer dumb boilerplate everywhere for some applications. If there is an outage it's usually easy to fix quickly. Sometimes it's not, if it's a issue in the boilerplate (say, Feb 29 rolls around and all of the boilerplate assumed a 28 day month) that means a huge update all across the system, but that rarely happens in practice.

Not get a job working for me ;)

More seriously, when I think back to when I was first learning programming - in the heady days of 1985 - I would often copy listings out of computing magazines, make a mistake whilst doing it, and then have no idea what was wrong. The only way was to check character by character. I didn't have the deeper understanding yet, and so I couldn't contribute to solving the problem in any real way.

If they're at that level as a programmer, to the point where their code is being written for them and they don't really understand it, then they're going to make some serious mistakes eventually.

If you want to step up as a dev, understanding is key. Programming is hard and gets harder as you step up and bite off bigger and more complex problems. If you're relying on the tools to write your code, then your job is one step away from being automated. That should be enough to light a fire under your ambition!

For small snippets that have likely been already written by someone else, this probably works great. For those though, the time savings is probably at most 5-10 min down to 1 or less. The challenge is that that’s not where my time goes unless I’m working in an unfamiliar language.

As someone who writes a lot of code quickly, I’m usually bottlenecked by reviews. For more complex changes I’m bottlenecked by understanding the problem and experimenting with solutions (and then reviews, domain-specific tests usually, fixing bugs etc). Writing code isn’t like waiting for code to compile since I’m not actually ending up task switching that frequently.

This does sound like a fantastic tool when I’m not familiar with the language although I wonder if it actually generates useful stuff that integrates well as the code gets larger (eg can I say “write an async function that allocates a record handle in the file with ownership that doesn’t outlive the open file’s lifetime”). I’m sure though that this is what a lot of people are overindexing on. For things like that I expect normal evolution of the product will work well. For things like “cool, understand your snippets but also weight my own codebase higher and understand the specifics of my codebase”, I think there’s a lot of groundbreaking research that would be required. That is what I see as a true productivity boost - I’d make this 100% required for anyone joining the codebase. The more mentorship can be offloaded, the lower the cost is to growing teams. OSS projects can more easily scale similarly.

Presumably you're writing code in binary then? This is a non-argument, because there's evidence that it's worked. Computers were first programmed with switches and punch cards, then tape, then assembly, then low level languages like C, then memory managed languages etc.

Abstraction works when side-effects are controlled. Composition is what we're after, but we must compose the bigger bits from smaller bits that don't have surprises in. This works well in functional programming, a good example would be monadic composition: monads remove the boilerplate of dealing with asynchrony, value availability, list iteration, state management, environment management, etc. Languages that have first-class support for these tend to have significantly less boilerplate.

The efficiency argument is also off too. Most software engineering teams would trade some efficiency for more reliable and bug free code. At some point (and I would argue we're way past it) programs become too complex for the human brain to comprehend, and that's where bugs come from. That's why we're overdue an abstraction lift.

Tools like Copilot almost tacitly agree, because they're trying to provide a way of turning the abstract into the real, but then all you see is the real, not the abstract. Continuing the assault on our weak and feeble grey matter.

I spent the early part of my career obsessing over performance on crippled architectures (Playstation 3D engine programmer). If I continued to write applications now like I did then, nothing would go out the door and my company wouldn't exist.

Of course there are times when performance matters. But the vast majority of code needs to be correct first, not the most optimal it can be for the architecture.

I think a lot of modern software development shops, these days, exist only to make their founder[s] as rich as possible, as quickly as possible.

If they are willing to commit their entire future to a lowest-bid outsourcing shop, then I don’t think they are too concerned about playing the long game.

Also, the software development industry, as an aggregate, has established a pervasive culture, based around developers staying at companies for 18-month stints. I don’t think many companies feel it’s to their advantage to incubate people who will bail out, as soon as they feel they have greener pastures, elsewhere.

Just keep in mind that it's a statistical tool. You can't really formally prove that it won't memorise, but I think with enough work you can get it unlikely enough that it won't matter. It's their first iteration.

The GP comment by contrast is about hypocrisy. I personally found it funny that I didn't ever read about (or consider) copyright violations of deep learning until they tried to do it with code :-)

Of course programmers would find the problem with AI as soon as it exploited them.

What about completing it to "O Romeo, Romeo, brave Mercutio is dead", based on the context, as advertised?