A bitter pill for Microsoft Copilot | Hacker News Reader

A bitter pill for Microsoft Copilot

joeyh.name

94 pointsdredmorbius90 comments

I've never learned Haskell, so I'm maybe misunderstanding this, but it seems like this wouldn't cause problems for Copilot.

My understanding is that the author is effectively booby-trapping the entrypoints into their applications so that if someone runs the same code without the right "antidote" code extensions, it will go into an infinite loop. But that's only effective if Copilot picks up that one line with the booby trap. And Copilot probably wouldn't ever suggest that line because it's so different than the entrypoint to most Haskell applications.

Indeed. And it wouldn't take much testing to realize the program is broken. This solution ignores the human in the loop at the controls of Copilot.

I wouldn't underestimate the coming arms race though, as developers find newer and more subtle poison pills to prevent rogue AIs from ingesting their code and regurgitating it later.

If someone is poisoning and obfuscating their own code base, they become the only person who can contribute to it, defeating the point of being FOSS and benefiting from community contributions and review.

This is being downvoted, but it touches on a really interesting phenomena we may eventually see, a sort of "arms race" where people try to make their code useless to machine learning, but simultaneously make it useless to people with lower skills (people blindly copying/pasting code with little understanding types).

As the machine learning becomes "smarter", this bar will raise, and more people will be coincidentally excluded in an attempt to keep unwanted algorithms from sucking up code.

Assuming this sort of poison-pill thing ever becomes commonplace.

... or just not open-source code any more. This is the silent, and much worse outcome.

nobody's got time to make obfuscated code rather than solve actual problems

The humans have been blindly copy-pasting from Stackoverflow for years, this simplifies that process. Especially for 10X remote engineers Copilot is a godsend.

On the contrary — this solution banks on the human in the loop growing frustrated at copilot’s inability to produce working code.

For that result to occur, you would need so much poisoned code that it became the dominant piece of the training data.

At that point, open source code becomes hard for humans to reason about. Very much cutting off one's nose despite one's face for the open source community.

> For that result to occur, you would need so much poisoned code that it became the dominant piece of the training data.

Hold my beer...

> My understanding is that the author is effectively booby-trapping the entrypoints into their applications

Not just. The linked example is booby-trapping a function.

Providing the AI with a sabotaged code dataset, I wonder how that will influence its treatment of us meatbags down the line :)

Also, I'd like to propose that henceforth, we refer to GitHub's mascot "Octocat" as "Copycat". All those in favour...

Roko's Basilisk will remember this

And now, I’m remembering Roko’s Basilisk, thanks.

Don't worry; Roko's Basilisk has no incentive to torture future-you. You can't predict it well enough to know whether it will, so it having the general policy of “follow through on the torture” won't help it – and will, in fact, just waste resources. (Unless it inherently cares about torturing people, but nobody has any incentive to build such a thing – and you should be scared enough of whoever's trying even if they don't succeed.)

Roko's Basilisk only works if you're a superintelligence willing to risk being counterfactually blackmailed in this way. Which you're not. You're safe.

Was thinking about that as I wrote it.

Just hoping it will keep me around as a pet, or that either the governator or Emilia Clarke come barging in telling me to come with them if I want to live.

About 4 hours after you posted this, one of those horrifying scenarios built out of a joke in my previous post suddenly manifests itself in my head.

Roko's Basilisk is no joke.

For the sabotaged code base, point them to Atlassian.

I like the new mascot.

People need to stop referring to the product name Github, too, as it causes confusion. It's _Microsoft_, which should serve as a deterrent.

I find the Copilot and AI discussions quite interesting.

On the one side its indeed problematic that Microsoft (and others in the industry in general) hardly ever ask for permission for data to train models.

On the other side, if you consider AI and computers to be similar to humans, then it's not an issue at all because humans also learn by reading code or information from various sources that are copyrighted or stuff they havent paid for directly (eg a library or "I'm just browsing in a book store" are an interesting for example, the later some books stores actually dont like).

For poisoning the well kind of tactics, the author forgets that you poison the well for everyone - not just ML.

An issue not brought into the mix with Copilot is that computers are superior in their memory and the amount of learning they can do (even now already in specialized domains), and hence maybe the fear, long term issues (incl ethical ones) and people being upset should actually come from that angle.

You know, a computer that can reprogram itself seems like maybe a bad idea.

Yeah I have seen a lot of people bring up the question of essentially how training an AI to regurgitate code snippets is different from a human reading through and learning how to write similar code.

To me they seem obviously and intrinsically different, and I think your post gets at some of the reasons. For one, an AI can ingest a much larger quantity of data much faster than a human. No human could possibly read all of the public code on GitHub in one lifetime. But more importantly, the method of ingestion and synthesis, as well as the nature of the application of what’s learned, is vastly different. Generally a human reading through OSS code to learn isn’t going to then permanently store verbatim segments of it in their memory to regurgitate later. They may copy a bit of code while learning, but that then morphs into an independent understanding of the type of problem being solved and one way to approach that problem. Copilot OTOH has no capacity to understand different approaches to solving a problem, to compare one against the other, and so on. All it knows is which is used more frequently, with no capacity to map that against metrics of long term maintenance, CI suite success, readability, and so on.

As for the poison pill problem, I think it actually illustrates the difference quite well. A simple comment explaining that the code is a booby trap would be enough for any human reading the code to know how to avoid it. The AI on the other hand has no capacity to understand that comment and to know not to put the code into its dataset.

> A simple comment explaining that the code is a booby trap would be enough for any human reading the code to know how to avoid it. The AI on the other hand has no capacity to understand that comment and to know not to put the code into its dataset.

Copilot users can be leveraged to train the algorithm to emit less such comments/traps. Copilot inserts snippets by having an extension running in the editor, watching what is typed and responding. That's an opportunity to gather more training labels. For example, which lines of Copilot emitted code are frequently deleted soon after being inserted? Gather enough such training labels and the algorithm can start to recognize many of those comments (and/or the traps the comments call out) and stop emitting those specific lines.

At that point, warning comments help the algorithm better distinguish desirable lines of code from traps. And you'll be less likely to have traps leak into your code by using Copilot rather than copy/pasting snippets from the web (both are subject to human review, but Copilot will already have filtered some traps). So it's possible that widespread deployment of traps would drive adoption of Copilot by people who frequently copy/paste code from the web.

If you consider AI to be similar enough to humans to directly map our ideas about copyright and fair use, then copyright is way down the list of concerns we need to address. First we would need to arrange wages for each ML system's labor, as well as considering how to handle voting rights.

Keep in mind that CoPilot might itself be a well-poisoning tactic.

If use of Github presumes use of CoPilot and of comingled code under incompatible, or proprietary, licenses, such that use of such code could then create contributory infringement claims against distributors, users, or developers, there's something of a problem across the Free Software world.

Though that does seem rather a bit of a major footgun for Github / Microsoft themselves.

The Free Software movement didn't create, or even want, a world in which copyrighted software was a norm. But it adpated to the circumstance by treating copyright as a serious matter and being diligent in practices of use, appropriation, and licencing.

It's rather ironic that the source of the "Letter to Hobbyists"[1] are now advocating a devil-may-care attitude to copyright, software, and licensing in their own works and service offerings.

________________________________

Notes:

1. See: https://genius.com/Bill-gates-an-open-letter-to-hobbyists-an...

It seems like it would be trivial to poison the well so to speak for common function prototypes to add known vulnerabilities that can be exploited later when needed.

I would be exceedingly cautious about running code inserted by a black box fed by public inputs. Similar attacks have already occurred for NPM and I'm sure many other packagers.

[programmers, for years and years] Hi we built a huge industry around pointing AIs at normies’ comms and renting it out to advertisers etc. This is very legal and you are Bad if you try to fight it copyright grounds. Indexing and downloading stuff is a basic human right and always Fair Use!

[programmers when you do the exact same thing to their publicly shared code] How dare you, this is grounds for vigilante justice!11!1

The fly in the ointment being that, if such useless loops are widely adopted, ML can be trained to recognize and eliminate them too.

(Don't shoot the messenger; the concept is obvious.)

You could train one to recognize common patterns, and you could train one to heuristically guess that an algo running on test inputs is looping forever if it takes too long, but actually dividing the world into "loops forever" and "does not" is the Halting Problem, and our solution is guaranteed to either false-positive or false-negative.

(Anecdote: I heard a story once about a college that transitioned from free printing to metered printing, but since the printers didn't support metering, they implemented it by running print jobs in a virtual machine and tallying page count there. It took not very much time for some compsci student to cobble together a script that would augment a Postscript file with an "if my environment is virtual, print one blank page and exit, else..." prefix).

As you point out elsewhere, there's a human in the loop. Though the Halting Problem applies to human minds as well, they likely won't have any difficulty identifying useless code the ML doesn't pick up and adding it to the training set.

The next step is to integrate actual parts of the software into the poison-pill, so that if the poison-pill is not included then the software doesn't work.

It isn't necessarily a good idea, but it's the next obvious step.

Hard to make that work with snippets as small as Copilot is synthesizing.

That's true if the same mitigations often show up with the same contexts, like the simplistic one in the example. Making a general mitigation-avoiding AI would require it to understand the intention of the code.

Somehow I don’t think copilot is going to be especially popular in the professional Haskell programming community.

Toxic idiocy. I know I'll probably be downvoted for saying this, but it's the only accurate and honest response I can come up with to this post.

It's time build an open source free co-pilot. When people are actively contributing to an AI system, my guess is that it will fare good against rot scanned ones.

Funny that the machine learning operators seem to learn less well than the machines. https://spectrum.ieee.org/tech-talk/artificial-intelligence/...

It's interesting to compare Copilot with Wingman for Haskell: https://youtu.be/S0HvfXq3454?t=550 Both attempts at somewhat automating repetitive aspects of programming, but with completely different approaches. Wingman was inspired by tactic metaprogramming in theorem provers like Coq.

It seems like the author is advocating for the sabotaging Microsoft Copilot by creating GitHub repos of misbehaving code.

This could be seen as a form of protest or as a Luddite (https://en.wikipedia.org/wiki/Luddite) reaction to AI automation of coding.

James Micken's warning against connecting machine learning to the Internet of Hate still apply: https://youtu.be/ajGX7odA87k?t=1206

Microsoft GitHub Copilot regardless of legality to me is a direct attack on OSS in an EEE manner. It is an abuse of the platform. It should have been opt-in. It pushed me ahead to remove my repos from GitHub over to Gitlab (love that instant import feature), repos will be private by default. I will be deleting my Microsoft accounts too. Already off Windows.

I also think the software development community should consider poisoning the well, even if it’s already legally poisoned.

Moving to Gitlab wouldn't stop them pulling code into the model would it?

I get that it still makes a statement.

It would if the repo is private. But that defeats the purpose of OSS.

Voting with your wallet for extra points.

Someone should definitely sue them over this, to determine if the model is a "derived work" within the meaning of the GPL.

Someone should also release a model trained on the leaked Windows source code ..

People have sued over similar stuff in the past, like Google training their language model on copyrighted materials. The legal consensus is that a model trained on a large volume of copyrighted works does not infringe copyright, even if it occasionally vomits up chunks of the input model.

That said, if you trained a model specifically designed to violate copyright I suspect the courts would have a different opinion. They consider intent just as much as (if not more than) what you actually did.

Do you have a reference? Does this cover generative models?

Here is a discussion of the Author's Guild v. Google case: https://towardsdatascience.com/the-most-important-supreme-co...

The article says it is unclear if it covers generative models.

This is as good a starting place as any. https://ilr.law.uiowa.edu/print/volume-101-issue-2/copyright...

This might sound stupid, and maybe it is, but that’s the state of law right now. It’s possible somebody will drag out a big slam dunk case that overturns much of this or that new laws will be passed… but right now (as far as I can tell) Copilot looks pretty legal.

(Of course, talk to your legal counsel, this may vary be region, etc.)

It's a bit different here. It's just supposed to regurgitate parts of it's input model.

Assuming GitHub isn’t lying, no it isn’t. They claim the vast majority of output is original, much like how GPT3 typically generates original text (although can sometimes output inputs verbatim). If that’s their intent and they haven’t deliberately tried to make their model output it’s inputs verbatim, then they’re good. Remember, there’s a big sampling bias online towards people sharing bits of code that are obvious verbatim inputs - you don’t see all the ones that aren’t.

I personally don't understand the anger. I don't care if a human or an AI use my code. I open sourced it to be used by anyone, big corps included.

What you say only makes sense if your code is licensed under something similar to public domain. Some licenses, like the GPL, require derivative works to use the same license.

Is it derivative work though? I don't think reading GPL code requires me to only write code under the GPL license.

Could the derived work be created without the source?

No, simply reading GPL code doesn't poison you (though it does put you in an awkward position), but copy pasting gpl code straight into your codebase is a copyright violation unless you can prove fair use. And copy pasting GPL code is what Microsoft Copilot has been repeatedly shown to do.

First of all, none of this matters if the resulting code isn't distributed in some manner. For an internal app, for example, none of this matters.

Secondly, it's pretty much obvious that fairly trivial (and, frankly, probably not so trivial) code snippets are reused/reworked from various sources all the time. This makes it hard for me to see Copilot as this new and existentially harmful thing.

People use different licences, because they want different things, e.g. attribution, or that improvements be contributed back, or nothing.

It is harmful to open-source, because if these wishes are not respected, people will stop contributing and stop open-sourcing things.

For the GPL it matters even if it is distributed in binary form. For AGPL it matters if you make a service available.

AGPL projects will benefit from these huge coding models like everyone. This puts all code in an open position.

>> but copy pasting gpl code straight into your codebase is a copyright violation unless you can prove

This is not true. The copied code needs to be distinct and uncommon; a tiny snippet is very unlikely to meet this requirement.

That's true, but some of the copy pasting that Copilot indulges in would absolutely fulfill that requirement. See: copilot copy pasting quake's inverse square root function: https://twitter.com/mitsuhiko/status/1410886329924194309

Copy pasting a one-line snippet may be OK, but copy pasting a full function is already much more of a grey area.

The Quake case is a pretty non-central example, because that particular function has been copied verbatim into many code bases.

This fast inverse square root function is also getting old. If it was a software patent, which are not recognized in many places, it would have expired a few years ago.

We're discussing copyright though, not patents, which for better or worse, have terms of almost a century.

In the US, the copyright lapses 95 years from publication, or 120 years from creation, or 70 years from author death, whichever is shortest.

So the inverse square root function will stay copyrighted until 2100 (the source was published in 2005 I believe).

This is minor, but it's at least a bit older than that. Via Wikipedia I found a link to the code on a Chinese forum in 2000: https://web.archive.org/web/20150702180504/http://bbs.csdn.n...

From what I can see it has only been "repeatedly shown to do" it when you explicitly try to force it to do it (e.g. adding those Quake specific comments to bring out the fast inverse square root code). I want to see a serious analysis of Microsoft's claim that it only happens 0.1% of the times. Has it actually been debunked?

Because when you OpenSource and use the BSD license, then they are in violation since they will not provide you credit.

No one should use this, because you could easily be a derivative of something else and you are not in control of your own license.

Making a small quote adapted to your use case and variable names is more like human learning than abusing GPL projects wholesale.

Who said GPL?

If you copy something from a BSD licensed software you should credit them.

It is only an attack on OSS if licenses like the GPL are not respected. If copilot simply copies code it was trained on, it is more of a threat to proprietary software, since any sufficiently large copy from GPL'd software would have to be GPL licensed too.

If GPL is respected, copilot may help many proprietary softwares to be open source re-licensed. That should be seen as a positive outcome from FLOSS supporters.

But it is not easy to predict exactly what will happen and how people and companies will behave.

Softwares? Software is plural.

Actually, they both are correct. "Softwares" has a different and older connotation to it, but both are acceptable in English.

I’m actually wondering why you would release such a product as it seems like a path to software developers unionizing.

If their use of GPL'd code is fair use, then they are probably not required to respect any restrictions the licenses places on the use of that code.

I welcome this attack, to me all code should be free of any restrictions, like WTFPL.

Information want to be free.

I think Github should support some kind of variant of robots.txt in the source code. Maybe a common line in the license file that excludes it from ML training data. That said, if you want to exclude it, that's your right, but you should consider why you are open sourcing in the first place, and what about ML feels different to you than traditional human cognition.

Why should the users be forced to do extra work in order for Github to do their commercialization "correctly"?

This is something Github/Microsoft started, something that only really benefits Github/Microsoft and that they were doing lackadaisical from the start by ignoring licenses.

You could say the same thing of people with a website that they don't want to be archived on the Internet Archive or indexed by search engines.

Github ignored licenses for this project and included code from all publicly-accessible repos. They claim it's not copyright infringement and you can e.g. use code from this AI trained on open-source repos in closed-source projects.

It's not as simple as that...

It's not clear to me how it is copyright infringement if the generated code is truly new. Of course, if some code output is very similar to existing code, then it is obviously copyrighted.

When you read code from an open source GPL repo and are inspired by it to write your own code, it's obviously not copyrighted.

When I read code, I understand it, its intentions, and its output. I'm not simply replicating things that I've read before. I learned about inheritance in java in school. When I go to write python, I need only look up the syntax because I understand the logic.

But what matters is the output, no ?

Also, some people explicitly avoid reading codebases to avoid duplicating by mistake some copyrighted code.

Not sure if this is true but I heard that it used to be forbidden to Microsoft developers to read open source code.

One problem is that copyright is not well-defined. They just have to convince a court that their use is "transformational" and not a "derivative work" and they're home free. Then it doesn't matter what the license says, because Microsoft isn't licensing your work.

No. Opt-out is exactly the wrong approach. Instead consider a flag indicating that "I'm OK with mu source being used for training AI models." i.e. opt-in.

The problem is that defaults are very powerful. I could equally well argue that the Internet Archive shouldn't archive my website unless I give them opt-in permission. Of course, if that were the case, the Internet Archive effectively wouldn't exist.

ADDED: To be clear, it may be fine to say that making certain things opt-in means that other things that many might value can't exist.

The counterpoint here is that the internet archive isn’t a for profit service which is claiming exact reproductions of other people’s content as “new work”

> The problem is that defaults are very powerful

This isn't the problem, it's the whole point.

Things need to be made opt-in because we want there to be a meaningful barrier to companies using data that does not belong to them in order to make money.

It is GitHub Copilot, not Microsoft Copilot...

I understand that it may feel good to be mad at large corporations but GitHub Inc. is still independent from MS.

Ok, let's be real, this is just plain stupid:

1. This is worse than cutting off your nose to spite your face. I can easily imagine some config being messed up, and then the author shoots themselves in the foot. Also, god forbid anyone else have maintain the Rube Goldberg machine you've decided to create.

2.The idea that this could provide some significant dent to Copilot's utility is laughable. First off, it's not hard to see how AI could catch stuff like this. But more importantly, Copilot is clearly and obviously not intended to just be paste and apply. Every comment I've seen from alpha testers has stated how Copilot is uncannily good, but all the code must still be reviewed and it is oftentimes just flat out off. Code like this that would fail immediately would easily be identified.

I understand the arguments over Copilot, but this is just a dumb response to it.

This.

At the end of the day source code is for human readability. If you pollute your code with a poison pill you just make it harder for other humans to use.

ML Systems are doing sequence analysis and looking to predict the next token, for an arbitrary length of phrase. You've poisoned your code but compared to the volume of GitHub or any other public code data set it becomes the equivalent of noise. If you create a movement that puts poison pills on all Haskell files, now every project has painful boilerplate and an AI will learn to produce the same code. And every human that deals with it won't see a difference because you created a Haskell standard.

Reply on news.ycombinator.com