GitHub Copilot regurgitates valid secrets
twitter.com
twitter.com
Don't get me wrong, the folks behind Copilot are clearly, without any doubt smart, creative, and capable. But then... None of these issues (reproducing licensed code ad verbatim, non-compiling code, getting semantics wrong, and now this) are 0.01% edge cases that take specialized knowledge to see or trigger. I remember some of them being called days ago in the initial HN thread by people who haven't had beta access.
I really wonder how this announcement/rollout looked like on the management side of things. Because a) these shortcomings must have been known beforehand and b) backlash from people who feel threatened for their jobs/"stolen" of their open source work was (I guess) foreseeable? I've already read calls to abandon GitHub for competitors; this can hardly have been an acceptable outcome here.
Nevertheless, Copilot is still one of the most innovative and interesting products I've seen in a while.
I have been on both sides of that, where I have had to point out obvious flaws in an idea to very smart people, and have had clearly obvious flaw pointed out to me in one of my idea's...
I think it is completely possible that some or even all of the issues co-pilot is facing were unknown at the time of release, even if they are obvious to some
This one, for example, seems it should be pretty easy to fix. You could even make a hack that replaces ALL sufficiently long and sufficiently random strings with garbage/zeroes, at the point of recall. The difference from the case of regurgitating GPL sources is that the information that it looks like an API key can be deducedd from the output of copilot, so you don't need to track it through the system like you would with a system of attribution.
// Add an arrow icon
var arrow_icon = base64decode("00000000000000000...");
add_image(arrow_icon);
There are obvious bordeline cases like large numeric constants that are actually core part of the logic. E.g. a method that multiplies by Pi with 14 digits wouldn't work very well if they were replaced by zeroes. So most likely numerical constants would need to be left alone.
They meant that a number written in hex (base16) is still a number, even though you use some letters. Similarly, a number written in base64 is still a number.
Obviously any string is a potential number in some encoding so the only encoding to exclude would be decimal.
edit: well this appears to be unpopular. It's a preview release, nobody is using this in production. They are offering the tech for free while they kick out the bugs and determine where things don't work as everyone expected. The fact that this is doing things they might not have expected suggests that this part of the process was necessary.
If you expected this to be production-ready, then you've misunderstood the purpose of a preview release. This applies to MS the same as it does any other developer.
It is more the rule than the exception that any service using AIs are less usable than the previous solution. That is, unless you think about how usable they are to extract money from gullible investors or for making laughing stock of their users and/or developers.
In fact I while I'm certain they exist I cannot right now come up with a single product that I use for anything other than fun or creativity (games, painting) that have been improved by recent AI additions.
Thinking of it maybe maybe Google Translate qualifies, but that depends on how you define recent.
Oh, and by the way maybe there is something that qualifies as AI in some of the new translation web applications I've seen recently.
And only for so long.
DuckDuckGo.com is rising exponentially and have been doing so for years. Ans yes, mathematically exponentially, not cool kids speak or journalistically exponentially: https://duckduckgo.com/traffic
They do not have an information advantage, just a political one.
And we can see how concerned the general public is with taking control of politics for its gain. It very clearly prefers to be hands off and let a minority manipulate public agency for their gain.
To rephrase my comment above: I don't want to blame the team behind Copilot for not getting everything right on the first try. Neither am I in a position to do so, nor would I want to live in a world where smart people aren't allowed to make mistakes.
What irritates me is that there are two possible scenarios here:
1) They knew about potential issues and decided to release it anyway (without at least addressing them verbally). 2) They didn't.
And frankly, I don't know which one I like less. Even though it's still a beta/preview, either option seems to signal a degree of negligence? that feels unnerving given the potential impact of such a system.
That being said, if we do live in scenario 1) than I am certain that better framing could have prevented the PR fallout that we're seeing right now (at least partially). IMHO, GitHub (the platform) is still a great product after all.
Hilariously, this results in stuff like Copilot getting released to great big legal problems. Only then does the C suite actually notice the project and get upset that it is a legal nightmare for them.
I think the real secret to winning in big tech is that your job is just to keep your head down and keep the money rolling in without causing headaches for higher ups. Increase sales, make customers happy enough to keep paying, maybe release a cool product. But more importantly, don’t cause a major outage, burn the PR team, or get caught up in a legal kerfuffle.
https://code.visualstudio.com/docs/getstarted/telemetry#:~:t....
Maybe something else still goes over the wire...
- implemented plagiarism detection to attribute code to its source (where possible), then present the result together with the link. This makes Copilot same with Googling your answer and then copy-pasting the code. You are fully responsible
- implemented some regexes to filter out secrets, or even better, change the secrets to random values in the training data
- implemented a robots.txt like system so people have a method to ban the Copilot spider from their code
If they did these things before release it would have been so much better. But they are simple fixes so I see no technical obstacle.
Everyone knows it’s usually far easier to write code than to read code. Writing code is a nonlinear process: you don’t start from the first character and write everything out in one single pass. Instead, the logic of the code evolves nonlinearly—add a bit here, remove a bit there, restructure a bit over there. Good code is written such that it can be mostly understood in a single pass, but this is not always possible. For example, understanding function calls requires jumping around the code to where the function is defined (and often deeper down the stack). Understanding a conditional with multiple branches requires first reading all the conditional predicates before reading the code blocks they lead to.
Reading, on the other hand, is naturally a linear process. Understanding code requires reconstructing the nonlinear flow though it, and the nonlinear thought process used to write it in the first place. This is why constant communication between partners during pair programming is essential—if too much unexplained code gets dumped on a partner, figuring out how it works takes longer than just writing it themself.
Copilot is like pair programming with a completely incommunicative partner who can’t walk you through the code they just wrote. You therefore still have to review most of it manually, which takes much longer than writing it yourself in the first place.
However, I've worked with people who struggle to write in English without introducing random punctuation, and don't "see" (or care about) the text on the screen enough to go back and fix it.
I think Copilot will be a great benefit to the lazy programmer, who understands the semantics, but just can't be bothered to get the indentation or other syntax correct.
That said, a more sophisticated linter might be useful in catching non-idiomatic, but syntactically/stylistically valid code that would thus be flagged as “valid” by current linters’ simple automata.
To me, this is a huge part of modern-day development. It's not only about producing functionally correct code, but also code that integrates well and is semantically relevant to the broader context of the application itself.
That doesn't mean Copilot's input will have no value, but it just means that developers will generally need to refactor that code in a way consistent with the app they're building.
If so, it means that you wouldn't find them by a search, but they would still be revealed by the A.I.
> What data has GitHub Copilot been trained on?
> GitHub Copilot is powered by OpenAI Codex, a new AI system created by OpenAI. It has been trained on a selection of English language and source code from publicly available sources, including code in public repositories on GitHub.
Sure, Microsoft Marketing Department says it was trained on public code. The PM probably told them it was.
Was it actually trained on public code only?
Any free text generation AI service has the same weakness, that you can't control what your model would spill out.
I know because I accidentally pushed a secret myself. Mistakes like that can happen super easily when it's a simple project and you've not setup all your things correctly.
So it's not too far-fetched, that they'd train AI on private repos data too...
Copilot is a glorified Markov chain autocomplete sitting on a huge dump pile of data. It is not aware of constructs such as "licenses" or "secrets" most people would have expected from AI. To prevent it from spilling secrets everywhere, a developer ~~should teach the AI a concept of secrets and the meaning of licenses.~~ has to implement a filter. A regexp-based one will do, I guess.
Deep neural networks have stuck us in hope-fueled uncanny valley, and very smart people tend to become very confused about their technology when they're subject to it.
This technology has its place about heuristic programming, definitely, but is not AI.
Still, it doesn't take more than a few sentences from GPT-3 to realize there's nothing on the other side. There's no spark of sapience, not even intelligence. There's the layer that can string together words, but there's no layer that can reflect on them and check whether they make any kind of sense.
To be fair, "sapience" may be a bit too high for the lower bound of what is an AI. I still think the bound is on that control / reflexivity layer. It's the problem that stifled old-school symbolic AIs. To this day, we can't even sketch an abstract, formal model for this[0]. Maybe DNNs are the way to go, maybe they aren't. But GPT-3 isn't even trying to solve that problem.
I'm not sure when I'll be ready to call something a proper AI, but I think it would first need to demonstrate some amount of on-line higher-level regulation and learning. I.e. not a pretrained, fixed model, but a system where I could tell it, "look, you're doing this wrong, you need to do [X]", and it'll be able to pick up on a refined pattern with a couple examples, not a couple hundred thousand. For this, the system would need to have some kind of model of concepts - it can't be just one big blob of matrix multiplications, with all conceptual structure flattened and smeared over every number.
--
[0] - I think the term of art here is "metacognition", but I'm not sure. It looks like what I'm thinking about, particularly metacognitive regulation.
> GitHub CEO acknowledges the issue... still waiting for them to pull the plug
I agree this is an issue for co-pilot as well - but it's really on send grid to invalidate keys that are known to be leaked?
Yes, that's inconvenient for the affected customers - otoh they won't get billed for other people's usage - or dinged for someone spamming using their keys...
I leaked one when I accidentally left one in a repo I was making public. Took 15 minutes for sendgrid to drop it after putting the repo public.
They are being generated. It's a Generative Pre-trained Transformer. It's generative. It generates things. That's what this technical term means. It's correct.
The generated output has building blocks (characters, words) which themselves are not "generated" in the sense that they are novel.
A random number generator generates numbers. It doesn't generate the digits used to represent the random numbers. Those are taken from a set (such as 0..9) and just "used".
But it's not generating the secrets, it's generating code that contain those secrets.
For a comparison point, afl fuzzer actually generates never-seen, valid JPEG files "out of thin air", see http://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thi... and matching conversation https://news.ycombinator.com/item?id=8571879
It would be interesting to combine both approaches and (interrupted by the sudden singularity ;-).
Well if it wouldn't replicate or even randomly generate malicious code, it would imply that CoPilot would somehow be able to solve Halting Problem or - at the very least - understand intent and purpose of both its output and training material.
Keep in mind that the very definition of "malicious code" is highly subjective, plus the intent and purpose aren't necessarily encoded in the program itself. If the latter were the case, there would be no need for documentation, requirements or specs.
This thing is worthless at best, annoying at everything and terrifyingly capable of destroying every programmer's productivity at worst. But it will stay, it will grow because stupid execs will keep dreaming of replacing their engineering talent with this, and Microsoft will laugh all the way to the bank.
It's like expecting that an AI trained on Shakespeare novels would be able to "help" a writer write Shakespeare-like novels. Sure, they might get something that might fool some people, but are they a writer? I think software is a lot more like "writing" than it is like "building".
What mostly annoys me is that this is a win-win for github regardless of the outcome. If people buy into it (even for just a while, and it currently seems like some really smart people are buying into it) they'll carve a huge piece of a new market. If it fails, they'll make it seem like an experiment into the whole ethical gray area of what should and shouldn't be used for training, and that they just wanted to draw attention to it.
How many times have you seen the two coincide in large orgs, especially ones as complex as Microsoft?
People treat organizational decisions from large orgs as if they were made by a particularly stupid, incompetent individual, but that's not what happens. They're made by organizational processes, incentive structures, and emergent behaviors.
That's not cynicism -- structuring over 100,000 people to collectively act in ways one might consider reasonable is a genuinely hard problem. We tend to blame unreasonable people or malicious behavior, but every person in an organization can be smart, ethical, and reasonable, and stupid stuff will still happen.
If there were one bad apple, like Enron, it'd be fine to blame the people there. If it's nearly every large organization in the history of humanity -- including formerly good ones like Google -- it's reasonable to look for a more systemic explanation than unreasonable people.
Your boss: "We need to launch this, or our division is shut down."
Your web search: Famous AI failure
You would like to stay richly compensated. What do you do?
This is one scenario, and waaay oversimplified, but the way you get richly-compensated is by doing what's required to be richly-compensated. That usually has more to do with corporate politics than long-term corporate outcomes.
And it's another (bigger) issue for Copilot to pick up that API key and put it in someone else's project.
Copilot is an issue, but I would assume malicious agents are already doing copilot-like things, just not in the open.
For example https://docs.github.com/en/code-security/secret-security/abo...
The fact that Copilot recreates API keys that still work makes me wonder if they come from a semi-public place, because SendGrid is usually quite fast at blocking API keys that were accidentally made public.
Just a quick search:
https://grep.app/search?q=%28secret%7Capi%29_%3Fkey%5Cs%3A%3...
Not saying putting keys in a private, but 3rd party hosted repo, is a terrific idea.
> COPILOT SECURITY BREACH
> SendGrid engineer reports API keys generated by the AI are not only valid but still functional.
> GitHub CEO acknowledges the issue... still waiting for them to pull the plug or make a comment. :popcorn:
Quoting https://twitter.com/pkell7/status/1411058236321681414
They should scrap it and Microsoft should be ordered to sell Github because they have a conflict of interest.
For example Microsoft has access to your private repos and can do things like co pilot with your data. Who knows maybe your code powers Windows 11 now.
However, while the keys are then already leaked, you'd have to go search for them. Copilot suggests you use them in you editor. That is not quite the same imo.
It goes from deliberately searching and using leaked keys to having them handed to you without context. I feel it is a bit like finding an unlocked bike, if you take it, it is still stealing. But here there is a guy at the bike parking lets say that is handing out bikes to anyone passing by. Not the best analogy, but i think it covers my point ;)
But yes I get your point but I also believe people still need to apply some sense to what co pilot suggests.
But there's another side to this - if you write code using co-pilot against a popular Api - and co-pilot gives you a valid key - and you access data or a system you aren't supposed to - would you be liable under the various draconian antighacker laws?
If you pick up a key card from the street, and enter someone's home - you'd be trespassing after all..
https://docs.github.com/en/code-security/secret-security/abo...
Doesn’t make sense to implement fixes on CoPilot.
Public repos are scanned by the secret scanner. Why is it CoPilot's responsibiity to also scan the keys too?
The issue is that it's impossible to tell if it's done that or not short of googling each line it comes out with
Anyway I've determined my reactions are now based on disappointment rather than thought, a bias that should be factored in
I hate this belittling attitude that HN has towards projects in certain fields. It's like if I dismissed any progress in graphics as "just glorified triangle-drawing".
It won't change nothing. It's madlibs for code. The 90% of programmers who are mediocre will drown out the 9% who are competent and the 1% who are talented.
All the API keys are still working and who knows, someone might complain about a huge fee right in here because they forgot to revoke it. Only time will tell.
I am certainly going to avoid this contraption. No thanks and most certainly no deal.
Downvoters: So are you saying GitHub Copilot DOES NOT do the following:
Leak working API keys in the editor.
Generate broken code AND give you the wrong implementation if you add a single typo?
Copy and regurgitates copyrighted code verbatim.
Guesses 1 out of 10 tries.
Send parts of your code when you type in the editor.
The content of your editor is sent by Copilot to its cloud service (the FAQ says: "The GitHub Copilot editor extension sends your comments and code to the GitHub Copilot service"). So yes any editor content is leaked, including sensitive information.
But is this content sent to other Copilot users? AFAIK, no. The FAQ mentions the OpenAI Codex as the training set.
They advertise an English-to-French translation service powered by AI. But it appears that nobody who is French native has even reviewed the service presentation. When the marketing material is just a joke, what can you expect in production if as a customer you use the service?
Just this example tells me much about the internal organization of the company.