Comment by iamlucaswolf | Hacker News Reader

Comment by iamlucaswolf | Hacker News Reader

What amazes me is how predictable(?) all of the recent issues were.

Don't get me wrong, the folks behind Copilot are clearly, without any doubt smart, creative, and capable. But then... None of these issues (reproducing licensed code ad verbatim, non-compiling code, getting semantics wrong, and now this) are 0.01% edge cases that take specialized knowledge to see or trigger. I remember some of them being called days ago in the initial HN thread by people who haven't had beta access.

I really wonder how this announcement/rollout looked like on the management side of things. Because a) these shortcomings must have been known beforehand and b) backlash from people who feel threatened for their jobs/"stolen" of their open source work was (I guess) foreseeable? I've already read calls to abandon GitHub for competitors; this can hardly have been an acceptable outcome here.

Nevertheless, Copilot is still one of the most innovative and interesting products I've seen in a while.

I’d be very surprised if management at the least didn’t have their heads in the sand about the potential failure mode. There are often “must deliver” dates at large companies because someone made a promise about a deadline and now heads will roll for missing it whether anyone actually cares or not. So long as middle management thinks the C suite is watching them, they are desperate to meet quota.

Hilariously, this results in stuff like Copilot getting released to great big legal problems. Only then does the C suite actually notice the project and get upset that it is a legal nightmare for them.

I think the real secret to winning in big tech is that your job is just to keep your head down and keep the money rolling in without causing headaches for higher ups. Increase sales, make customers happy enough to keep paying, maybe release a cool product. But more importantly, don’t cause a major outage, burn the PR team, or get caught up in a legal kerfuffle.

You make a good case for innovation through acquisitions rather than in-house development. Once the derisking aspect is factored in acquisitions suddenly look a lot more attractive.

I saw this type of thing coming a mile away and left GitHub as soon as they were bought by Microsoft. TBH even despite my inherent distrust of Microsoft this is way beyond the hypotheticals I had in mind when I deleted all content from my GitHub account. Now I’m worried about VSCode as another potential vulnerability vector. Has anyone done a recent independent audit of what is sent across the wire to Microsoft from VSCode?

Well they have Telemetry enabled by default so you should disable it:

https://code.visualstudio.com/docs/getstarted/telemetry#:~:t....

Maybe something else still goes over the wire...

Yes, it looks like unfinished work. They could have:

- implemented plagiarism detection to attribute code to its source (where possible), then present the result together with the link. This makes Copilot same with Googling your answer and then copy-pasting the code. You are fully responsible

- implemented some regexes to filter out secrets, or even better, change the secrets to random values in the training data

- implemented a robots.txt like system so people have a method to ban the Copilot spider from their code

If they did these things before release it would have been so much better. But they are simple fixes so I see no technical obstacle.

We should keep in mind that the product is still in beta / technical preview.

This is a good point. There is a lot of outrage now, but the product when finished might have every single wrinkle removed.

This one, for example, seems it should be pretty easy to fix. You could even make a hack that replaces ALL sufficiently long and sufficiently random strings with garbage/zeroes, at the point of recall. The difference from the case of regurgitating GPL sources is that the information that it looks like an API key can be deducedd from the output of copilot, so you don't need to track it through the system like you would with a system of attribution.

How do you tell a “long and random string” from a base64 encoded PNG file or embedded script or…

Should any of those be autocompleted?

You don't. The logic is unchanged if the data changes. A snippet of code would be unchanged, apart from the data.

    // Add an arrow icon
    var arrow_icon = base64decode("00000000000000000...");
    add_image(arrow_icon);

That is: the prerequisite for this approach being viable is if one assumes that "code" and "data" are distinct, and that data can be seen as irrelevant placeholders. That is: in the example above I was after the code to add the icon, not the icon payload itself.

There are obvious bordeline cases like large numeric constants that are actually core part of the logic. E.g. a method that multiplies by Pi with 14 digits wouldn't work very well if they were replaced by zeroes. So most likely numerical constants would need to be left alone.

Often times secrets are numerical constants. In your own example, the icon is a base64-encoded number. How would you tell secret numbers apart from the rest?

Base64 isn't numeric it's alphanumeric. The only reason this is reasonable (again) is that alomost all secrets like api keys or complex passwords are maximizing their information content and are therefore alphanumeric (or better). Base64 encoded data does too, and is an innocent casualty in that censorship.

> Base64 isn't numeric it's alphanumeric.

They meant that a number written in hex (base16) is still a number, even though you use some letters. Similarly, a number written in base64 is still a number.

Yes, and in that case I’d like it of Copilot erased/replaced any string over (say) 15 or 20 characters unless they are [0..9]+.

Obviously any string is a potential number in some encoding so the only encoding to exclude would be decimal.

Should we really be forgiving one of the world's richest corporations for launching a marketing campaign with expansive claims for a half-baked product because, in the fine print, they call it a technical preview?

Presumably you can get your money back if you don't like the results.

edit: well this appears to be unpopular. It's a preview release, nobody is using this in production. They are offering the tech for free while they kick out the bugs and determine where things don't work as everyone expected. The fact that this is doing things they might not have expected suggests that this part of the process was necessary.

If you expected this to be production-ready, then you've misunderstood the purpose of a preview release. This applies to MS the same as it does any other developer.

The point of GP is that the grand PR campaign doesn't really state it's an unfinished product and that they are looking for free testers and security and legal audits.

Do you parse "preview release" in some way other than that which I outline above?

It’s unacceptable to ship this kind of security flaw even for a half baked internal proof of concept.

If they're encouraging users to distribute my GPL code under a CC0 license that doesn't even mention my name, how am I gonna "get my money back?"

This isn’t the first time Microsoft has introduced an AI that became hobbled by relatively easy to imagine exploits.

Lets widen this a bit (possible slight hyperbole ahead, but generally this is my feeling now):

It is more the rule than the exception that any service using AIs are less usable than the previous solution. That is, unless you think about how usable they are to extract money from gullible investors or for making laughing stock of their users and/or developers.

In fact I while I'm certain they exist I cannot right now come up with a single product that I use for anything other than fun or creativity (games, painting) that have been improved by recent AI additions.

Thinking of it maybe maybe Google Translate qualifies, but that depends on how you define recent.

Oh, and by the way maybe there is something that qualifies as AI in some of the new translation web applications I've seen recently.

I assume you are referring to Tay? That was wild

https://en.m.wikipedia.org/wiki/Tay_(bot)

It depends on whether they fix the problems before launch and whether the issues found during preview cause Real Problems (eg, secrets found this way resulting in significant cybercrime).

Seems to work for Google?

Only because people feel they have no option.

And only for so long.

DuckDuckGo.com is rising exponentially and have been doing so for years. Ans yes, mathematically exponentially, not cool kids speak or journalistically exponentially: https://duckduckgo.com/traffic

What do you suggest that will make a difference in yet another human era of aristocratic capture of our lives and agency.

They do not have an information advantage, just a political one.

And we can see how concerned the general public is with taking control of politics for its gain. It very clearly prefers to be hands off and let a minority manipulate public agency for their gain.

Absolutely. I also believe that Copilot is getting more flak than appropriate at the moment.

To rephrase my comment above: I don't want to blame the team behind Copilot for not getting everything right on the first try. Neither am I in a position to do so, nor would I want to live in a world where smart people aren't allowed to make mistakes.

What irritates me is that there are two possible scenarios here:

1) They knew about potential issues and decided to release it anyway (without at least addressing them verbally). 2) They didn't.

And frankly, I don't know which one I like less. Even though it's still a beta/preview, either option seems to signal a degree of negligence? that feels unnerving given the potential impact of such a system.

That being said, if we do live in scenario 1) than I am certain that better framing could have prevented the PR fallout that we're seeing right now (at least partially). IMHO, GitHub (the platform) is still a great product after all.

If my product is in beta, is it ok for it to leak your secrets?

Unfortunately this is something large corps like AWS have been getting away with for a while now. Releasing half-baked product clones as GA when in fact they're still clunky and are probably beta at max.

Is it more innovative than for example: tab9?

There are things that already do what Copilot does, eg Kite so it's hardly innovative imo.

Kite does only fraction what Copilot is currently doing. It is great at suggesting function names and parameters, but it does not really suggest complete code or generate somewhat new code.

That's the norm for a Microsoft product. Sell something full of holes, deal with it only when it starts posing an existential threat to the product

I dont know if that true. Just because you are "smart, creative, and capable" does not mean you can predict every possible outcome or be incapable of missing the obvious

I have been on both sides of that, where I have had to point out obvious flaws in an idea to very smart people, and have had clearly obvious flaw pointed out to me in one of my idea's...

I think it is completely possible that some or even all of the issues co-pilot is facing were unknown at the time of release, even if they are obvious to some

Though they could have proofed the small number of handpicked examples on copilot.github.com to see that they compiled/didn't blow up on first run. Or one further, that they did what they were supposed to, in a somewhat reasonable way.

Reply on news.ycombinator.com