Hacker News Reader

I've noticed that after using copilot on a code base for a while, you can effectively prompt the AI just by creating a descriptive comment.

// This function ends the call by sending a disconnection message to all connected peers

Bam, copilot will recommend at least the first line, with subsequent lines usually being pretty good, and more and more frequently, it will recommend the whole function.

I still use GPT-4 a lot, especially for troubleshooting errors, but I'm always pleasantly surprised at how good copilot can be.

Copilot is a game-changer and very underrated IMO. GPT4 is smart but not really used in production yet. Copilot is reportedly generating 50% of new code and I can't imagine going without it.

Where do you get that 50% number? Do you mean 50% of all new code in the industry? That seems beyond extremely unlikely.

(I read it as 50% of their code)

The number is 40%, and it's 40% of code written by Copilot users. It's also just for Python:

> In files where it’s enabled, nearly 40% of code is being written by GitHub Copilot in popular coding languages, like Python—and we expect that to increase.

https://github.blog/2022-06-21-github-copilot-is-generally-a...

I wonder if this properly counts cases where copilot writes a bunch of code and then I delete it all and rewrite it manually.

From what I remember they check in at a few intervals after the suggestion is made and use string matching to check how much of the Copilot-written code remains.

It's all about the denominator!

There was some discussion by the copilot team that x% of new code in enabled IDEs was generated by copilot.

It varies, but here's one post with x=46 from last month. So, very close to half.

https://github.blog/2023-02-14-github-copilot-for-business-i...

Measuring output by LOC is not a very useful metric. The sort of code that’s most suited to ai is closer to data than code.

For my side projects, copilot easily generates 80% of the code. It snoops around the local filesystem and picks up my naming schemes and style to help recommend better. It makes me so much more productive.

For work projects, I tried it on some throwaway work because we're still not allowed to use it for IP reasons, but it is very good at finding small utility functions to help with DRY, and can help with step by step work, but can't generate helpful code quite as easily since some of our API and codebase just doesn't follow its own norms or conventions, and it seems to me that copilot makes a lot of guesses based on its detected conventions.

> It snoops around the local filesystem and picks up my naming schemes and style to help recommend better.

Are you sure about this? It doesn't seem to work on my machine. I think it will infer things that might be in other modules, but only based on the name. I'm basing this on the fact it assumes my code has an API shape that's popular but that I don't write (eg free functions vs methods).

It looks at your recently-viewed files in your IDE. I don't think it looks at anything outside your open workspace but maybe...

I would really love to see that. So far, all I've seen is cookie cutter code to reduce a bit of typing time. Everything else was more or less hot garbage that just stood in the way of typing. Maybe in a few iterations or years. So far, personally, I haven't seen anything useful. Not saying there isn't anything, just that I haven't seen any use and code offered by it stank. Is there a demo of someone using it to showcase this game-changing power?

It surprises me to hear this. Have you used it as I described by writing a descriptive comment first then waiting to see its response?

I only noticed it getting good at this after I was somewhat far along on a project, so I assume it requires an overall knowledge of what you're trying to do first.

Copilot only writes boilerplate, it can't really handle anything non-trivial. But I write a lot of boilerplate, even using abstraction and in decent programming languages. A surprising amount of code is just boilerplate, even just keywords and punctuation; and there's a lot of small, similar code snippets that you could abstract, but it would actually produce more code and/or make your code harder to understand, so it isn't worth the effort.

Plus, tests and documentation (Copilot doubles as a good sentence/"idea" completer when writing).

Absolutely. People will quickly realize that for coding, the natural language part of LLMs is a distraction. Copilot is much better for someone actually writing code, but unfortunately doesn't get as emphasized due to the narrative surrounding LLMs right now.

> Copilot is much better for someone actually writing code

I haven't used copilot yet, but I'm using occasionally chatgpt with prompts such as "write a bash/python script take takes these parameters and perform this tasks". Then I iterate if needed, and usually, i can get what i want faster than without using chatgpt. It's not a game changer, but it's a performance boost.

How natural language is a distraction here? and how copilot would do much better for the same task?

Try not using natural language and just type what you'd type into Google. You'll get the same results and realize that all of the natural language fluff is totally unnecessary. I just typed in "bash script recursive chmod 777 all files" (as a dumb toy example) and got a resulting script back. It was surrounded by two natural language GPT comments:

> It's generally not recommended to give all files and directories the 777 permission as it can pose a security risk. However, if you still want to proceed with this, here's a bash script that recursively changes the permission of all files and directories to 777: [...] Make sure to replace "/path/to/target/directory" with the path of the directory you want to modify. To run the script, save it as a file (e.g., "chmod_all.sh"), make it executable with the command "chmod +x chmod_all.sh", and then run it with "./chmod_all.sh".

It's up to the reader to decide if those are necessary, but I'd lean towards no.

you can always include the instruction to only return the code and no other text

Sure, but I want a system built for coding that does that by default... like Copilot.

... and it'll describe the code anyway, at least to me.

No script needed:

  chmod ugo+rwX . -R

(This is for GNU chmod like in Linux, BSD will be slightly different)

Of course, that's not exactly what you asked for (it's better, read the chmod man page: X applies executable only to directories) but you could just replace ugo+rwX with 777 or 0777.

I tried this with the following:

"Bash script to add a string I specify to the beginning of every file in a directory, unless the file begins with “archive”"

I tried looking for this on Google and didn't find anything that did this -- although I could cobble together a solution with a couple of queries.

The interesting thing is that I wanted ChatGPT to append the string to the filename -- that's what I meant. But it actually append the string to the actual file. That's actually what I said, so I give it credit for doing what I said, rather than what I meant. And honestly my intent isn't necessarily obvious.

I definitely see this as a value add over just searching with Google.

> Try not using natural language and just type what you'd type into Google. You'll get the same results and realize that all of the natural language fluff is totally unnecessary.

I can get similar results with Google sometimes and I can put together what I learned from different places.

But I can get scripts that meet my exact requirements with ChatGPT. Most of my ChatGPT related code is scripting AWS related code and CloudFormation templates.

I’ve asked it to translate AWS related Python code to Node for a different projects and a bash shell script. It’s well trained on AWS related code.

I don’t know PowerShell from a hole in the wall. But I needed to write PS scripts and it did it. I’ve also used it to convert CloudFormation to Terraform

I think you (and kenjackson above) are misinterpreting what I was saying. I'm not saying use Google instead of ChatGPT; I'm saying pretend ChatGPT is Google and interact with the ChatGPT text prompt the same way. You don't need fully formed coherent sentences like you would when talking to a person; just drop in relevant keywords and ChatGPT will get you what you want.

Isn’t that the game changer though that you can use natural language and treat it like the “worlds smartest intern” and I can just give it the list of my requirements?

It’s the difference between:

“Python script to return all of the roles with a given policy AWS” (answer found on StackOverflow with Google)

And with ChatGPT

“Write a Python script that returns AWS IAM roles that contain one or more policies specified by one or more -p arguments. Use argparse to accept parameters and output the found roles as a comma separated list”

> “Write a Python script that returns AWS IAM roles that contain one or more policies specified by one or more -p arguments. Use argparse to accept parameters and output the found roles as a comma separated list”

Again, this is completely unnecessary. This is like in the old days when technically illiterate people would quite literally Ask Jeeves[0] and search for full questions because they didn't know how to interface with a search engine.

A prompt that does exactly what you're asking: "python script get AWS IAM roles that contain a policy, policy as -p command line argument, output csv"

We'll see more of that terse, efficient, style as people get more comfortable, similar to how people have (mostly) stopped using full questions to search on Google. The "talk to ChatGPT like a human" part is entirely a distraction from taking advantage of the LLM for coding purposes. Perhaps more importantly, the responses being humanized is a distraction, too.

[0] https://en.wikipedia.org/wiki/Ask.com

At first, when I didn’t specify “use argparse” it would use raw argument parsing

It also thought I actually wanted a file called “output.csv” based on your text and gave me an actual argument to specify the output file that I didn’t want.

There is a lot of nuance to my requirements that ChatGPT missed with your keywords.

Sidenote: there is a bug in both versions and also when I did this for real. Most AWS list APIs use pagination. You have to tell it that “this won’t work with more than 50 roles” and it will fix it.

> It's not a game changer, but it's a performance boost.

The story of all AI in 2023 - maybe 2x performance improvement, maybe a bit less. The big problem is that you can't trust it on its own, so it doesn't improve productivity 100x. Not even a receipt reader is good enough to reach 100%, you got to check the total, maybe it missed the dot and you get the 100x boost after all.

Has the Copilot backend been updated to use anything more advanced yet? I tried it out when it was new and free for a while and it really struggled with anything that wasn't incredibly common. GPT 4 in its chat form works a whole lot better for niche stuff than that one did.

It's definitely far better than when it was free but not GPT4 yet for most people.

It's the opposite of chatGPT: it takes more time to produce useful output but it gets much better in more complex programs while ChatGPT gets worse.

Copilot's original underlying model is currently deprecated, if I remember correctly

Idk for sure autocomplete is a great interface for someone in the ide coding, but LLM can understand requirements whole and spit out full classes and validate that the output from the server matches the specs, they work great from outside an ide.

Either way, you’re sending your companys biggest asset to another company, aren’t you? I’ll try these tools when they start being able to run locally

I surely hope they use my copyrighted code and make millions out of it. Ideal case for me to sue them for lots of money.

How would you ever know? It will come in chunks of a dozen or less lines at a time and it will be written into your competitor's proprietary codebase (that you don't have access to).

Right.

If you are building something truly valuable locally, and it is innovative or otherwise disruptive and relies on being a first mover, centrally hosted LLMs are a non-starter.

Most software corps have countless millions of lines of code. You'd be spending lifetimes tracing where someone ripped your "copyrighted" techniques and methods.

The complete lack of security awareness and willingness to compromise privacy for convenience in people deeply saddens me.

> willingness to compromise privacy for convenience

I have to ask: do you carry a cellphone?

This is not really a valid comparison.

The cellphone is not a compromise for convenience. It allows me to make a living, providing internet connectivity and lets me keep in contact with friends and family. Without it, my freedoms would be drastically diminished.

With software we develop with, we have choices. We can use OSS. We can try to use open hardware. If we are working on sensitive things, we can use an airgapped system with vim.

When you practice these kinds of routines, they are not a burden. Actually, using vim instead of something like vscode increases productivity eventually. It does take a little bit of time.

When we couple our productivity with centrally hosted services, we greatly diminish our freedom to be productive on a wide range of problem areas. I don't say this to brag, it is to maximize all of our freedom.

In my view, most of us SHOULD be working on "sensitive" things. There is so, so much work to be done for the cause of freedom and liberty in software. We need to reserve that capability in us, we cannot let nameless people have an inside access to our expression.

A cellphone literally tracks your every move. If that's not a privacy concern then I don't know what is. Maybe a device with a microphone that's constantly on you. Oh no wait, that's also a cellphone.

I was born in the 70's, and I can tell you, you can survive just fine without a cellphone.

All of what you describe can be done on a desktop. But hey, if you want to compromise your privacy for some convenience, that's your choice.

Are you going to carry your desktop into the forest on a hammock and work? How about on a plane to other countries?

Will you carry your desktop in your car while living on the road? In the middle of forests and on top of mountains?

Will you work from a campsite with your desktop while not connected to the internet?

Can you have a meeting via your desktop from a rocky beach and no internet service?

A cellphone can't track what I type on my laptop, and it can't read encrypted comms my laptop makes to remote systems. I can put a cellphone in a distant location and use a portable, open source router with a VPN on the router, with encrypted, private DNS.

Not everyone lives inside a comfortable little box. There are all kinds of ways to do life.

Sure, if you are willing to compromise your privacy, which you clearly are.

In these companies, people are not permitted to carry their cellphone into workspace.

> GitHub Copilot [for business] transmits snippets of your code from your IDE to GitHub to provide Suggestions to you. Code snippets data is only transmitted in real-time to return Suggestions, and is discarded once a Suggestion is returned. Copilot for Business does not retain any Code Snippets Data.

Likely, some employee would whistleblow that they're not complying with their privacy policy, and either government litigation or a class action lawsuit would ensue. That legal process would involve subpoenas and third-party auditors being granted access to GitHub/Microsoft's internal code and communications history, which makes it pretty hard to hide something as big as collecting, storing, and then training from a huge amount of uploaded code snippets they promised not to.

It's not inconceivable that they're noncompliant, but my bet would be that if they are collecting data they explicitly promise not to it's an accidental or malicious action by an individual employee, and they will freak out when they discover it and delete everything as soon as they can. If they intended to collect that data, it would be much easier to write that into the policy than deal with all the risk.

Notably, this applies to Copilot for Business, which is presumably what you're using if you are at work.

Couldn't it happen more subtly, without having the code lying around for long? The model could be doing online-learning (ML term) and only then they discard code that they get send. This means your code could appear in other people's completions/suggestions, without it having to lie anywhere. It is basically learned into the model. The code could appear almost or even completely verbatim on someone else's machine, possibly working for a competitor. Even that it is your code would not be obvious, because MS could claim, that Copilot merely accidentally constructed the same code from other learned code.

Not sure that this is how the model works, but it is conceivable.

I was talking about someone stealing my codebase.

Talking about integrating the code into the LLM, others get the same benefit that you are getting, so I don't really see the issue.

So you can either develop everything on your own, or you can leverage LLM's, helping both yourself and others.

I sort of disagree that code is the biggest asset. Take the Yandex leak. What can you do with it? Outcompete them?

> Take the Yandex leak. What can you do with it?

Obviously, add it to the big training set of the next code model.

No or no company would be able to use it. As you type fragments of code are sent and discarded after use. You need to trust Microsoft to actually do the discarding but contractually they do and you can sue them if they accidentally or deliberately keep your code around or otherwise mismanage it.

They are obligated to give data to the government, and government took part of spying in Brazil for Boeing in the past, but I guess they are using this capability only for a few strategic companies, and most companies are not that.

> government took part of spying in Brazil for Boeing in the past

Do you have more details? Please elaborate.

But that is naive, isn't it? Who has the money and time in their life, to actually sue MS? Even if "you" is a business, few will have the resources for that.

Individuals do not (although a class action would be feasible), but large companies that use Github and other Microsoft products, of course they have both the means to sue Microsoft and the motivation should their business be impacted.

Exactly

As a hobbyist developer with no formal training, I wish Copilot had a 'teaching' or "Senior Dev" mode, where I can play the role of the Junior Dev. I'd like it to pick up on what I'm trying to write, and then prompt me with questions or hints, but not straight up give me the code.

Or, if that's too Clippy-like annoying, let me prompt it when I'm stuck, and only then suggest hints or ask suggestive questions that guide me to a solution.

I agree, very exciting to see where all this goes.

One thing you might try with Copilot is to ask it to explain the code. It can often give insight, even on code that you yourself wrote a few minutes ago.

The Github Copilot Labs extension has "codebrushes" that can transform and explain existing code instead of generating new code, but none of it only gives "hints". Maybe one of the codebrushes can take a custom prompt.

You can create custom brushes, or open the "CoPilot Labs" panel and "explain" with a custom prompt.

Exactly this. I've tried to implement ChatGPT into may daily workflow, but you have to give it an excruciating level of detail to get something that remotely resembles real code I'd use, and even then you have to hold its hand to guide it in the correct direction, and still have to make some manual final touches at the end.

This is why I'm looking forward to Copilot X so much. It will hold much more context than the current implementation, and integrate the Chat interface that's so natural to us.

People have different preferences and habits. Having tried both models I much prefer having a conversation in one window and constructing my code from that in another. Although copilot is about to add some interesting features that may win me back.