Hacker News Reader

Comment by gamegoblin | Hacker News Reader

The place where Copilot shines most for me is boilerplate like:

    up = foo.north()
    right =

and then it will correctly suggest

    up = foo.north()
    right = foo.east()
    down = foo.south()
    left = foo.west()

This requires a kind of linguistic understanding that is beyond basic intellisense. If you are just wanting it for basic intellisense autocomplete, it's not much better than JetBrains stuff. But for things that require linguistic comprehension, it's incredibly useful.

For example, if I have a `Point` object with `x` and `y` fields, and I make a function called `lowest_point`, it knows I want to the point with the minimum `y` value. Because it knows the name `y` is most typically used for vertical direction, and the word `lowest` also implies vertical direction. This is kind of a linguistic/semantic knowledge that is not found in just the type system.

View on HN

What's funny is that i often see Copilot give you code that Intellisense/a type system will tell you is obviously wrong. I wish Copilot was more integrated with type systems.

It produces very good results with typescript. When there is an interface and I'm creating an object that implements the interface; it produces a result that fits into that type.

This has not been my experience with Copilot and Typescript.

Same here.

Same with Rust. Sometimes I just have to write a comment saying "display implementation for X" and it will autocomplete with the trait implementation. This is extra magical because I don't need to remember the exact detail of the trait.

It is definitely hit or miss, but I don't find it too annoying to just clean up the small mismatches that crop up sometimes.

What's worse, is that if you make a mistake in the code leading up to it's completion (even one picked up by Intellisense), Copilot will now assume that you actually wanted the mistakes, and it's autocompletion will give you even more mistakes.

Right now Copilot is just given the surrounding code, which is obviously limited. I suspect the next big jump will be a model that is given:

1. The surrounding code

2. The type/intellisense information of all the identifiers in the surrounding code

3. The class/function definition of any identifiers in the surrounding code

I think #3 will be huge. There is a huge difference in Copilot's performance when dealing with types defined right above where you are writing code, and types defined in some far-away module or library. Right now if you use a class you defined in another file or adjacent project, Copilot can't see it. But your IDE has jump-to-definition functionality that could trivially be wired in to something like Copilot.

The team behind Stable Diffusion (the image synthesis model that's been all over HN recently) have said they are working on stuff like Copilot as well, so I expect we will see a huge transformation in this space in the next year or two.

This.

It feels like something that is obvious and should go in the copilot plugin rather than copilot itself.

Given N suggestions, validate the code + snippet on them using normal error checking.

Promote suggestions with no errors.

The intellij plugin, for example, will often suggest 'default' suggestions that are entirely wrong. Given that `foo.bar.foobar()` exists, it'll suggestion `foo.something.SomethingClever()` when those functions don't exist, or exist on types other than foo.

However, if you look at all the suggestions on the side panel, you'll see several of them are actually valid. It's just that the default completion isn't.

Yes, but no.

It feels like magic because it really does get some things right.

But I find that in most cases, because it's often wrong, you have to deal with wrong suggestions, and be very careful about what you accept.

I feel that for most coding scenarios, it doesn't actually, in most cases, do that much but save a few strokes.

I suggest that if there is 'high boilerplate' - then maybe there's upside.

But for most things, it really doesn't help, despite how in some cases it's really magical.

I turned it off and don't miss it.

What I would however prefer is merely 'much better autocomplete'.

Somewhere between 'ok autocomplete' and 'code suggestions' is the sweet spot for dev. that's not boilerplate.

Yep, for the boilerplate alone it's worth the cost for me. Oh you need to create a DTO from an internal object/class? Define the fields, write your `populateFromX()` function signature, maybe you have to type `$this->id = $input->id` and then CoPilot does the rest. And that's only one feature that I love, it does an excellent job most of the time writing the boilerplate for loops and the like based on my function names. Often I have to make few or no changes to it's code.

At a worst it provides a great starting point to jump off from, at best it writes it for me just like I would have (matching style/variable naming/etc).

One of the biggest boilerplate time savers for me is I have a service that maps HTTP requests to some specific class. I can paste an example HTTP request into a comment like:

    /*
    <http request here>

    The well-type class representation of this is:
    */

And then it spits out a totally sane and reasonable class. In this file I already have 10+ classes already defined and manually touched up by me to rename some things, remove some fields, etc. So every time I add a new one, Copilot takes into account the previous examples and each addition requires less or no manual tweaking.

100%. Most of my calls to the backend literally only need me to type the function name `fetchItems()` and CoPilot correctly uses my axios instance, set the return type to be my base API response wrapper, sets the generic in my base wrapper to be `IItemDTO[]` (it guesses this correctly probably 90%+ of the time), and even shoves the response into a Vuex store just like I would have done (same deal, guesses right about 90%+).

It's not like it's a ton of work to do that myself but normally the way I would accomplish it is to copy an existing function, change the return type, change the url, change the Vuex store it puts the data into but with CoPilot it handles that perfectly for me the vast majority of the time. And if I'm doing CRUD endpoint I don't even have to write the other function names, it will autocomplete the "RUD" part after I write the "C" (and by write I mean I put `createItem(payload: ICreateItemDTO)` or similar and it writes the body of the first function as well).

For me it's the perfect blend of "generated code" and "human tending" to get exactly what I need without having to do the repetitive/boring parts. It's been a joy to work with.

Is it the case that the code you wrote with copilot is much more prone to be refactored.? The usage you describe sounds like a lot of repetition and hard to update if any of those requirements (Vuex store for example) change.

That’s not been my experience. I mean the code is no harder to refactor than anything else, the code follows the style I prefer (and my linters enforce), if it doesn’t I change it as needed.

If I were to switch from Vuex to Pinia it would be the same amount of work with/without copilot. I don’t let copilot generate code without me reading/reviewing it and sometimes making tweaks. CoPilot literally (and I mean that) generates exactly what I was planning to write, if it doesn’t I start typing it my preferred way and often CoPilot then offers a completion in that style.

Without CoPilot the code would be virtually the same, so refactoring isn’t impacted one way or the other.

If you find yourself generating so much boilerplate, it's probably a better design and more enjoyable to craft a single reusable abstraction than it is to rely on a glorified copy and paste tool to create a mess of redundancy.

> For example, if I have a `Point` object with `x` and `y` fields, and I make a function called `lowest_point`, it knows I want to the point with the minimum `y` value. Because it knows the name `y` is most typically used for vertical direction, and the word `lowest` also implies vertical direction.

Unless, as in many systems, y=0 is the top and y increases downward.

The magical thing is once shown examples of this in your code, or a comment saying as much, Copilot will adapt.

For example, I just gave Copilot:

    # The screen is inverted, so bigger y values are lower
    def lowest_point(points):

and it completed:

    # The screen is inverted, so bigger y values are lower
    def lowest_point(points):
        lowest = points[0]
        for point in points:
            if point.y > lowest.y:
                lowest = point
        return lowest

Even if you write some code that implies that bigger y is lower, Copilot will pick up on it.

Notably, I think writing with copilot encourages you to write comments that provide necessary context.

Naturally, this makes it easier for other programmers to understand the code.

>This requires a kind of linguistic understanding that is beyond basic intellisense

Are you sure? If you would look at the statistics what comes after

    up = foo.north()
    right =

I bet most of the time it's

    up = foo.north()
    right = foo.east()
    down = foo.south()
    left = foo.west()

but it contextualizes it with adapted variable names, case and everything. That's very hard to do with only statistical analysis.

Exactly, for example if I give it:

    hn_example_up = foo.north_garbage_jlakd()
    hn_example_right =

it correctly spits out

    hn_example_up = foo.north_garbage_jlakd()
    hn_example_right = foo.east_garbage_jlakd()
    hn_example_down = foo.south_garbage_jlakd()
    hn_example_left = foo.west_garbage_jlakd()

Even though those identifiers including random letters are certainly not in the training data. It can do surprisingly good basic reasoning.

Here is another example. I give it:

    hn_example_up = foo.x_0_y_1()
    hn_example_left = foo.x_neg_1_y_0()

So my function names are like referring to unit distances on the x/y axes. Copilot correctly figures out what I'm doing and completes:

    hn_example_up = foo.x_0_y_1()
    hn_example_left = foo.x_neg_1_y_0()
    hn_example_down = foo.x_0_y_neg_1()
    hn_example_right = foo.x_1_y_0()

A few weeks ago I coded a rubik's cube solver that involved some very nuanced and weird spatial reasoning and was blown away by Copilot's ability demonstrate it "understood" concepts such as turning the right side down is identical to turning the left side up when viewed from the back of the cube, or stuff like that.

No it isn’t. Variable names are literally fungible in this context.

Yes, ultimately it's just a really good statistical model.

But it does a really good job, and it's not just matching code it's seen before. You can use any variable name. You can do this in a weird programming language which uses weird symbols for assignment and calls. copilot will probably see though it and give you a good prediction.

You can probably even swap west for not_east in code elsewhere in the file, and it will manage to predict that you also want not_east here. And I guaranteed nobody has written code like that before.

Especially boilerplate in a language or framework you don't usually use. It's such a time saver.

I've never used Copilot, but are you worried it will only spit out:

    up = foo.north()
    right = foo.east()
    down = foo.south()

And you won't notice?

Not really no

Why not?

From what I've seen GTP3 works exceptionally well until it doesn't, which can be easily missed.

But I read the code, and I run tests.

does the bot write most of the tests too?

Do you use it to write anything you want - even works on READMEs and comments! But like any good test - test it in a different way to how to implement it.

My rule using copilot is to use it as a fancy typing assistant which will frequently make mistakes - so you need to read everything it writes for you, and have solid test coverage too in order to avoid accidentally accepting a dumb mistake.

This made it useless for me because writing code is easier and faster than reviewing someone else’s code. And less error prone.

And much more enjoyable. I haven’t tried Copilot yet, but the process of letting it generate something and then checking it for accuracy sounds mind-numbing and stressful. Kind of like supervising a car that’s on assisted cruise: I don’t enjoy that either.

I've simply never had an issue with this sort of thing in months of using co-pilot to be my fancy autocomplete. And if there was an issue I'd notice quickly enough, because I am still the primary pilot.

It is also great as a starting point for error messages, comments, etc.

I think the key is to still give a shit about the code you're writing.

You have to think about it like auto complete. You don't blindly accept what auto complete tells you, your code would never compile otherwise, you read what it proposes to you and you modify what you don't like.

I get that, but I meant how often does it introduce bugs in the situation the OP mentioned, using it for "a language or framework you don't usually use", where I won't know if it looks right or not because I'm unfamiliar. Are they going back to fix the code often?

Personally, I don't even rely on normal autocomplete very much when I code because I feel like if I don't know what I'm trying to call, then I don't know the language or library well enough, and it's best to go look at the documentation.

I'm not sure, I use it A LOT for cryptographic code in Rust and I don't blindly rely on its output. I believe if there's a bug that was introduced it would be my fault with high probability, not Copilot's.

I recently used Copilot while learning Rust and it was immensely helpful for the learning process. There were a lot of language features that I learned about by seeing Copilot use them.

As for mistakes, I've got a good enough sense for reading code in any language to know if something is obviously wrong with an algorithm. The mistakes Copilot makes aren't usually to do with language-specific syntax, they're typically algorithmic and therefore easy to spot in any language. This makes sense if you think about GPT-3's output for English: it tends to be syntactically correct, but often completely wrong about facts.

It helps that Rust's compiler is so picky and so helpful, because on the rare occasions when Copilot was completely wrong about the language the compiler could set me straight pretty quickly. Writing automated tests also helped and was made easy by Copilot. I could write a single test case showing how to use the API, and then just write function names to generate the rest of the tests (obviously I would sanity-check the assertions Copilot produced).

Auto-complete generally completes a single identifier, and the possible completion list is usually restricted by the types involved. So the likelihood of it getting a wrong one is lower, but even if it does, it's easy to spot.

This is why I stopped using copilot. It was very good at spitting out code that looked correct enough that I didn’t spot the mistakes. And they were mistakes I’d never make myself if I wrote it manually.

Maybe don't use it for huge chunks > 10 lines? The point of copilot to me is avg typing speed is 40wpm and read is 250wpm. A 5.25x speedup considering even after I type code I still read it over. There will be errors in copilots code sometimes so even if you said its only 50% accurate thats still a 2.5x speedup on the boilerplate it's used for.

No more worried than in the past if I made the same mistake myself by manually typing it or by copying / pasting and leaving something off.

As with everything I always do visual sanity checks.

I've been writing a compiler, and I guess enough people have because it always knows exactly what my enums should be, what the name of my types should be, what I should parse next, etc.

Writing a simple compiler is a fairly common CS exercise.

Reply on news.ycombinator.com