What's funny is that i often see Copilot give you code that Intellisense/a type system will tell you is obviously wrong. I wish Copilot was more integrated with type systems.
1. The surrounding code
2. The type/intellisense information of all the identifiers in the surrounding code
3. The class/function definition of any identifiers in the surrounding code
I think #3 will be huge. There is a huge difference in Copilot's performance when dealing with types defined right above where you are writing code, and types defined in some far-away module or library. Right now if you use a class you defined in another file or adjacent project, Copilot can't see it. But your IDE has jump-to-definition functionality that could trivially be wired in to something like Copilot.
The team behind Stable Diffusion (the image synthesis model that's been all over HN recently) have said they are working on stuff like Copilot as well, so I expect we will see a huge transformation in this space in the next year or two.
It feels like something that is obvious and should go in the copilot plugin rather than copilot itself.
Given N suggestions, validate the code + snippet on them using normal error checking.
Promote suggestions with no errors.
The intellij plugin, for example, will often suggest 'default' suggestions that are entirely wrong. Given that `foo.bar.foobar()` exists, it'll suggestion `foo.something.SomethingClever()` when those functions don't exist, or exist on types other than foo.
However, if you look at all the suggestions on the side panel, you'll see several of them are actually valid. It's just that the default completion isn't.