I don't know ... the video at least is straight up bullshit, right? The terminal at the bottom of the editor session shows that the user has already run "copilot_solution_puzzle_two.py" with the desired answer before he types anything.
This is also something I'm curious about regarding Copilot. I mentioned that "GitHub Copilot has a list of possible solutions to code completions, so I wonder if it’s just luck that the suggested solution was the correct one".
I'm going to test further to see if it just always produces the same code completion, or there's some bit of randomness to the completion.
The Codex model on which Copilot is based had about 30% accuracy on the first solution to a coding problem, but 70% when it was allowed to generate 100 solutions and choose the one that passed unit tests. On the other hand, when the best-of-100 solution was chosen according to the probability assigned to it by the model it scored 45% [1]. So it's kiind of luck.
Basically, Copilot has no way to tell whether it's giving you a right or wrong answer, other than to select the answer with the highest probability according to its training set which usually means the most common answer in its training set. So the probability that the answer you get is the "right" answer depends on the probability that the right anwser is the most common answer to the problem you give it. If that makes sense?
__________
[1] https://arxiv.org/abs/2107.03374
See Figure 1.
While I am not 100% sure of the sources, my use of Copilot makes me pretty sure it uses other open files in the editor, other files in the current project folder (whether or not open in the editor), and to suspect it may use the past history of the current file (at least in the same edit session).
I think it's more simple to assume that "get_*_input" is a common name for a function that reads input from a stream and so that this kind of string is common in Copilot's training data. Again, remember: GPT-3. That's a large language model trained on a copy of the entire internet (the CommonCrawl dataset) and then fine-tuned on all of github. Given the abundance of examples of code on the internet, plus github, most short progams that anyone is likely to write in a popular language like Python are already in there somewhere, in some form.
The form is an interesting question which is hard to answer because we can't easily look inside Copilot's model (and it's a vast model to boot). The results are surprising perhaps, although the way Copilot works reminds of program schemas (or "schemata" if you prefer). That's a common technique in program synthesis where a program template is used to generate programs with different varaible or function names etc. So my best guess is that Copilot's model is like a very big database of program schemas. That, as an aside.
Anyway I don't think it has to peek at other open files etc. Most of the time that would not be very useful to it.
> GitHub Copilot uses the current file as context when making its suggestions. It does not yet use other files in your project as inputs for synthesis. [1]