If I were to turn around and resell it, then you could sue me; but that wouldn’t be a chat bot’s fault.
Just like it’s not the clipboard’s responsibility to ensure I’m not violating licenses, I don’t think it’s copilot’s responsibility either. Use it at your own risk.
That is definitely not how US copyright law works (although you are welcome to argue that it should), that there is such a bright line around selling. It's possible for something to be a copyright violation even if you give it away for free (see torrent sites!), and it's possible for to qualify as fair use and not be a violation even if you sell it.
Under USA copyright law (others are similar but not quite the same), the first step would be deciding if the output of this chatbot counts as a "copy" or "derivative work" at all. If it does not, then there is no copyright violation whether you sell it or not. If it does, then it is a copyright violation (whether you sell it or not) unless it's use can count as "fair use". Whether the use is "commercial" is just part of one of four factors that are balanced to determine if it's fair use. For instance, if you are only using a tiny portion of the copyrighted work and it doesn't have much effect on the profits of the original copyright holder and the use is considered highly "transformative" too (sound like copilot?) -- it could well be fair use even if you are selling your output.
(Also... Github Copilot is literally selling it, right? You have to pay them to use Copilot! It would accordingly be considered a commercial use. If I make a copy of a hollywood movie and sell it to you, I'm probably violating copyright (unless I can convince a court it's fair use), regardless of what you do with it, it doesn't matter if you re-sell it or not. If you re-sell it or make another copy, you may be additionally violating copyright another time yourself.)
I do think there's a reasonable argument that copilot is fair use. It wouldn't mainly hinge on whether the output is sold or not. This is presumably the argument MS/Github would make if brought into court. Since Oracle v. Google, I've stopped trying to predict what courts will do on software-related copyright cases, the law seems to be pretty chaotic and the actions of courts unpredictable. (also i'm not a lawyer this is not legal advice).
In general, I am in favor of an expansive bounds of fair use, and think it serves "the people" to have such.
A FOSS license doesn't mean it's all fucking freebies all the way down as any big tech company is quick to remind you (by attaching trademarks all over the place, for example), but if it's a big tech company taking your stuff and running with it, it's suddenly all fair game.
All in all, it's how companies have been behaving since about forever. Fuck little people all the way.
I have not heard of what people have already been discussing about copyright legal issues around corpuses of Other People's Content, I am curious to read more if anyone has a link.
While I understand that in this case it seems like (or is!) a Big Corporation taking advantage of the Little People -- I would urge extreme caution in advocating for reducing and limiting "fair use" rights because in this case it will hurt the Big Corporation. It used to be clear to everyone that fair use helped the Little People against the Big Corporation copyright holders and should be encouraged. Lately, people (especially software devs, who of course write IP) have been excited about strenghtening protections for copyright in order to somehow limit the Big Corporations (see all the stuff around Amazon and open source), but this is a dangerous game. Fair use is one the only tools we have protecting us from a dystopia where you can't open your mouth or type on your keyboard without paying someone a license fee, which is what OTHER Big Corporations would love to see.
However I don’t quote verbatim which is what this tool has been doing.
Plenty of trigrams exist in many novels that are exactly the same, and i bet that there's plenty of n-grams in programming which would be construed as copies of each other.
The nature of using this tool could hang you in theory more than if you didn’t use it.
In several countries, humming a song - reciting a melody - is cut and dry de jure copyright infringement. It's just not rigorously punished.