It gives bad advice to users, from which it learns that that advice didn't work, and only the good stuff gets used going forward (may take some time to retrain and incorporate user feedback). I expect a proliferation of advice that works well enough to convince you that your problem is solved, but which might be subtly broken in ways that you have to come back to.

Tracing the fix-it sessions backwards in time to the broke-it sessions (such that the AI can learn from its mistakes) might be tricky unless we somehow manage to connect copiloting sessions through git history. Imagine a big database of changes where the AI is waiting to see whether the code persists, or if somebody comes along and fixes it.

vscode is owned by microsoft and could send runtime results back.

so is github...