I don't know much about AI/SL but I think the logical outcome is that using a transformer to generate source code is going to result in an over-trained system that produces sections of code verbatim because of the relative size of the space of all valid programs vs the space of all possible programs. It's not like art where you get soft failure if a single pixel or word is wrong: the inclusion, exclusion, or replacement of a single instruction or symbol is enough to introduce fatal bugs into a computer program. If the system doesn't have the capacity to understand programs generally (which it probably won't if it's just a transformer), then you're going to end up with a system that spits out samples from training data that do work.
ahem napster, pirate bay ...
In 999 out of a 1000 cases it’s just spitting out boilerplate though.
If someone is motivated to search through our entire (proprietary, private) codebase. They match it with repositories that are freely available. They’re properly motivated to make a problem out of it (some twitter randos?), and most importantly they gain some benefit out of spending hundreds of thousands of dollars engaging with our legal team.
By the time you satisfy all the conditions required for it to be an issue you are talking nation-state actors.