Expecting to go from having seen no code to being able to code within a context window is a pretty high bar. More useful would be asking for code that does something you're pretty confident no-one has done before.

(TBH, when people claim this, I do wish there would at least be occasionally an actual pointer to something that was copied. I understand it's probably not going to be doable in all cases, but without examples it sure feels like a weak statement. In my experience the recent models are good at doing things I'm pretty sure there is not a close reference for in the training data, though for most part I wouldn't classify them as particularly difficult tasks either)

Thanks for the elaboration. (Though I'm not sure dmarinus would agree with your criterion?)

> For example, remove all code from your training data, and then see if the model can code regardless.

Would you accept this weaker version: make up a new programming language (that's guaranteed not to be in the training set), and see if the model (which has trained on programming in existing languages) can cope?