We can discuss that scenario when it happens. Currently we aren't there and it isn't obvious to me that the current algorithm can get there.

> Hopefully, at one point you reach GPT-3 level performances where it is able to imagine programs for tests it never saw.

You mean nonsense programs just like GPT-3 generates nonsense articles? GPT-3 doesn't remember the logic in its sentences, and in order to solve programming competition problems you need to translate logic from human text into code.

I agree that it might be possible to get something useful this way, but until it actually works I'll doubt it will work. There is just way too much coherence required that doesn't seem to be there yet, and from what I've seen the coherence problem gets exponentially worse as you get larger problems.