Maybe, but you won't be able to test all behaviors and you won't have enough time to try a million alternatives. Just because of the number of possibilities, it'll be faster to just read the code.
I think we need to think outside the box here and realize ideas can be generated, evaluated, and settled upon far faster than any human operates. The idea of doing what a trillion humans evaluating different functions can do is actually realistic with the path of our present technology. We are at the cusp of some very remarkable times, even more remarkable than the innovations of the past 200 years, should we make progress on this effort.
A trillion is 8 symbols. You still haven't reached the end of your first import statement.
I just took a random source file on my computer. It has about 8000 characters. The number of possible files with 8000 characters has 12500 digits.
At this point, restricting the search space to syntactically valid programs (how do you even randomly generate that?) won't make a difference.
By using a grammar. Here is an example on how to only generate valid JSON with llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
> A trillion is 8 symbols. You still haven't reached the end of your first import statement.
Since LLMs use tokens from a vocabulary instead of characters, the number is likely somewhere in the lower billions for the first import statement.
But of course, LLMs do not sample from a uniform random distribution, so there are even fewer likely possibilities.
But I hope we have more efficient ways to do this in a century.