I disagree, to keep it short, what you're describing is not understanding, it's superstition. And I think it's a wrong direction of engineering, relying on some sort of irreproducible expert intuition, one that has been successfully replaced by enlightenment and scientific method.

There are 3 major obstacles in understanding LLMs:

1. They use inscrutable internal language of embeddings

2. They communicate in natural language which is itself ambiguous

3. The weights and training inputs are being hidden as a "trade secret"

"with the added advantage of having been trained on a HUGE number of codebases"

This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand. Without this understanding it's just a superstition.