Awesome first step. Next is to figure out how to apply syntax trees for diffs and then train the LLM on code and diffs but all in syntax trees somehow.
Edit: Here's some of the papers: https://arxiv.org/abs/1911.09983 and https://aclanthology.org/2021.findings-acl.384.pdf
I’ll try to explain something I’m thinking, it comes down to a type of agglutination.
https://en.wikipedia.org/wiki/Agglutination
The ASTs need to become sequences of tokens for LLMs to work well. Also the embedding space is related, this should all be specifically optimized for code, not based on general human language .
Of course the description of the code would be english. But my point is that a subspace of the encoding should be specifically designed for an agglutination based sequential expression of ASTs.
Not sure any of that makes sense to someone with more expertise in this space than me.