If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
I'd love to do research on this when I have the time.
That's what I was insinuating through "better encoder"; the model creating more efficient representations of ASTs using something like JEPA
we do think there's a lot of potential though and do want coding themed releases soon
As you said, if it worked, they would have demoed it haha
youd ask it to pick a location on the ast to add something from the grammar?
i dont see how this stays confined well enough? make a new output space every time? does that end up auto-regressive?