Testing a compiler-driven full-stack web framework

> Not only does one need to develop compiler, runtime, libraries, and IDE support (which is a tall order by itself)

CC can do that by itself in a loop, in ~3mo apparently. https://cursed-lang.org/

I know it's a meme project, but still it's impressive. And cc is at the point where you can take the repo of that language, ask it to "make it support emoji variables", and 5$ later it works. So yeah ... pretty impressive that we're already there.

Research takes some time, both to do but also to publish. In my area (programming languages), we have 4 major conferences a year, each with like a 6-to-8-month lag-time between submission and publication, assuming the submission is accepted by a double-blind peer review process.

I don't work in this area (I have a very unfavorable view of LLMs broadly), but I have colleagues who are working on various aspects of what you ask about, e.g., developing testing frameworks to help ensure output is valid or having the LLMs generate easily-checkable tests for their own generated code, developing alternate means of constraining output (think of, like, a special kind of type system), using LLMs in a way similar to program synthesis, etc. If there is fruit to be borne from this, I would expect to start seeing more publications about it at high-profile venues in the next year or two (or next week, which is when ICFP and SPLASH and their colocated workshops will convene this year, but I haven't seen the publications list to know if there's anything LLM-related yet).

It's not only the amount of code but also the quality of the available code. If a language has a low barrier to entry (e.g. python, javascript), there will be a lot of beginner code. If a language has good static analysis and type checking, the available code is free of certain error classes (e.g. Rust, Scala, Haskell).

I see that difference in llm generated code when switching languages. Generated rust code has a much higher quality than python code for example.

Do most people consider important for LLMs to be able to generate code for the language they use? I think I'd consider it a positive if they can't.

Just anecdotally, I'm more productive in languages that I know _and_ which have good LLM understanding, than in languages that I'm just experienced with.

As much as I dislike Go as a language, LLMs are very good at it. Java too somewhat, Python a fair amount but less (and LLMs write Python I don't like). Swift however, I love programming in, but LLMs are pretty bad at it. We also have an internal config language which our LLMs are trained on, but which is complex and not very ergonomic, and LLMs aren't good at it.

Not really, I’m into learning new languages but couldn’t care less about LLMs or IDEs.

And 99% of the time tooling isn’t built by the same person that builds the language compiler

if the abstractions are good, than the LLM has no problem writing the code. That's what we've noticed for Wasp at least. Its a simple config language, then the rest is react/nodejs, so it works surprisingly well.

Do note that we are moving away from the custom DSL part. We will move our configuration to TypeScript which:

1) Frees us from maintaining a DSL (parsing, LS, ..).

2) Uses something familiar to most web developers.

3) Actually expands our configuration features. E.g. with TypeScript we can change the configuration depending on env vars.

Though we want to keep the same level of abstraction as in DSL. Doing that with great DX is what we want to tackle.

We recently did explore Phoenix as an inspiration, and we really liked how docs work in the Elixir ecosystem. We can't really copy everything, but we could copy e.g. automatic docs generation for the API (and JS/TS has solutions for that that even Phoenix uses e.g. https://hexdocs.pm/phoenix/js/index.html). I love the fact that API docs and code comments can't get out of sync that way.

Still parts of the docs have to be handmade, and those are usually not supported directly by the ecosystem, so you have to build your own solutions.