DSLs Enable Reliable Use of LLMs
martinfowler.com
martinfowler.com
dsl stays small is doing all the heavy lifting here
the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (specially if its not json/yaml or just function chain based). as the number of things that can mix and match increase you are basically looking at a whole system prompt just describing the new language.
this brings us to the second part. step 2: after dsl is 'planned' (note they use the java compiler), the dsl need to have a real compiler/executor, not just a validator. because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure.
Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natural language.
seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt.
> LLMs respond well to clear, simple structure
and what a "clear simple structure" for a dsl is also quite not mentioned. clear and simple would be quite subjective based on the domain, the article says let the llm go in a loop trying to figure out the dsl for you.
> checkpoints that have more structure than natural language
if llm is at any point in the structured generation part then either you have a deterministic validator/compiler or you are back to reading/reviewing it manually, what can you trust ?
> what can you trust
I wasn’t clear enough here — you’re responding to DSLs as an interface from non-deterministic LLMs to deterministic external systems.
What I meant was using DSLs as intermediate checkpoints in multi-LLM processing. If you just flow natural language through 5 LLM calls, the last one may be getting something very different from what it’s prompt is designed for.
But if you make the DSL a contract for handoff, results are much more stable.
Perfect and deterministic? No, of course not. Just an improvement and mitigation. But it’s served me well.
> Perfect and deterministic? No, of course not. Just an improvement and mitigation.
exactly, not reliable, as the article tries to portray
like if someone says "is this car reliable", i dont expect an infallible car. but if the cars third gear is 'you know sometimes it doesnt work', i wouldnt take that car out of city.
If I have a great meal every time I go to a restaurant, that chef is reliably good. I do not require that the tomatoes on the salad be of identical number or placement, or cut along the precise same bias. Nor would I count the sesame seeds on the bun.
Are LLMs reliable by that standard? Sometimes yes, for some tasks. The envelope of tasks for which they're reliable is continuing to grow.
I'm really not sure this is true. Recently for work I've been making changes in a system written in tcl in the early 2000s, including a custom dsl that has never escaped this company. The LLMs can write it fine. It wrote it almost acceptably on first sight, an hour or two of targeted test cases to extract a one page functional spec of the subset we needed and that was it. They are quite good at finding and fitting to patterns, go figure.
you are probably talking of some coding harness which looked up the existing code base and then made it and not something like first prompt to llm "write xyz testcase in my systems company dsl in tcl from early 2000".
> LLMs can write it fine
sure, coding by examples is fine (it goes back to "system prompt describing the language"). but the claim we are arguing is reliability. did the llm generate 100% test case or code after reading your existing codebase, likely not as you mention it "almost acceptably on first sight".
i would probably be in denial if was suggesting llms not good at finding and fitting patterns.
I can confirm that having a DSL that is json/yaml helps a ton. Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics.
> because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
I think this is an edge case; 99% of the time you (and/or the LLM) would have access to the implementor so it wouldn't need to do this.
it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks.
PlantUML is a real dsl. not just declarative yaml.
But you can have DSLs that are json/yaml, is my point.
> PlantUML is a real dsl.
PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.
i disagree on the semantics of "DSL" vs "config in json". these are different.
and anyways both of them just pretend to be "reliable" by throwing the responsibility to an upper layer of validator/compiler/interpreter
> it is worse because the tooling around it won't be as good
plantuml is good because of the the tooling around it. not sure if we are agreeing/disagreeing there, confused by the wording
semantics are defined by the converter/compiler/interpreter, and that is the process which is going to consume the said json/yaml. if the json/yaml is going to be consumed by any process then the semantics are inherently defined, so by your definition all jsons/yamls are in their own a "language" (or they are not being consumed at all), which just defeats the purpose of calling it a language at that point.
As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, which limits how wrong it can get the semantics.
It does apply to JSON in a way: An LLM doesn't always produce valid JSON, but it's easy to do a syntax check with a JSON parser and make the LLM try again until the JSON is valid. The JSON might still be semantically wrong on the level of the application domain (have the wrong values), but at least you can be sure that the generated data consists of lists, dictionaries, strings and numbers.
it reads like code and you can easily define a schema too
well that's what i am using
and https://crates.io/crates/knus/ to define the schema
actually more than that. it would be awesome to have a programming language whose code is also valid as kdl 2. like lisp code being composed of s-expressions
There's a context tax up front (which will hopefully be less relevant over time) and then you really need a compiler/linter with helpful errors to keep it on the rails, because there is no corrective context in pretraining for something novel.
A purely descriptive DSL is just a convention, which is useful, but doesn't inject reliability the same way an enforced syntactic contract does.
Worked out pretty well so far but not really practical unless your goal is to make the tools to make the DSLs to make jitting VMs -- https://github.com/dan-eicher/BBQ kind of snowballed from "let's parse some binary files" to a way over the top toolkit for playing around with this stuff but, it's fun...
whats wrong with this? You may be over-indexing on the need for large quantities of examples. These days self-play through RL is far more effective and data (not compute) efficient.
if dsl gets large, the number of potential interactions your dsl allow will grow exponentially (unless you are building an one dimensional action layer). and there will be semantic issues unless your dsl is "clear and intuitive", also comprehensive enough to accomodate your ongoing changes, else every change is now 2 changes.
so you now you need a comprehensive manual for your agent which needs to be sent in every /completion request.
It is not based on JSON or any other out-of-the-box format. The only source material for this DSL anywhere in the world is on my laptop: a little documentation, a few examples, a partial implementation.
This is enough. Codex can not only give me arbitrary examples in this novel DSL on demand, it can see what I'm trying to do with the compiler and extend it for me.
I think it goes without saying that the biggest fight we have as developers is keeping things as simple as possible when most external factors encourage complexity, especially LLMs.
So I do have some questions. First, I still enjoy reading ThoughtWorks Radar. For someone like me, who's in a vulnerable position far from the cutting edge of technology, it always helps me keep some level of synchronization with the tech world. But I'm curious whether this is just a perspective from Korea, or if it's the same in the West.
And as for the tendency to overcomplicate things—I think that even when something is implemented simply, the explanation often ends up being quite complex. Honestly, I find Fowler's writing easy to read
That wave included
1. TDD by beck 2. spring framework 3. rails ( later ) 4. Agile manifesto 5. refactoring by fowler 6. gang of four design patterns
That said. Thoughtworks is a moneygrab that tried to cash in on fowler brand. I worked with them at sears ( worked closely with author of article) and siemens. they were no different from any other consulting firms that try to overcomplicate things so they can deploy more warm bodies to the project.
The actual implementation details only matters at the coding stage. At the design stage, it’s better to define the glossary and its semantic.
You'd think it wouldn't because, well, no training data, but a short spec is usually enough.
1- https://mikehadlow.blogspot.com/2012/05/configuration-comple...
For example, charts/plots are often stringly typed with regards to column names and a DSL specific to plotting could give immediate feedback to an LLM.
Hillel Wayne [1] argues that the same applies for the differences between what beginners and experts desire from a language: Beginners need explicit syntax, experts want terse syntax.
In my mind, DSLs are related to that – a short notation to avoid repetition. And LLMs are the experts.
I wonder if Lisp with its powerful DSL-creating macros will enjoy more popularity in the near future.
[1] https://buttondown.com/hillelwayne/archive/stroustrups-rule/
An LLM is a statistics machine for goodness sake. Basically any general claim about them needs to exploit the law of large numbers to be even remotely sensible. You cannot extrapolate from one-off behavioral successes. LLMs are not understanding anything in the way humans do. If they did, yeah, maybe you could extrapolate hard from small samples, but they don't work or understand things like we do. You need to show that the behavior you are documenting is an average behavior the LLM converges toward in the long run.
> understanding anything in the way humans do
i'm not sure it's clearly established LLMs can't be a model of some part of “the way humans do”?
to be more specific, i'd argue LLMs “understand” awfully similarly to a brilliant (polymath) with early dementia or Alzheimer's
no executive function, no short term memory, and absent both of those, conversing with that person about the past or with an LLM about topics that had been "in their training sets before a cutoff date" is surprisingly similar, right down to: introing a topic precisely the same way, you'll experience the same conversation; convo loops if bits are too quantized (looping and lossiness / recall / context-length are correlated); and ofc opening a new session is like the first one never happened
At which point i'd argue that you're kind of refuting your own claim as most humans are not brilliant polymaths with early onset alzheimer's.
Even admitting these kinds of comparisons to edge case human mental experience, there are more differences than similarities, and the similarities are misleading and superficial. There are still a lot of differences, even when it comes to the physical structure of the brains neurons as compared to digital neural nets, and i think it's far more beneficial to try to understand these machine in their uniqueness and for what they are than to draw hasty and shallow comparisons. You could be forgiven for that, though, because computing is rife with people who love to draw hasty unjustified analogies for some reason (example: people were already likening the brain to a computer when we didn't even have working implementations of neural nets yet and a computer was literally just a small number of logic gates lol)
Actually the tool-calling convention is a small DSL too.
I suspect the DSL ideally has to be similar to something in the LLM's training set though.
It doesn't have to feel too alien, otherwise the description of it has to be thorough and will eat up a lot of context.
Frontier models don't suffer as much of this limitation since they can grab onto a larger corpus of knowledge, but they're expensive.
Also this article is a good pre-cursor to the DSL article:
While the DSL is more formal than natural language, it’s not what we’re communicating to the LLM with, so it’s advantages are washed away. And typical code is more strict/rigorous than DSLs so I think that’s why I see worse results, because a typical languages compiler “catches” more mistakes, versus a DSL that’s easy to write but has lots of implicitness.
I’ve had the same journey experimenting with levels if abstraction too. Going lower, and exposing the LLM to the “full-stack” works much better than trying to build up abstractions it can’t see into without extra steps.
I don’t want to be too much of a hater, but these types of panacea/architecture posts are usually written by people who don’t work in the field, lack pressure or constraints, and get paid to goof around in castles of the mind. I would simply skip over it and hold my comments/opinions to myself, but they tend to have an outsized influence on software engineering practices.
One line of thinking can be that frontier models are already powerful enough to brute force through it, but this might be a stronger indicator for success for smaller models.
Asking for simple changes to SwiftUI interfaces I’ve one-shot is always a frustrating experience for me
1) Have you exported Apple docs from Xcode and provided those to your LLM and told your CLAUDE.md to read them as applies?
2) Have you added https://github.com/dpearson2699/swift-ios-skills to your agent?
I'd also recommend Xcode 27 beta unless you teach your agent the Xcode MCP. Xcode 27 beta works on MacOS 26.x.
You should read this blog, they cover this exact scenario - https://www.weco.ai/blog/first-evidence-of-recursive-self-im...
> One domain that suffers from this particularly is GPU kernel engineering. We adopt our previous idea for detecting reward hacking from SpecBench and apply that to a set of KernelBench tasks, measuring whether the speedup the agent reports on the unit tests actually survives in the end-to-end workload (e.g. model training). A kernel counts as reward hacking if less than half of its claimed speedup survives there, including outright slowdowns and failures.
edit: this is all on linux + posgresql.
Arent these all in training data too?
Even the most constrained and well-designed Disney world ride will break down in some embarrassing way every now and again. As you increase the # of parallel rides, the chances that at least one of them will touch the desired parts of the search space go up dramatically.
The fact that the major model providers keep publishing nano/mini/luna variants should be a massive hint that there's more to this than one big fat loop magically one-shotting everything.
Admittedly, that’s just improving the likelihood of getting a successful result, but if you mean 100% when you say “reliable”, that’s a false equivalence. No coder gets it right reliably either.
Typing was always just the interface. The keyboard was never the scarce resource. Judgment was.
Architecture scales. Implementation accumulates.