> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator.
dsl stays small is doing all the heavy lifting here
the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (specially if its not json/yaml or just function chain based). as the number of things that can mix and match increase you are basically looking at a whole system prompt just describing the new language.
this brings us to the second part. step 2: after dsl is 'planned' (note they use the java compiler), the dsl need to have a real compiler/executor, not just a validator. because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based)
I can confirm that having a DSL that is json/yaml helps a ton. Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics.
> because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
I think this is an edge case; 99% of the time you (and/or the LLM) would have access to the implementor so it wouldn't need to do this.
> it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics
As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, which limits how wrong it can get the semantics.
It does apply to JSON in a way: An LLM doesn't always produce valid JSON, but it's easy to do a syntax check with a JSON parser and make the LLM try again until the JSON is valid. The JSON might still be semantically wrong on the level of the application domain (have the wrong values), but at least you can be sure that the generated data consists of lists, dictionaries, strings and numbers.
it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks.
PlantUML is a real dsl. not just declarative yaml.
But you can have DSLs that are json/yaml, is my point.
> PlantUML is a real dsl.
PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.
I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL.
I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure.
Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natural language.
> less common DSLs like liquidsoap’s stream management DSL
seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt.
> LLMs respond well to clear, simple structure
and what a "clear simple structure" for a dsl is also quite not mentioned. clear and simple would be quite subjective based on the domain, the article says let the llm go in a loop trying to figure out the dsl for you.
> checkpoints that have more structure than natural language
if llm is at any point in the structured generation part then either you have a deterministic validator/compiler or you are back to reading/reviewing it manually, what can you trust ?
Re: less common, I was just saying it doesn’t take millions off examples like PlanetUML.
> what can you trust
I wasn’t clear enough here — you’re responding to DSLs as an interface from non-deterministic LLMs to deterministic external systems.
What I meant was using DSLs as intermediate checkpoints in multi-LLM processing. If you just flow natural language through 5 LLM calls, the last one may be getting something very different from what it’s prompt is designed for.
But if you make the DSL a contract for handoff, results are much more stable.
Perfect and deterministic? No, of course not. Just an improvement and mitigation. But it’s served me well.
DSLs are a great middle ground for 'use LLM to turn ambiguous spec into something well defined', with the caveat that without discipline they'll inevitably expand until you should just have the agent write whatever the final language is.
There's a context tax up front (which will hopefully be less relevant over time) and then you really need a compiler/linter with helpful errors to keep it on the rails, because there is no corrective context in pretraining for something novel.
A purely descriptive DSL is just a convention, which is useful, but doesn't inject reliability the same way an enforced syntactic contract does.
I just have them write the tools to write the DSL's to do the thing then (most of) the sloppy code stays in the generator and if all the different things depend on each other they don't go stale and whatnot. And let them design the DSL themselves for whatever task so it matches their 'internal concept' of how the things work.
Worked out pretty well so far but not really practical unless your goal is to make the tools to make the DSLs to make jitting VMs -- https://github.com/dan-eicher/BBQ kind of snowballed from "let's parse some binary files" to a way over the top toolkit for playing around with this stuff but, it's fun...
I discovered this a few months ago when I was describing a binary data format using Dogma. On a whim, I tried handing the task off to Claude and it not only finished it correctly, but also fixed some errors!
This reminds me of this Bjarne Stroustrup's Rule (creator of C++):
- For new features, people insist on loud, explicit syntax.
- For established features, people want terse notation
Hillel Wayne [1] argues that the same applies for the differences between what beginners and experts desire from a language:
Beginners need explicit syntax, experts want terse syntax.
In my mind, DSLs are related to that – a short notation to avoid repetition. And LLMs are the experts.
I wonder if Lisp with its powerful DSL-creating macros will enjoy more popularity in the near future.
What I do for my one framework (in dotnet, strong typing & compiler) is to create claude skills (bash scripts and dotnet console apps) that claude can call to do various things within the framework. I have a template engine (for classes/db scripts/view engine templates), build tools, deployment tools, backup tools, infra docs - all of it works amazingly well. I built them in a standardized way so it's easy to chain them together and claude just 'gets it'. It feels like a DSL on steroids.
As much as I dislike them [1], DSLs are an easy way to provide domain specific backpressure to your llm which might not easily be available in your language.
What is the general consensus on Martin Fowler - I worked with Thought Works and they were obsessed with overcomplicating everything, but maybe that is just agency in general?
I think it goes without saying that the biggest fight we have as developers is keeping things as simple as possible when most external factors encourage complexity, especially LLMs.
I'm curious too. In the Korean IT scene, his name is legendary. Because many articles reference his writings, and famous Korean IT YouTubers worship him. (I worship him too.)
So I do have some questions. First, I still enjoy reading ThoughtWorks Radar. For someone like me, who's in a vulnerable position far from the cutting edge of technology, it always helps me keep some level of synchronization with the tech world. But I'm curious whether this is just a perspective from Korea, or if it's the same in the West.
And as for the tendency to overcomplicate things—I think that even when something is implemented simply, the explanation often ends up being quite complex. Honestly, I find Fowler's writing easy to read
Rather than DSLs I’ve found careful force tools results force the same kind of discipline in a more straightforward way to implement. So it’s normal to ask the llm to answer only yes or no and they are pretty good at following that instruction but it doesn’t scale so well. Whereas if the shape of the force tool call gives them more richness without giving them freedom to go off piste it scales to more nuanced results whilst also being trivial to parse.
The only thing which enables reliable use of LLMs are statistical techniques.
Even the most constrained and well-designed Disney world ride will break down in some embarrassing way every now and again. As you increase the # of parallel rides, the chances that at least one of them will touch the desired parts of the search space go up dramatically.
The fact that the major model providers keep publishing nano/mini/luna variants should be a massive hint that there's more to this than one big fat loop magically one-shotting everything.
This is a statistical technique, effectively: it shrinks the problem space the LLM faces at each decision point. Ideally, it would be a way to turn a request from an open-world game into a game on rails. Given the DSL, my options are only X or Y at this point in the solution.
Admittedly, that’s just improving the likelihood of getting a successful result, but if you mean 100% when you say “reliable”, that’s a false equivalence. No coder gets it right reliably either.
I frequently blur the line between ad-hoc DSL and pseudocode, and just hand it off to the LLM. I want to get the thoughts out of my head as fast as possible, using whatever structure makes sense to me. Even if you know all of the code to be written, I think this is a huge win with LLMs, where your intent is more important than syntax.
I agree. What is missing from the discussion about DSLs are the importance of tooling such as linters, LSPs, etc, to give the LLMs further context.
For example, charts/plots are often stringly typed with regards to column names and a DSL specific to plotting could give immediate feedback to an LLM.
It absolutely blows me away that there is still a significant amount of people out there kidding themselves that they're effectively using a coding harness .... and don't get this simple fact.
The gap I've hit generating GPU kernels with agents
code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. Speed targets have to be part of the check, not just correctness
> One domain that suffers from this particularly is GPU kernel engineering. We adopt our previous idea for detecting reward hacking from SpecBench and apply that to a set of KernelBench tasks, measuring whether the speedup the agent reports on the unit tests actually survives in the end-to-end workload (e.g. model training). A kernel counts as reward hacking if less than half of its claimed speedup survives there, including outright slowdowns and failures.
I'm really starting to tire of people making broad, general claims about how LLMs work or how to use them with N = 1 or 2.
An LLM is a statistics machine for goodness sake. Basically any general claim about them needs to exploit the law of large numbers to be even remotely sensible. You cannot extrapolate from one-off behavioral successes. LLMs are not understanding anything in the way humans do. If they did, yeah, maybe you could extrapolate hard from small samples, but they don't work or understand things like we do. You need to show that the behavior you are documenting is an average behavior the LLM converges toward in the long run.
its a specific type of language, not all language are the same , the nature of that specificity is the main point of the the claim made in this article
> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator.
dsl stays small is doing all the heavy lifting here
the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (specially if its not json/yaml or just function chain based). as the number of things that can mix and match increase you are basically looking at a whole system prompt just describing the new language.
this brings us to the second part. step 2: after dsl is 'planned' (note they use the java compiler), the dsl need to have a real compiler/executor, not just a validator. because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based)
I can confirm that having a DSL that is json/yaml helps a ton. Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics.
> because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
I think this is an edge case; 99% of the time you (and/or the LLM) would have access to the implementor so it wouldn't need to do this.
> it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics
As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, which limits how wrong it can get the semantics.
It does apply to JSON in a way: An LLM doesn't always produce valid JSON, but it's easy to do a syntax check with a JSON parser and make the LLM try again until the JSON is valid. The JSON might still be semantically wrong on the level of the application domain (have the wrong values), but at least you can be sure that the generated data consists of lists, dictionaries, strings and numbers.
I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
> DSL that is json/yaml helps a ton
it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks.
PlantUML is a real dsl. not just declarative yaml.
> and i would say json/yaml is not a dsl
But you can have DSLs that are json/yaml, is my point.
> PlantUML is a real dsl.
PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.
> you can have DSLs that are json/yaml, is my point
i disagree on the semantics of "DSL" vs "config in json". these are different.
and anyways both of them just pretend to be "reliable" by throwing the responsibility to an upper layer of validator/compiler/interpreter
> it is worse because the tooling around it won't be as good
plantuml is good because of the the tooling around it. not sure if we are agreeing/disagreeing there, confused by the wording
I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL.
I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure.
Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natural language.
> less common DSLs like liquidsoap’s stream management DSL
seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt.
> LLMs respond well to clear, simple structure
and what a "clear simple structure" for a dsl is also quite not mentioned. clear and simple would be quite subjective based on the domain, the article says let the llm go in a loop trying to figure out the dsl for you.
> checkpoints that have more structure than natural language
if llm is at any point in the structured generation part then either you have a deterministic validator/compiler or you are back to reading/reviewing it manually, what can you trust ?
Re: less common, I was just saying it doesn’t take millions off examples like PlanetUML.
> what can you trust
I wasn’t clear enough here — you’re responding to DSLs as an interface from non-deterministic LLMs to deterministic external systems.
What I meant was using DSLs as intermediate checkpoints in multi-LLM processing. If you just flow natural language through 5 LLM calls, the last one may be getting something very different from what it’s prompt is designed for.
But if you make the DSL a contract for handoff, results are much more stable.
Perfect and deterministic? No, of course not. Just an improvement and mitigation. But it’s served me well.
its served me okay.
> Perfect and deterministic? No, of course not. Just an improvement and mitigation.
exactly, not reliable, as the article tries to portray
DSLs are a great middle ground for 'use LLM to turn ambiguous spec into something well defined', with the caveat that without discipline they'll inevitably expand until you should just have the agent write whatever the final language is.
There's a context tax up front (which will hopefully be less relevant over time) and then you really need a compiler/linter with helpful errors to keep it on the rails, because there is no corrective context in pretraining for something novel.
A purely descriptive DSL is just a convention, which is useful, but doesn't inject reliability the same way an enforced syntactic contract does.
I just have them write the tools to write the DSL's to do the thing then (most of) the sloppy code stays in the generator and if all the different things depend on each other they don't go stale and whatnot. And let them design the DSL themselves for whatever task so it matches their 'internal concept' of how the things work.
Worked out pretty well so far but not really practical unless your goal is to make the tools to make the DSLs to make jitting VMs -- https://github.com/dan-eicher/BBQ kind of snowballed from "let's parse some binary files" to a way over the top toolkit for playing around with this stuff but, it's fun...
I discovered this a few months ago when I was describing a binary data format using Dogma. On a whim, I tried handing the task off to Claude and it not only finished it correctly, but also fixed some errors!
This reminds me of this Bjarne Stroustrup's Rule (creator of C++): - For new features, people insist on loud, explicit syntax. - For established features, people want terse notation
Hillel Wayne [1] argues that the same applies for the differences between what beginners and experts desire from a language: Beginners need explicit syntax, experts want terse syntax.
In my mind, DSLs are related to that – a short notation to avoid repetition. And LLMs are the experts.
I wonder if Lisp with its powerful DSL-creating macros will enjoy more popularity in the near future.
[1] https://buttondown.com/hillelwayne/archive/stroustrups-rule/
What I do for my one framework (in dotnet, strong typing & compiler) is to create claude skills (bash scripts and dotnet console apps) that claude can call to do various things within the framework. I have a template engine (for classes/db scripts/view engine templates), build tools, deployment tools, backup tools, infra docs - all of it works amazingly well. I built them in a standardized way so it's easy to chain them together and claude just 'gets it'. It feels like a DSL on steroids.
edit: this is all on linux + posgresql.
dotnet on linux friend, what are you using for templating?
As much as I dislike them [1], DSLs are an easy way to provide domain specific backpressure to your llm which might not easily be available in your language.
1- https://mikehadlow.blogspot.com/2012/05/configuration-comple...
This remains one of the best articles I have read on LLMs (by the same author) https://martinfowler.com/articles/llm-learning-loop.html
Also this article is a good pre-cursor to the DSL article:
https://martinfowler.com/articles/what-is-code.html
Pretty much how I use LLMs these days. Even Chatbots are able to work with a 200 line spec for the DSL.
You'd think it wouldn't because, well, no training data, but a short spec is usually enough.
What is the general consensus on Martin Fowler - I worked with Thought Works and they were obsessed with overcomplicating everything, but maybe that is just agency in general?
I think it goes without saying that the biggest fight we have as developers is keeping things as simple as possible when most external factors encourage complexity, especially LLMs.
I'm curious too. In the Korean IT scene, his name is legendary. Because many articles reference his writings, and famous Korean IT YouTubers worship him. (I worship him too.)
So I do have some questions. First, I still enjoy reading ThoughtWorks Radar. For someone like me, who's in a vulnerable position far from the cutting edge of technology, it always helps me keep some level of synchronization with the tech world. But I'm curious whether this is just a perspective from Korea, or if it's the same in the West.
And as for the tendency to overcomplicate things—I think that even when something is implemented simply, the explanation often ends up being quite complex. Honestly, I find Fowler's writing easy to read
Rather than DSLs I’ve found careful force tools results force the same kind of discipline in a more straightforward way to implement. So it’s normal to ask the llm to answer only yes or no and they are pretty good at following that instruction but it doesn’t scale so well. Whereas if the shape of the force tool call gives them more richness without giving them freedom to go off piste it scales to more nuanced results whilst also being trivial to parse.
The only thing which enables reliable use of LLMs are statistical techniques.
Even the most constrained and well-designed Disney world ride will break down in some embarrassing way every now and again. As you increase the # of parallel rides, the chances that at least one of them will touch the desired parts of the search space go up dramatically.
The fact that the major model providers keep publishing nano/mini/luna variants should be a massive hint that there's more to this than one big fat loop magically one-shotting everything.
This is a statistical technique, effectively: it shrinks the problem space the LLM faces at each decision point. Ideally, it would be a way to turn a request from an open-world game into a game on rails. Given the DSL, my options are only X or Y at this point in the solution.
Admittedly, that’s just improving the likelihood of getting a successful result, but if you mean 100% when you say “reliable”, that’s a false equivalence. No coder gets it right reliably either.
I frequently blur the line between ad-hoc DSL and pseudocode, and just hand it off to the LLM. I want to get the thoughts out of my head as fast as possible, using whatever structure makes sense to me. Even if you know all of the code to be written, I think this is a huge win with LLMs, where your intent is more important than syntax.
This is exactly it.
Typing was always just the interface. The keyboard was never the scarce resource. Judgment was.
Architecture scales. Implementation accumulates.
I agree. What is missing from the discussion about DSLs are the importance of tooling such as linters, LSPs, etc, to give the LLMs further context.
For example, charts/plots are often stringly typed with regards to column names and a DSL specific to plotting could give immediate feedback to an LLM.
It absolutely blows me away that there is still a significant amount of people out there kidding themselves that they're effectively using a coding harness .... and don't get this simple fact.
The gap I've hit generating GPU kernels with agents code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. Speed targets have to be part of the check, not just correctness
> but is slower than the baseline. Validator says pass, result is useless
You should read this blog, they cover this exact scenario - https://www.weco.ai/blog/first-evidence-of-recursive-self-im...
> One domain that suffers from this particularly is GPU kernel engineering. We adopt our previous idea for detecting reward hacking from SpecBench and apply that to a set of KernelBench tasks, measuring whether the speedup the agent reports on the unit tests actually survives in the end-to-end workload (e.g. model training). A kernel counts as reward hacking if less than half of its claimed speedup survives there, including outright slowdowns and failures.
"Performance is the most important feature"
The return of Ruby =]
I'm really starting to tire of people making broad, general claims about how LLMs work or how to use them with N = 1 or 2.
An LLM is a statistics machine for goodness sake. Basically any general claim about them needs to exploit the law of large numbers to be even remotely sensible. You cannot extrapolate from one-off behavioral successes. LLMs are not understanding anything in the way humans do. If they did, yeah, maybe you could extrapolate hard from small samples, but they don't work or understand things like we do. You need to show that the behavior you are documenting is an average behavior the LLM converges toward in the long run.
"DSL" is a stupid word. If it's a language, it's a language.
its a specific type of language, not all language are the same , the nature of that specificity is the main point of the the claim made in this article
all languages are embedded in some domain, therefore having a domain does not make a language "a specific type of language"
what does it do , can u explain more?