DSLs are costly. My concern is that the cost will outweigh the benefit and people won't like it and we'll be suck with it. My hope is that it'll stabilize and be expressive enough to fit everybody's needs.
Did you ever write a DSL used in a production environment? How complex was the DSL, more declarative or closer to a general purpose language?
Chances are that you're severely over-engineering it. Just take the English specification for any particular problem your DSL is supposed to solve and slightly tweak it into looking like a formal language. That's it. Extremely easy with a bit of practice.
But that's probably because I don't think using templates for code generation is good enough. At least if you want to do something interesting with it.
Language workbenches cut down the cost of DSL design to minimum, but more interesting problem is providing valuable output from it.
Mind explaining why? What can be costly and complex in a chain of trivial transforms, each being very simple, flat and comprehensible?
So an example of the problem I deal with is a database migration. Let's say we have an entity with a value
entity SqlTable { List<TableColumn> nestedTuple; }
value TableColumn { int i; }
and if I change column type in value object to long I want my compilers to prepare a DB migration with the appropriate SQL statements for the specific DB I use. Of course, there is nothing too complex about it, but it's not trivial either (in this case you have to prepare a second field, unnest the whole hierarchy to get to the nested field, copy it to new type and compact the hierarchy back again).
I find it costly since there are gazillion of such features. And when they start interacting with each other things gets messy.
If you're using a meta-language, there is no difference between external and embedded DSLs. External (or standalone) DSLs also should be built exactly the same way, on top of a hierarchy of existing DSL components.
In your case, you're likely not compartmentalising your DSLs properly, trying to stuff too much functionality into a single DSL while it must be a chain of different DSLs. E.g., one DSL for describing a schema of the DB you want to migrate (with a tool for inferring it from the existing DB schema), another for representing the intermediate uniform data format and transforms over it, and a third for mapping the uniform intermediate representation on your target DB schema. Of course I do not know any details of your particular problem, just describing how I solved a DB migration problem before. Your specifics could add more to the DSL chain design.
Also, there are no tools for describing DB schemas that way (except if you consider DB DDL such a schema). So my DSL is used as uniform data format. And it's not a problem of mapping between formats, but within the logic required to do such a mapping. It can't be expressed as a simple transformation, compiler is required to analyze and transform it appropriately in various scenarios. And if you want optimizations, good luck with "simple mapping".
So yeah, it's complicated, but it needs to be complicated to support simple modeling DSL. Otherwise you are better off with having several DB schemas, various POOs, Protobuf/Flatbuffer IDL etc...
That's the problem here. 'DSL' is a vague and ambiguous notion. An ad-hoc 'fluent interface' usually isn't considered a DSL.
It's not that challenging if you do it the right way.
> DSLs are costly.
Plainly wrong. DSLs are the easiest and cheapest way of eliminating complexity. You simply have to follow a proper discipline and use the right tools.
When I build a DSL, it's rarely more than 100 lines of code including literate comments, and the resulting DSL is a proper optimising compiler with debugging support, and this new DSL is immediately supported by an IDE (at least with syntax highlighting, autoindentation and autocompletion).