Benefits of Writing a DSL in Ruby | Hacker News Reader

Benefits of Writing a DSL in Ruby

engineering.zenpayroll.com

80 pointsedawerd34 comments

For any struggling to understand Ruby's metaprogramming, the rather short and succinct Metaprogramming Ruby is an excellent read. There's a method to the madness of define_method, method_missing, the various eval methods, the difference between Blocks, Procs, and Lambdas. Most of it revolves around manipulating scopes.

It also shows how you can iterate from the ugliest, most powerful, most dangerous eval method by pulling out the functionality you need into less-powerful, safer manipulations. It's all Ruby code until it's actually executed, so it can all be manipulated at runtime.

http://www.amazon.com/Metaprogramming-Ruby-Program-Like-Pros...

Yup this is great stuff, just be sure to get the 2nd edition which just came out:

http://www.amazon.com/Metaprogramming-Ruby-Program-Like-Face...

It focuses on modern Rubies, so it talks about Module#prepend and such. Nice to keep moving forwards.

Seconding. The book revolutionized my understanding of Ruby, and inspired me to create a whole visual novel framework: https://github.com/thirdtruck/rubyai.

While I agree with all of the benefits of DSL writing, I kind of feel like DSLs in Ruby are a bit silly now. There is always a thin line between "lots of function abstraction" and DSL (e.g. a shallow embedding is pretty tough to distinguish from... well, just a library) and that line is even thinner in the fast-and-loose world of Ruby DSLs.

It's a vague enough question to lead to nothing but religious wars, but I really want to push people who are interested in (e)DSLs to consider their implementation in typed languages. Haskell is, of course, a favorite of mine, but you can achieve similar things in Scala and the ML family. You can even go much further using something like Idris, mostly by using its stronger dependently typed system.

Haskell is not very suitable for the efficient and straightforward eDSL implementation. Template Haskell is a nice thing, but it is staged (cannot use the template definition from the same module) and enforces explicit template syntax. And the usual Haskell DSLs are in fact all just ad hoc interpreters, missing all the benefits of having an underlying compiled runtime environment.

And typing is not a big deal, once you have a proper metaprogramming, you can easily add typing on top of your underlying language.

You can take advantage of the Haskell compiler with a shallow embedding, you can do your own compilation steps with a deep embedding, you can do partial evaluation to speed up compilation.

You can introduce your own type system, of course, but it's really nice to have most of that infrastructure pre-built for you (otherwise why embed?).

The only reasonable ways of implementing fully compiled DSLs in Haskell are standalone DSL compilers (e.g., as in Happy) or Template Haskell (restricted syntax, stages, limited reflection).

Both ways are not as convenient as with the more metaprogramming-oriented languages, plus the lack of reflection limits the potential for integrating DSL into the host language and the other DSLs. Yet, TH is really nice. I cannot stand languages without any metaprogramming capabilities at all, and TH just ticks the box for me.

Most of the benefits of DSLs can be achieved without "full compilation" I strongly believe. DSLs are great, eDSLs are a great intermediate sweet spot.

Compilation is much simpler than interpretation, especially an ad hoc one, in a Haskell or Ruby idiomatic style.

Compiled DSL implementations are much more declarative - they simply define how a DSL is mapped to the other DSLs or a host language.

Compiled DSLs are more efficient - you don't want an interpreted regexp engine, don't you?

Compiled DSLs are IDE- and static-analysis-friendly, at no additional cost.

Compiled DSLs are more modular and reusable - by sharing the same runtime environment, they can be easily mixed together, their fine grained properties can be picked individually and mixed into new DSLs.

Having said that, I see no single advantage of the interpreted DSLs.

I simply disagree with all of that, or at least partially with the 3rd point. But I also don't see the point in continuing this argument. Sorry.

If you can provide me with any evidence that interpretation can be more efficient than a static compilation outside of the marginal cases, I'd be very grateful.

I did a lot of research in tracing JITs, runtime partial specialisation and all the related stuff, but so far there is no practical and useful implementation of any of such techniques which could beat a mere straightforward static translation.

I never claimed it could be more efficient. I'm completely in agreement that compiled DSLs are more capable. I'm even a big fan of eDSLs which produce compiled output. I don't find completely static, compiled exterior DSLs to be as frequently useful, however, due to the massive increase in maintenance cost---this holds especially true when you have a sufficiently expressive type system to embed the type system of your eDSL into.

Then I did not understand your reference to the 3rd point.

I'm not talking about fully standalone compiled DSLs (totally agree they're usually unmaintainable) - I was referring to the metaprogramming-based compiled eDSLs, implemented as macros on top of a host language (or a range of host languages). This way it's really easy to mix traits into your DSL design, including various type systems, not even necessarily directly compatible with the host language type system.

I found it very productive to have Prolog (or anything comparable) as one of the host languages, it's trivial then to map most of the practical type systems to it.

Wow. That was an amazing article. I have been doing Rails/Ruby for years and never understood any of that. Thanks for putting that together, please consider doing more tutorials for other interesting things going on in ZenPayroll.

That's a great writeup, well done.

Along the same lines, here's the ever-entertaining Rich Kilmer talking about DSL experiences on some government projects:

http://www.infoq.com/presentations/kilmer-ruby-dsls

I don't get it. How is this a DSL?

Programmer with a couple of decades' experience here, but no ruby. I have no idea what half the tokens in the example mean, or what changes would be valid beyond editing the four obvious values.

"DSL" is slang for "API" in the ruby community [0].

[0]: https://gist.github.com/geeksam/24ef10be8c773a2c1bd4

That seems like unnecessary use of metaprogamming and DSLs in my opinion[0]. That problem could be solved with good ol' object-orientation.

Also, capturing blocks and instance_eval are notoriously slow.

0 - https://www.youtube.com/watch?v=yuh9COzp5vo

1 - https://speakerdeck.com/sferik/writing-fast-ruby

> Also, capturing blocks and instance_eval are notoriously slow.

...and, in this case, are evaluated once, at startup. It's so fast that you couldn't possibly notice a difference.

A good read for anyone just starting out with some of the more intricate parts of Ruby.

I often think Ruby is unsurpassed for writing readable, English-like DSLs - perhaps because of the combination of the convenient lassitude of Ruby's syntax (not requiring brackets - in many cases - or semi-colons) and the dangerous power of its metaprogramming abilities. The only other language that perhaps comes close and immediately springs to mind, before I've had my first coffee of the day, is (appropriately) CoffeeScript. Any other takers?

on the jvm i would say groovy is probably one of the better languages for dsl's. Gradle is a dsl and groovy has inbuilt Builder classes to help you in the creation of dsl's

Clojure being a lisp is extremely powerful in this regard as well, but if you are programming in any lisp I would expect a fair amount of meta programming.

Languages with expressive type systems, Haskell likely the most mainstream, allow for deep embedding of an DSL, such that the type system of your DSL can be embedded into the parent language.

IMHO absence of static analysis is a huge downside of using an interpreted language. Having type safety in an embedded DSL can only be dreampt of in a language like ruby.

http://tom.lokhorst.eu/2009/09/deeply-embedded-dsls

What does typing have to do with interpreted vs. compiled? Hugs is an interpreter.

And it's not a problem to build typed eDSLs on top of an untyped core language - see Typed Racket or Shen for example.

Lisp provides a couple really powerful DSLs. Some of more easily understandable ones I can think of:

Loop: http://www.gigamonkeys.com/book/loop-for-black-belts.html

Symbolic Manipulation: http://norvig.com/paip/macsymar.lisp

Generating HTML: http://allegroserve.sourceforge.net/aserve-dist/doc/htmlgen....

> Any other takers?

Well, obviously on the type safe DSL front nothing compares to Scala or Haskell.

Groovy's MOP was a bit buggy/lacking when I was using it (this is as of 3 years ago mind you, not sure how Groovy MOP has evolved since then) but I recall Gradle and Spock being impressive DSLs, very natural/readable.

> Spock being impressive DSLs, very natural/readable

The Spock syntax isn't a DSL, it hacks the ossified restrictions of the Antlr 2 based Groovy syntax to pull off tricks like overloading the | operator to give the appearance of tables, and the break/continue labels via an AST plugin to replace function calls with sections, as in this oft-quoted example from their website:

    expect:
    Math.max(a, b) == c
    where:
    a | b | c
    1 | 3 | 3
    7 | 4 | 4
    0 | 0 | 0

Ruby eDSLs are not that interesting - they still force you to stick to the Ruby syntax, which can be counterproductive.

The best way of implementing eDSLs is to extend the syntax of your core language, which is very easy for the languages based on PEG. See https://github.com/combinatorylogic/mbase and https://github.com/combinatorylogic/clike for example of this approach.

There is a list of (some) languages here - http://en.wikipedia.org/wiki/Dialecting (NB. I would add Factor, Haskell, Io and Perl6 to that list)

I personally love how easy it is to write lucid & robust dialects in Rebol. Some examples of Rebol dialects...

* List comprehension - http://blog.revolucent.net/2009/04/dirt-simple-dsl-in-rebol....

* Cron/scheduler - http://softinnov.org/rebol/scheduler.shtml

* Excel - http://www.robertmuench.ch/development/projects/excel/dialec...

* PDF builder - http://www.colellachiara.com/soft/Misc/pdf-maker-doc.pdf

* GUI example (written in Rebol 2 VID) - https://news.ycombinator.com/item?id=7070349

* And a small ditty that I left on Reddit comment just to show what can be done - http://www.reddit.com/r/programming/comments/1tw17i/rust_is_...

Great list. Thank you!

If you would like to see an example for a command line invoice tool I wrote, have a look at this:

https://github.com/emson/invoicerb

I wrote it because I used an invoice app, that went bust and I needed something really simple to send out invoices and I was in complete control of.

Also I didn't want to use a database so just used Ruby DSL meta files that described my invoice. Admittedly it probably needs a bit more work... but it was fun to do.

Any ruby experts know why the AttributeScope class is required ? Cant the functionality be included in the containing class ?

AttributeScope provides the common language for defining attributes. It's shared functionality, validation rules for the attributes.

If it were included in the containing class it would only be available to that class.

Company or Employee specific attribute-creation methods could be placed on CompanyScope or EmployeeScope.

One of the best articlets I've read about the ruby dsl's was on practicingruby. You guys can check it out here https://practicingruby.com/articles/domain-specific-apis is free now.

Ruby is really great for metaprogramming. To the point where you have to refrain yourself from sprinkling too much of it all over the place at times.

Reply on news.ycombinator.com