Hacker News Reader

Comment by peterhunt | Hacker News Reader

The point here is not that metaprogramming is universally bad, it's that it should be used as a last resort (or second-to-last resort to codegen, depending on your platform/application) and you should feel bad when you need to use it.

Simply using it to save keystrokes is pretty lame since you'll spend much more time maintaining code than typing out the original and explicitness is valuable when returning to a piece of code. Additionally, if you find yourself that you need a lot of metaprogramming for a lot of things it's often an indication that you could just refactor your code using static idioms and be better off.

Not to mention that the more you use metaprogramming, the more likely it is that you or someone else will kill some runtime optimizations of the JIT.

IMO, syntax matters way less than people tend to think it does, and the additional implementation complexity and astonishment cute syntax introduces tends to make the trade-off not worth it.

View on HN

In my experience overuse of meta programming is usually a sign that the architecture of the system is bad and the software should be split up into separate libraries or web services.

That all depends on the language. Lisp (pretty much all of its dialects) heavily emphasizes the use of macros to extend the language and build all kinds of control structures and embedded domain-specific languages (EDSLs).

Absolutely. One could make the argument, though, that many (including me) believe that expressive power and conceptual beauty is Lisp's core strength, not code maintainability or runtime performance.

expressive power and conceptual beauty is Lisp's core strength, not code maintainability or runtime performance.

Well, Arc is actually very maintainable, and quite fast. After all, HN is powered by Arc, and it serves >100k daily uniques on a single core.

Yep! Not saying that it's impossible, just that some of the concepts that Lisp is based on make these things harder to pull off. The reason for this is that Lisp emphasizes flexibility, and the more flexible your runtime environment is, the more ways to do something, which makes it easier to have hard-to-maintain code (and harder to reason about performance, too). But it's not impossible to do with Lisp for sure.

Lisp macros also provide ways to take ugly, hand-optimized code and clean it up to provide a cleaner, friendlier interface. Like all macros, this comes at no runtime cost (since all macros have been fully expanded by this point).

Actually, it's not any harder to write maintainable code in Arc. In many ways, it's much easier, because you write your program as a set of parts which fit together like an arch.

HN is fast because it is feature-barren. That's a design choice pg is free to make (and cutting features is a great way to improve performance), but not a proof of Arc/Lisp's ability to support complex applications with high-performance.

Which features would convince you that Arc is fast?

Just for the record, let's count to 100 million. And add the numbers up while we're at it.

  ~ $ time (echo '(do (= n 0) (for i 1 100000000 (++ n i)) prn.n (quit))' | arc)
  Welcome to Racket v5.3.5.1.
  Use (quit) to quit, (tl) to return here after an interrupt.
  arc> 5000000050000000
  
  real	0m28.561s
  user	0m28.308s
  sys	0m0.249s
  ----
  ~ $ time (echo -e 'n=0 \ni=0 \nwhile (i <= 100000000): \n  n += i \n  i += 1 \n\nprint(n)\n' | python3.3)
  5000000050000000

  real	0m30.244s
  user	0m30.230s
  sys	0m0.013s

It would appear to be competitive with Python on my machine on this particular task. (Also you can make it faster by dropping into Racket.)

A while loop and temporary mutable variables are definitely not the Pythonic way of doing this. More idiomatic:

    $ time python -c 'print sum(xrange(100000000 + 1))'
    5000000050000000
    
    real    0m1.398s
    user    0m1.383s
    sys     0m0.012s

Comparison to baseline:

    $ time (echo -e 'n=0 \ni=0 \nwhile (i <= 100000000): \n  n += i \n  i += 1 \n\nprint(n)\n' | python)
    5000000050000000
    
    real    0m33.140s
    user    0m32.939s
    sys     0m0.023s

I see. That is good to know; I merely chose mutating global variables in a loop because I knew how to do that in both languages. (I am not very familiar with Python.) That is not the idiomatic way to do it in Arc, either. I would normally use a recursive function, like this:

  arc> (time:xloop (i 0 n 0) (if (> i 100000000) n (next (+ i 1) (+ n i))))
  time: 9121 cpu: 9130 gc: 0 mem: 480  ; the times are in msec
  5000000050000000

Or perhaps a "higher-order function":

  arc> (time:sum idfn 1 100000000)
  time: 19889 cpu: 19908 gc: 0 mem: 1224
  5000000050000000

Or use a deforestation macro that I wrote, which is closest to your Python example:

  arc> (time:thunkify:reduce + (range 1 100000000))
  time: 17971 cpu: 17985 gc: 0 mem: 3592
  5000000050000000

Also, here's what you can get by dropping into Racket:

  arc> (time:$:let loop ((i 0) (n 0)) (if (> i 100000000) n (loop (+ i 1) (+ n i))))
  time: 402 cpu: 403 gc: 0 mem: 920
  5000000050000000

I suppose Python has an analogue of that--dropping into C, or at least loading C libraries. Which Racket can do too. Mmm.

Common Lisp was designed for development of large/complex applications ('applications' really means here software that is used in production) with good performance.

For example this CAD system from PTC is based on 6+ million lines of Common Lisp code and under development for two decades:

http://www.ptc.com/product/creo/direct

Lisp macros are a lot simpler and easier to understand than what goes on with the types of metaprogramming done in ruby/rails. Expanding a macro is a lot easier for me to wrap my head around than stuff like dynamically defining methods when something is subclassed or when magic happens in method_missing.

At the same time the Lisp community is one of the fiercest in supporting the idea that nothing that can be a function should be a macro. Metaprogramming just because it is clever is more abhorred in the Lisp community than any other I've seen.

Of course! Macros are distinctly less powerful than functions for one reason: they are not first class values at runtime and thus cannot be passed to higher-order functions.

This is solved by using an operative lisp - fexprs (or 'vau'/'$vau' in kernel and wat) are non-applicative combiners that execute at runtime, so you can have a first class combiner that chooses when/how/how many times to evaluate its arguments.

I've been having a lot of fun recently writing code in wat-pl (my perl port of Manuel Simonyi's wat-js interpreter)

Yep, and it surely depends upon the problem being solved too.

The heavy emphasis of macros in Lisp may also be its greatest weakness. I see all sorts of Clojure code that uses macros where they needn't have, and it tremendously impedes my ability to use the code in ways the library author may not have expected. I love the way petehunt put it - metaprogramming is a last resort, you should feel bad when you use it, and language designers should include it but deemphasize it. (Scala and Haskell nail it I think; Clojure perhaps emphasizes macros a bit too much which harms the output of intermediate programmers. But, Clojure is not designed for intermediate programmers.)

While I mostly agree with what you are saying I think that there is a place for metaprogramming and some programming languages implement metaprogramming very well.

Using metaprogramming should not make you "feel bad", however it should not be used excessively. Metaprogramming gives you a lot of power, it allows you to extend the language in any way you please. With this great power, of course, comes great responsibility but I would rather have the option to use this power than to not have it at all.

I would also argue that metaprogramming can enable your code to be more efficient, if the macros/templates you write are evaluated at compile-time. This is the approach that the Nimrod programming language (http://nimrod-code.org) takes and it works well. This also means that in most cases compilation will fail if the macro is incorrect instead of causing some silent errors at runtime.

There's some strong (i hope a decent number of people here would say "winning") counterarguments, scala macros and implicits are incredibly useful. (i see lots of complaints about C++ template error messages, but i haven't been doing C++ so can't comment informedly)

I think a better rule is if the tooling can't look at the source, intermediate compiler stages/ AST and bytecode/binary and tell you unambiguously where something came from, then you can characterize it as a "last resort"

And there's the counterargument that for statically typed languages with good type systems (scala, haskell, ocaml, F#), there's REPL prototyping/testing and code generated from template haskell or camlp4 has to type check, so you have multiple safety mechanisms

The point here is not that metaprogramming is universally bad, it's that it should be used as a last resort ... and you should feel bad

you could just refactor your code using static idioms

you or someone else will kill some runtime optimizations of the JIT.

syntax matters way less than people tend to think ... astonishment [sic] cute syntax

Many intelligent, design-savvy people would strongly disagree with you. Besides the obvious gains in the ability to create DSLs, there is a power and fluidity in metaprogramming that is not possible any other way.

Speed (as, say, provided by runtime optimizations from a JIT compiler) is not always critical. If it was, one probably should be using a language other than Ruby or Javascript; they're meant to be flexible, powerful languages, and their emphasis is not on pure speed.

Syntax is not "cute." Humans are going to read and use this code, and the more it reads like English, the more likely it is to be understood.

You seem to be suggesting that DSLs are universally accepted as a pinnacle of good design.

Indeed. I, for one, think that over-proliferation of DSL's can be a Bad Thing. To use one example from my own experience: I spend a lot of time coding in a Groovy / Grails environment. Now, I like Grails and Groovy, but Grails introduces it's own weird DSLs for things like:

configuring Spring Beans configuring Log4J defining URL mappings configuring Hibernate mappings & queries etc.

Now some of those are cool, but at least some of them replace other techniques that I already know, and find much more readable and understandable. Configuring Log4J, for example. I'd much rather simply jam a log4j.xml in there and forget about the Grails DSL.

The thing is, none of these DSL's is, in and of itself, necessarily bad in any way... but there's a bit of "cognitive overload" in having to deal with 3 or 4 or 5 new DSLs, on top of the base Groovy stuff. The flip side is, these DSLs mean that your configuration is done mostly in .groovy files and you can use normal groovy syntax for looping and accessing variables, etc. So it's easier to code up more dynamic configurations than if you were using XML files or .ini files.

Anyway, the point of this rant is not to say "DSLs are bad" but just to lend weight to the suggestion that they aren't universally Good either.

Not knowing anything about Grails, does the Log4j DSL not do the equivalent of generating the XML config, and if not, would you have preferred a system where you could choose to write your own config or let the DSL do it for you?

Unsure why you [sic]'d astonishment; I was referring to https://en.wikipedia.org/wiki/Principle_of_least_astonishmen..., a well-accepted principle of UX which certainly applies to the UX of languages and APIs.

I am unsure what you mean by the "obvious gains" you get from DSLs. I see many DSLs as a code smell -- that the runtime environment is almost expressive enough to express the syntax construct the DSL creator wants, but not quite.

The implementation of DSLs tends to judiciously use closures, operator overloading, dynamic getters and proxies such that it's non-obvious what's going on under the hood. Sometimes that trade-off is worth it. Usually it isn't, in my experience.

I believe (I could be wrong) that DSLs were coined around 2004. I don't think that is a long enough history for us to even think about simply accepting that they are a good idea in and of themselves -- garbage collection was invented in 1959 and is still being debated!

I see the speed argument all over the internet being presented as a binary argument: either you care about speed or you don't. That's simply false -- I care about order of magnitudes for speed. I personally don't care if I can encode h264 faster in JS than I can another way, but I do care that my event handlers execute within 16ms so I don't drop frames in the browser.

Some syntax is certainly cute. Additionally, making syntax readable like English should be a non-goal IMO; for many people, Lisp is far more readable than SQL, which is much closer to English.

2004 seems really late for the inception (or possibly formalisation) of Domain Specific Language, I assume that you're talking specifically about embedded (also called internal) DSLs, otherwise SQL at least is from 1986.

As an earlier example of an Embedded DSL, the book PAIP[1] included prolog embedded in common lisp in 1992.

It could be argued that the loop macro in Common Lisp is a DSL for describing iteration. If not loop, then certainly regular expressions are a common EDSL for describing a regular language, and performing operations with those languages against strings?

I found this snippet from Computers in Crisis [2] which seems to describe domain-specific languages in familiar terms from 1975

    Most domain-specific programming languages can be categorized in one
    of two ways; either as a "sugared" general ... of programming, in
    fact the style of problem solving, embedded in and supported by that
    language remains unchanged.

[1] http://www.norvig.com/paip/README.html [2] http://books.google.co.nz/books?id=QndQAAAAMAAJ&q=%22Embedde...

Yes, you're right, I did mean embedded DSL.

Smalltalk has been doing embedded DSL's since its creation in the 70's; they are not a new phenomenon.

Reply on news.ycombinator.com