Scala Macros: “Oh God Why?”
blog.empathybox.com
blog.empathybox.com
Why macros? I was a long-time sceptic. I now tend to think about them differently because I believe we hit on a brilliantly simple scheme that can express a lot of different use-cases. In particular we'll be able to do the analogue of Microsoft's LINQ with macros. And we can express optimizations such as turning foreach applications into while loops. And we can remove a lot of compiler plugins. And finally it looks like we can remove down the road several special cases in the language and the compiler. So macros might pay for their own complexity handily. But I should also say that at present this is a SIP, a Scala Improvement _Proposal_. It's not yet accepted. And part of the current proposal is also that macros will be enabled only under a special compiler flag.
Regarding macros: it's quite clear that they simplify the job of the compiler writer: you can provide more features, more efficiently to the end user. But every feature has a cost, and increasing the end user's surface area onto the language by providing 1) macros, 2) new features enabled by macros, is very much a double edged sword. Being enabled by a compiler flag definitely helps to minimize end-user exposure to the former.
Macros in lisps can be used as a replacement for many of the features that Scala provides, including call-by-name parameters, (certain types of) implicits, and many uses of generics. If we could deprecate some of those features in Scala by replacing them with macros, then this would seem like more of a net win to me. But that doesn't seem likely: instead we will probably have all N, and Scala programmers will need to decide which of the growing list of ways to accomplish cat-skinning is best.
import language.macros
Of course that will not prevent misuse, but it will make it much easier to enforce policies, if someone desires that.
Also, macros cannot be used in Scheme to implement implicits because implicits, like type classes, rely on static type information. I have actually heard of an attempt to implement type classes in Scheme and it required manual type annotations in all use cases. And, while it hasn't been formally proven, intuitively implicits and typeclasses feel like the kind of languages features that would be difficult, if not impossible, to implement with classic macros.
I feel like you have some pre-conceived notions, both about what macros can and can't do, and about what affect they will have on the Scala language when added. However, as others more knowledgeable than me have already pointed out, macros have already managed to simplify the language as is, and their full potential has yet to be untapped. I would suggest giving them a chance.
I'm certainly not an expert in this area and have only toyed around with the SIP and Retronym's examples so take everything I say with a huge grain of salt. But I do hope everyone critical of the SIP and the concept has at least done the same.
Open source patches are all itch-scratchings. For Scala macros, it's a researcher at EPFL, I believe. I don't think you could write a thesis about fixing bugs in the IDE. So this contributor was not available to submit IDE bugfixes; they _were_ available to submit macros.
It's the same for other contributors. Someone building an app on Scala may be available to submit a fix for some scalability issue they are hitting in production; they will not be available to work on macros, or on IDE bugfixes.
Open source projects get the fixes they get. You can't act like the priorities are set by some central product manager.
So I think it's just not correct to argue that "the priorities are wrong, why are they doing this instead of that, etc." - everyone is doing what matters to them. Some people _are_ working on IDE bugfixes, other people are working on macros. People can do what they want.
Also, as for why macros are being put in the language: this is what Martin Odersky had to say on the subject (see http://www.scala-lang.org/node/12494):
"The intention of macros, and of Scala's language design in general, is to simplify things. We have already managed to replace code lifting by macros, and hopefully other features will follow.... Optimizations such as on Range.foreach are another use case. I believe that in the long run macros can be a simplifying factor."
Macros therefore can actually satisfy at least one of the complaints the Yammer CEO made about the language: slow foreach loops, along with simplifying the language in general.
Yes, whoever contributes can choose whatever they wanted to contribute. But most successful project, companies or startup or anything else for that matter, involves doing things that most people would rather do, but necessary.
When you choose a programming language for your business, your business/life or anything related to it will depends on the language. I think its fair for anyone to choose a language to make sure those necessary boring stuff are robust in the language.
Those who contribute can choose what to contribute. Those who consume, can choose what they want to consume. Probably all of us have been on both side of the fence, so I would say its a fair assessment.
InfoQ: There has been some debate in the Scala community about the need to add features versus keeping it lean. Can you give us a sense of where Scala is headed? Does Scala need to become a bloated multiparadigm and multipurpose language to be considered mainstream?
Martin: I think the consensus is very much that we keep it lean. Scala is a compact language despite its enormous spread of use cases because it has always tried hard to unify things. The same concepts describe objects and functions and components. Instead of adding features, Scala has very powerful abstraction capabilities that let users define similar features themselves in libraries. In the future my ambition is to make Scala an smaller language, not a larger one. I know it is hard, but we will try.
In contrast, I think Python and Ruby both owe a lot of their success to the people willing to put a lot of energy and polish into this essential but less glamorous work.
The other thing with Scala is that it compiles to bytecode; once compiled, it's just Java. And you can always use any Java library. So there's that base of a very solid JVM runtime and very solid ecosystem of libraries that's always there.
If this was a more esoteric language feature, I'd be inclined to agree, but macros (if done well), I think will be a huge boon to the language.
I do all sorts of hackery in Java now (external code generation, heavily using annotation processing tools (APTs) which are a very, very limited form for language-based code generation) to do semi-metaprogramming stuff and can't wait to do the real thing.
(And do it well, e.g. not to make things "even more cryptic" just for the fun of it, but to solve real problems.)
There are good reasons to doubt the overall value of macros in production code - I won't argue this either way - but OCaml has had success by separating the macro expansion mechanism (camlp4) from the main language. People who want macros can have them, people who want a production language without macros can have that too.
False. There isn't much else to say, but if this is all he thinks macros are about, he doesn't know anything about them, their uses, or their potentials.
Macros would basically take a large amount of nasty extra-lingual stuff (code-generation, byte-code generation, compiler plugins) and codify it into a systematic, easier to use and easier to understand framework.
F#-style Type Providers? Compile-time typechecked regex literals? Compile-time typechecked html? Compile-time typechecked CSS? This sort of thing is already being done (see Play!) and is pure awesome as a user (i <3 the type-checked HTML templates) but the implementation involves a huge amount of mysterious extra compilation steps and magic that macros could help simplify and codify.
The goal of users of those projects is immediate usability which clashes with goals of developers, research.
I have some doubts where one can get PHD (publish an article with high citation index) by improving a compiler error messages. As opposed to adding a new cool feature to the language/compiler, cryptic error messages be damned.
I still think it is a shame that neither Red Hat nor Jet Brains has joined forces with Scala/Typesafe. Both of these companies know how to ship products and make developers happy. This is something that Typesafe has to learn now. On the other hand, Red Hat and Jet Brains need to learn how to do language research... (which may actually be harder, once they go beyond fixing the obvious Java flaws).
There are also other incentives for researchers to create usable tools and implementations (assuming that the tools are actually meant to be more than experimental prototypes).
To begin with, compiler developers traditionally "dogfood" their compilers. They may still be willing to live with things that you wouldn't have in a commercial product, but they can't and don't completely neglect usability, either.
Second, usable tools are more likely to be adopted by other researchers, creating a positive feedback loop with respect to future publishing opportunities.
As a former professional Common Lisp dev, those two features strike me as the real difference between a macro and function call.
Although, Scala specifically, I ran away from. The language itself isn't terrible, it just forces one to confront the endgame of the bankrupt Java philosophy. The stdlib is basically the cross product of pre-optimized category theory with noun-hell, and idiomatic scala is far too overabstracted through abuse of first-class modules (which, admittedly, was a design goal).
I tried to use the JVM for a long time, I really did. I thought the advantages of multiple implementations, platform flexibility, and a common type system were fantastic. The fundamental problem is that no matter the skin, when you want to peek through the abstraction you're still fundamentally writing Java. I absolutely detest the Java language itself, as it's basically what should come out of a compiler, not be fed into one (I realize this criticism is about 15% unfair given that Java did actually introduce some new concepts to mainstream programming but implemented them in the VM rather than the compiler, but it has really failed to keep up with the times).
At this point if I were using the JVM, it would most likely be to make use of some hypothetical damn good libraries. In that case, as I'd be gluing together things that are already typed, I'd use whatever dynamic language seemed prudent. The Java type system is an everpresent fact of life on the JVM, and I'm now of the opinion that trying to use a second type abstraction concurrently is foolish (especially if one wants to run their code on anything other than Hotspot). I think Clojure is an amazing piece of work, and would still be using it, but its lack of importance on typed data structures really kills my ability to reason while developing new abstractions (the lack of well-supported algebraic data types and pattern matching especially). Taken together, those last two points imply that a statically typed JVM language has to utterly commit itself to the Java type system. If I absolutely needed to write a library on the JVM, I would look into and most likely go with Ceylon/Kotlin (or Gosu if they ever actually did a source release), but failing those probably write java-in-kawa for the straightforward class definitions with macros to ease the pain.
In any case, I'd be damn certain of exactly what I was writing beforehand. The Java ecosystem is absolute garbage for prototyping.
Macros in Scala are primarily of interest for metaprogramming; it's a fairly natural direction for Scala to take, given that Scala has always positioned itself as a language that supports internal DSLs.
Similarly, in syntactically richer languages, macros could add syntactic constructs that are not akin to function application. With sufficiently powerful macros, you can add, for example, new infix operators, a new form of switch/case or a new way to declare variables.
myObj explode 10 times
which translates into myObk.explode.10.times()
Through the magic of implicits, you can make any builtin type auto-cast to your wrapper type, so there really is no need for syntactic macros - scala is already DSL-enabled.
(That the compiler is slow has other reasons, but is also something that they've been working on simultaneously for 2.10, so it's not as though that has been pushed aside by macros.)
"Unlike Java, type signatures in Scala don’t really explain idiomatic usage to mortals very well."
This is a great diagnosis of one of the big issues with Scala's style of static typing. Everyone knows that the type descriptions take a bit to grok compared to a C-style languages. Why? Well, in C-style languages, a function definition is itself a representation of idiomatic usage. In Scala and its brethren, you have to translate from one to the other.
I'm sure that's a commonplace observation, but it had not jumped out at me like that before.
The only exception where the signatures are really long and complex are parts of the collection infrastructure, because Scala developers tried to make it work as intuitively for users as possible: E.g. making non-collection classes support collection operations, always returning the most "precise" type, etc. etc, which is very hard and painful work not done in any other language to this day.
In the end while I hope signatures get shorter and more like the use case signatures already existing in documentation, I don't think it matters much.
Untyped languages have more or less no useful signature at all and they are still pretty much alive.
In Play, we have generated code for forms validation and json serialization. This enables the compiler to see code that can't work, while a more dynamic approach would lead to an exception at runtime.
ref: http://strangelyconsistent.org/blog/macros-progress-report-d...
Macros are supposed to be a replacement for writing compiler plugins in many cases (compare with Java annotation processing vs. Java compiler plugins). The API is easier and more straightforward, semantics are easier to understand and don't require separate setup step.
In the meantime work has of course not stopped:
- Syntax has been simplified and several confusing bits have been removed/deprecated (Octal literals, FP literals without digit after dot, ...).
- The jar size of the library has been reduced, while adding fixes and functionality.
- Value classes have been integrated, which provide the performance of primitives with user-defined classes.
- Inlining has been improved a lot.
- Compilation speed has been improved a lot.
- Some collection classes have been performance optimized and are now a lot faster (and sometimes faster than the Java implementation).
- Play 2.0 and Akka 2.0 have been released, both with huge amounts of documentation.
- Documentation has been improved.
- ...
My personal impression is that while Scala has roots in academia, solving real-world issues is the focus of the team.