Stop Writing JavaScript Compilers, Make Macros Instead
jlongster.com
jlongster.com
You really have no idea what this is doing (the canonical example from sweetjs.org):
class Person {
constructor(name) {
this.name = name;
}
say(msg) {
console.log(this.name + " says: " + msg);
}
}
I do believe with a steady hand this could lead to some great things, but the example on sweetjs.org seems rather heavy handed. Having different `class` macro definitions between codebases that use sweet.js will devolve into madness.
There are a lot of ripe ES6 features to implement as macros. That `class` macro is actually taken from ES6 and has been standardized for JavaScript, so it's not exactly random.
import { foo, bar } from "macros";
The scoping of macros will all stay in tact; any other macros from "macros.js" will be available, but the code that `foo` or `bar` expands to will be able to access them (this is really nice for helper macros and other things).
So yes, modules are going to be a big part of distributing and composing macros.
I'm sure everyone who has ever looked at C macros has thought the same thing. It's a great theory.
In practice, having macros will lead to someone abusing them and eventually, that abuse will be become institutionalized.
Don't say "it could never happen here" because it can and it inevitably will.
I don't have a specific problem with all of this, except I keep thinking a better solution would be to hammer browser manufacturers to keep their shit up to date or explain why they are not doing so.
That's the thing though. When a new proposal comes out, if it's released in the form of a macro it can be tested without vendor buy in. The pressure relief on vendors to keep up to date with the standard is just an unintended (maybe negative * ) side effect.
* Personally I like being able to count on "current" features and I'm not opposed to using shims to patch the environment up to a workable level.
(Forgive the irresistible pun, but the sentiment is sincere: This feature is based on Scheme, so looking at C seems misguided.)
I'm willing to run into the occasional macro abuse to get goodies like core.async and core.typed.
On the other hand, there are merits in their differences. Chrome, although it would prefer Dart, looks upon JS as legacy that they can approach via a good VM/JIT, and a brilliant Developer Tools suite.
FF on the other hand is leading into the JS futures on two fronts: asm.js for both performance and C/C++/.. integration into the browser, and ESnext. Google is very reluctant to invest in ESnext now, just look at the compatibility charts.
This makes sweet.js strategic, it lets us at least experiment with ESnext features as far as macros can take us. I'm hoping sweet.js can also help us make use of asm.js from within the browser buy building macros for things like structs, which halve or more memory footprint of object arrays.
Lets be aware of the war, and do what we can do to resolve it.
It's true that different compilers and interpreters sometimes do slightly different things, but atleast there's a standard which should improve over time. I don't see that happening with macros.
Not sure how modularity is achieved with macros either.
So yeah, "correctness of implementation" is something to worry about, you just haven't had to (yet).
sweet.js is more or less a Lisp macro system for Javascript. Gaining some familiarity with Lisp macros would probably serve you well in terms of being better able to understand what's on offer here and how useful it can potentially be.
Urgh, I just got a sinking feeling in my stomach because I know how right you are.
As long as you don't have different definitions in the same JS file, it's not really a problem because each file can be compiled down to JS independently of the others. I'd imagine you could even add a package.json flag to specify which version of a sweet.js "syntactic sugar" library to use for a given package, and npm could compile the files automatically on install.
The more transparent the sweet.js workflow can be (much like LESS/SASS is for CSS), the more likely it is to be adopted.
But it'd be a complete mess of a code base. Code is read much more often than written, easily understanding a code base is incredibly important, and really hard in larger systems.
If even your syntactical constructs behave subtly different (let alone the code you're writing!), you're setting yourself up for unpleasant surprises.
There's an excellent book on UX, "Don't make me think". The same applies to code: strive to reduce the cognitive load it takes to understand a piece of code as much as possible.
Agreed. And it seems macros can really help with that, by encouraging a DRY codebase.
Syntax to help with common tasks is nice, but if you end up with lots of ad hoc syntax for lots of very divergent, and maybe not so common tasks, you're in trouble. The problem is just the multitude of syntax constructs that typically end up way underspecified, as opposed to the much more formal and strict models of a language that went through a specification process. Imagine learning not three kinds of loops (for, while, do), but 300 subtly different ones in one code base. Ugh.
On the other hand, lots of boilerplate make code unreadable just by the sheer amount of it. This is a very hard balance to strike, I'd rather err on not giving every developer on a code base the power to add new syntactical abstractions.
Scheme and other Lisps macros to build new syntax on top of a small core language.
Are you tired of typing (function() { ... })() and other such forms? If so, you will like macros.
No it really cannot, except if it's exported there.
Here is the full list:
Using DSL to describe your domain (application) is normal. One application = one domain.
Learning in addition 15 different DSLs for libraries and tools, just because it's cute, is not the way to go. Soon you will argue about the exact definition of monad instead of shipping code.
On the building side we have Grunt and the less popular but still useful Jake.
Ember's property system is a DSL.
Angular's DI syntax is a DSL and as much as I love Angular there's really nothing to like about the horrible mix of ([{ that one has to use here.
Actually Ember and Angular and just about every meaningful javascript framework defines several small DSLs.
Pencil Code is an excellent learn-to-program DSL built on Iced Coffeescript.
Coffeescript's crazy syntax makes DSLs slightly more approachable than Javascript but only for a while.
Express and most server frameworks define a DSL.
Anytime you find yourself passing a non trivial config struct you're really playing with a DSL.
This is by no means a complete list.
Macros are used to great effect in Lisp land. They can be used to great effect in JavaScript land, too. With great power comes great responsibility.
Does anyone know Arc well enough to point out some macros used in HN's source?
In most lisp dialects, macros are lexically scoped.
As long as sweetJS follows lexical scoping, a set of macros could be contained within a self-executing lambda like most libraries already are.
I'm not saying you're wrong, exactly, because there are qualitative differences. Manipulating syntax is definitely more powerful, and enables some truly crazy and wonderful constructs. It should be used responsibly.
But when we define an API, we're inventing a new system of primitives for the problem at hand. It's really not as conceptually dissimilar from a language as we'd like to think.
You might argue that the distinction between APIs and language features is arbitrary. But I'd say there is a meaningful difference. An API helps you tackle a particular problem domain, such as making an HTTP request, communicating over a USB port, or writing an image. Whereas a language feature is much more generic. It's not limited to any particular problem domain. Rather, it cuts to core of how we express problems in general.
If we're all ostensibly writing JavaScript, I don't think we should be using multiple, incompatible variants of the same language features.
Why? Because learning languages gets easier every time you do it. And it opens your eyes to languages' internal structure, the way languages are composed, which makes using given language effectively easier.
So now you know 5 or 8 languages and you discovered a few basic principles behind most of them and you understand how semantic constructs compose and you know about various syntactic representation of any given semantic statement (and the other way around, similar syntactic constructs having different meanings). And you come upon a macro, like the one in the example.
You don't know what it's doing - well, you just check the docs and then you know. It's as easy as learning what a function does, really. So what was learning those other languages for? It was just to reduce your fear of introducing new syntax for things. You won't think "oh, how dangerous this could be" but rather you just start using it, because you do it (change syntactical constructs you use) all the time when you switch between languages.
At this stage you can learn some Lisp. And write some real macros. And maybe try Nimrod, which has wonderful macros too. And maybe Rust, which I didn't try but I read good things about. After this you'll understand what macros are, how they are constructed and how they compose - and write many of your own - and then you look at the class example above and you instinctively[2] know that there has to be a syntax transformer which matches class keyword, a keyword and a block, and then there has to be another transformer, perhaps recursive (like in Scheme) or just iterating over the body (like in defmacro) and that all they do is to add some keywords here and there and a statement or two at the top of the block. In short - you know exactly what to look for and even docs are unneeded, you just take a quick look at a macro source and go on using it happily. Alternatively you can look at generated code and work out macro implementation from it easily, too.
I'm a happy user of LiveScript, and CoffeeScript before that. The only reason for not switching to JS with sweet.js is the fact that I would need to write a whole host of macros by myself, while in CS and especially LS they are already written. And of course modifying their grammar is not that hard either. Had I somehow been unable to use LiveScript, I'd use sweet.js for sure. What I'd implement? Probably some kind of "let" statement for sane scope management, a loop macro for iterating over custom collection classes (loop in CL, for/* in Racket, also in others), function currying (partial application) and maybe something like "with" context managers from Python would be among the first. There are many things which are better abstracted with syntax than with (for example) functions.
Would the language be JavaScript? Well, hard question, it would be a superset with underlying semantics intact, but it sure wouldn't "feel" like JS. But would that matter? Not at all - syntactic extensions would save me a ton of time and I'm used to many different syntaxes anyway. And I expect anyone who'd like to work with me to be able to pick up any language in reasonable time - the ability to pick up a few additional syntactic constructs on top of already known language is essentially the same thing, so I think such person would have no problems with it either.
Anyway, macros are good; using them is good; the only dangerous thing is having many people implement essentially the same macro over and over again, but I think this could be solved with a sane module system for macros which I read is already planned. There should be more languages rather than less; more exploration of different syntaxes and semantics for things, not less - and macros are one way of making this happen. And if you have problems with learning what a "class name body" does, then I can only refer you to the first paragraph.
[1] After I wrote this post I realized I could sound a bit rude. It was not my intention and I'm sorry if I offended anyone; while written generally, it's actually only a description of a road I personally traveled, so obviously YMMV.
[2] I know completely nothing about sweet.js in particular, so I'm completely just guessing.
please correct me if i'm misunderstanding.
Not at all - I think it's ok to invent languages, period. Little or huge, DSL or general, using established syntax or creating new ones - making them is good. The results can be bad (and frequently are), but that's to be expected of every process.
Knowing different languages just helps in seeing syntax for what it is: another level of abstraction, not a set of rules cast in stone. You can come at the same conclusion with a proper university class or with any number of other means - I mentioned learning languages just because that's what worked for me.
Unless your problem doesn't absolutely need them. It's considered bad style to use them when not absolutely necessary because they are much harder to reason about.
Macros can be dangerous in the 'give a man a hammer, everything looks like a nail' way. Like when some Python people went mental over meta classes using them everywhere they weren't needed.
I love macros (in lisp) but you shouldn't need to use them that much. They aren't always good.
Isn't this true for almost everything? For example a while back there was this talk "Stop writing classes", where someone argued that you should just use functions for many simple cases (with nice examples etc).
Generally you should abstract on the level most suitable to the problem at hand. Some problems are best (for some value of "best") abstracted with a global var, others with a set of functions, others yet with hierarchy of classes. Syntactic abstraction is just another tool in your toolbox and it is very handy sometimes; of course it's not fit for every problem. But no one is going to do a "Stop writing macros" talk anytime soon, just because no one is writing macros - most languages lack them completely or have only silly string-oriented preprocessors, and even in languages that support them macros are feared and avoided.
I would just like to see macros go "mainstream" - if this means they'd be abused a bit (like classes/OOP now, for example) then I think I can live with it. Probably - it's quite possible I'd come to regret it very quickly :)
The reason is because I've come to understand that social factors trump technical factors nearly every time. The hardest part of writing a software system is almost never solving engineering challenges or writing code, it's communicating the solution to everyone who will have to touch it. Over a software system's lifetime, you will have to read code much more often than you will have to write code. Macros make writing code really easy and communicating everything about what it does, how it does it, and what invariants need to hold for it really hard. That's usually a poor trade-off for anything non-trivial.
This does not refute your point at all -- that macros can be very helpful when implementing DSLs -- just wanted to point out that macros are not a requirement for doing so.
Also, a DSL expressed using first-class entities will fit into and integrate with its host language more nicely.
The reason I think DSLs are good but macros are bad is because DSLs make you think long and hard about the costs of implementing a new language. You have to pay for it up-front by writing a compiler and hooking it into your build system, even if you have help from the host language's libraries, and then you'll end up with something that's clearly a separate language and must be documented and learned separately. That cost makes it clear that creating a DSL is not something to do lightly, and you need a problem domain where there really are much higher-level abstractions that can't be captured by an existing programming language.
The problem with macros is that you can create a new language construct with a half dozen or so lines of code, use it in a few hundred places in the codebase, and now your maintainers have a few hundred problems.
Is it just that, when a maintainer sees a function call it's easier to guess what it does, then when they see the use of a macro?
You know nothing about sync, I presume, yet you assume he/she only knows very few languages because you don't agree. Don't do that.
I don't agree with sync, just pointing out that "go home and learn some programming first" is not a great argument.
Can we all agree that each project needs to use a set of utility functions and move on?
In the example you cited, it seems like a pretty good language versus other examples I have seen for defining Javascript types?
So there will be newbies who make messes with macros. It's happened in the Lisp world for decades -- though not as often, probably, as some people fear -- and it will surely happen here. Some people will overreact to that (as they have to misapplication of operator overloading) and prohibit their teams from using them. Eventually, knowledge of how to use them well will spread wide enough that most people will be able to be trusted using a language that has them, even if that means, in many cases, that they know they don't have the experience to use them properly and so avoid writing their own.
In this vein I'll close with a quote from the great Lisp hacker David Moon: Functions compute; macros translate.
I will not be doing metaprogramming again any time soon.
I'm doing just fine using composition/iteration and creating better, more fluent APIs.
I can see a potential argument that doing type checking via macros costs more than precompilation in efficiency terms, but on the other hand, that's not necessarily so. How does TypeScript do type checking for non-constant values? That is, if I define function foo(bar: string) in a TypeScript library, then compile it, load the library in a Javascript interpreter, and call foo(Integer.parseInt("42")), what happens? I'm guessing that either foo() misbehaves, or that foo() throws an exception.
In the former case, there is no benefit to TypeScript because your type annotations are evanescent and misleading -- you can't rely on them, so you either have to write fences around input values just like you would without them, or risk your code misbehaving because it's written to rely on unreliable type annotations.
In the latter case, TypeScript is adding those fences automatically, so that a function which expects an integer argument can check whether it actually got one and fail if it didn't. That's basically what a macro system implementing type checking would do, too, and it seems to me that in both cases optimization would be merely an implementation detail.
Secondly, it's difficult to get a community around a single type system. Thirdly, it's difficult to get IDEs to support these sweet.js macro type systems.
The fact is, a sweet.js macro type system which is widely in use and supports everything TypeScript has and is supported by many IDEs doesn't support, and is unlikely in my opinion, if not technically impossible. In theory it might work.
If you compile TypeScript and then call it using JavaScript with the wrong type the code will misbehave. It's a downside, true, but a dynamic type checking system written in JavaScript would be too costly. Sweet.js macro type system wouldn't bring any benefit in that regard either.
You can rely on TypeScript type checking to make sure your code is correct. Similarly, if you develop in JavaScript, you can use unit tests instead of type checking to make sure your code is correct. Unit tests nor type checking can't guarantee that other code is correct, even if that other code happens to use your own code.
The benefits of TypeScript's type checking include the elimination of a class of unit tests, automatic refactoring, full intelli-sense support, better readability and automatic documentation (which doesn't remove the need of manual documentation).
[1] http://docs.racket-lang.org/ts-guide/ [2] http://www.ccs.neu.edu/racket/pubs/pldi11-thacff.pdf
No you can't. First of all, you can't implement operator overloading, because Sweet.js is quite limited, as the macros have the form:
<macro-identifier> <expression...>
I can still see a potential method of rewriting functions, whose arguments were decorated with type specifiers, to provide type checking. But that would involve prefixing the function body with type checking code to be evaluated at runtime, which would probably produce a monstrous increase in execution time, and would be ugly either way.
Well. I suppose I shouldn't have expected anything different, knowing that a real macro system requires support at the language level, and that a retrofit like sweet.js is perforce going to be limited in what it can hope to accomplish. But I find myself disappointed nonetheless that sweet.js is what it is, rather than what I wanted it to be -- and, I think, what a lot of people, not least of whom the author of the linked article, imagine it to be.
[0] http://siek.blogspot.com/2012/10/is-typescript-gradually-typ...
For onlookers, here's a lucid explanation of soundness and completeness in type systems: http://eschew.wordpress.com/2009/08/31/sound-and-complete/
The author of that article argues that soundness is more important than completeness, but concedes that there are other ways to view the world too.
The bottom line is that most ad-hoc lint tools are both unsound and incomplete, but they are still useful for finding bugs!
For example, someone wrote a macro system for coffeescript: https://github.com/mrluc/macros.coffee
Related discussions: https://github.com/jashkenas/coffee-script/pull/3171 https://github.com/gkz/LiveScript/issues/328
It's worth noting that sweet.js is a robust macro system grounded in academic research (the core of it from the Honu paper). That CoffeeScript system is a bit of a hack and its flaws would show it the whole community started using it (I don't have time to go into this right now, but things like hygiene, etc are really difficult).
And purely a labor of love! Slides: http://mrluc.github.io/hammy/12-7-rum-macros/deck.html#1 Code: http://mrluc.github.io/macros.coffee/docs/macros.html
I wanted to see how easily I could add (non-hygenic) macros to CoffeeScript, and if I could do it as an npm-installable module instead of a fork of the language.
It was also an experiment in making macros very clear to myself. Non-hygenic (CL-style) macros are very simple conceptually.
But the basic hack that made it possible to get true CL-style macros also makes the whole thing incredibly dependent on the structure of the CoffeeScript AST. Which makes it fragile.
I'm excited by BlackCoffee, actually. This is the first I've heard of it. Thanks a lot for giving me a nod in the github PR, and of course let's mention Oran Looney as well, who wrote the best deepcopy ever for Javascript (I wrapped this up as owl-deepcopy and made an npm module for it).
There's a point where I think this breaks down. It's great for syntactic sugar but I think things like type analysis would become a bit more difficult.
But because the article doesn't really frame it this way, it doesn't really explain to me why I would choose Sweet.js over ClojureScript or Scala.js or Black Coffee or whatever.
Language design is hard. Scheme has gotten it wrong. Over. And over. And over. They didn't get hygiene right for more than a decade. They still don't have it right. But people think they can easily invent new syntax that somehow doesn't do unexpected things. Your clever macro isn't so clever when some poor son of a bitch has been beating his head on a bug all day long just to find out your macro-that-looks-like-a-function isn't evaluating all its arguments because your macro-that-looks-like-a-function is really a function-that-is-a-macro. Don't get me started on missing IF branches and undefined behavior...
Honestly, that's pretty damn exciting. In my book, if JavaScript did that, it would very nearly complete a "worst to almost-first" transformation as a language. Starting with its early versions and all their ugly "bad parts", and advancing all the way to one of the most exciting, expressive languages to work with.
However, it is true it would be better to have one standard-ish compiler that can just execute whichever macros you throw into it.
That being said, you still run into the issue that `sync` brought up.
sweetjs has this hygienic thing that renames identifiers. To prevent it from renaming, there are many hoops that you have to jump through to prevent it.
sweetjs introduces of a lot of new concepts. These concepts are not as easy to understand as the ones in C. The macros are certainly more powerful. It's just that assuming it's C-like macro will give you the wrong kind of expectation with sweetjs.
Granted, I'm not sure how much I like the idea of a macro system which finds macro instances by string-matching the source it's given. But in a non-homoiconic language, I'm not sure anything else is feasible, and I can easily see how something like this could improve Angular.js's dependency injection, for example.
It is the best of both worlds.
Hard to write (way harder than a function), hard to debug, hard to reason about, they're not functions (so fit worse in the rest of the language)... but powerful when you really need them.
"When you have a hammer...", and a desire to stand out from other languages made the lisp-macro myth flourish.
In addition to "deep" things, often you use them as practical, simple alternatives to using stuff like "snippets" or heavy IDEs. You can "DRY" annoying patterns, without resorting to external tooling.
Although macros used badly can be mysterious, so can any excessive pre-processing and build magic.
Macros provide "an API for the compiler", and since the compiler is involved, it can be smarter than external tools.
As I said, when you really need them they're useful (e.g. in typed Clojure). Most (if not all) DRY patterns can be fixed using only functions.
The problem is, the macro mantra has been parroted for so long now it's part of the Lisp culture and its external image ("Lisp is homoiconic! You can modify code! Macros! MACROS!"), when it's actually one of the ugly (but powerful) parts of Lisp you should avoid most of the time. This confuses beginners and people interested in Lisp.
I see macros fitting mainly in DSLs (which is probably a code smell most of the time) and extending the language, as typed Clojure does. What other real use cases do you see for macros?
1. Changing order of evaluation.
2. Creating new binding forms.
3. Implementing a data sub-language a.k.a. DSL
Although arguably #3 doesn't require macros, arguably it requires macros to do elegantly (Rubyists might argue not) and reliably (instead of monkey-patching, using hygiene and a macro-aware module system that can handle towers/layers of such systems).
There are many more real-life Lisp applications which are not written in Clojure.
I'm a long-time user of Scheme and especially Common Lisp. Macros are a consequence of Lisp. Lisp is originally about computation with symbolic expressions (see McCarthy). Application of that to Lisp itself leads to macros and similar. Macros were introduced 1963 to Lisp, extremely early on.
As a Lisp programmer one needs to understand what macros are for and when they are useful. By default I write functions. I also tell people to think in functions first.
But in real life, the more advanced Lisp applications are full of macro usage and macro definitions.
Sure they are harder to debug. But Lisp programmers have developed tools to deal with that. Not everyone wants to learn such a complex language, or is able to learn it or has time to learn it. There are a lot of simpler tools and even simpler Lisps - but for those who want and need this, a language like Common Lisp will provide the necessary flexibility.
ftp://publications.ai.mit.edu/ai-publications/pdf/AIM-057.pdf
You won't be able to directly debug your code.
Also it can be really fun to reason about nested macros.
I think that if it is easy to investiage things like syntactic sugar, and not have it be buried in something like a compiler or lang spec, then DSL/language implementers (and anyone else, if the language permits it) could get away with implementing things that objectively make the language more complex to deal with, because to decipher it is only a query away, anyway.
/head asplodes