Macro-ts: TypeScript compiler with typesafe syntactic macros (2022) | Hacker News Reader

Macro-ts: TypeScript compiler with typesafe syntactic macros (2022)

blainehansen.me

96 pointsaabbcc124143 comments

I wouldn't call this a macro system. It's a preprocessor for another language that is transpiled to typescript.

A ts macro system would be integrated with ts language service and tsc. As a compile time language, the main advantage of tsc is, well, developer experience, which, unless I'm missing something, gets thrown out the door here.

Also, the demo includes very weak use cases, much of it could be done directly with js functionality without the additional macros. It's not clear to me how much effort and computation time does this saves.

Agree. It’s template TS, which is why I shied away from it when I first encountered it. I do think it has some language service support FWIW, but I don’t want even more new arbitrary syntax in a constantly churning language. I do want TS macros, I just want them to be compile time functions over existing syntax with, well… lisp but I can’t have that.

TypeScript has plugins now, so that would be possible?

> We can't change the parser

Aw :-(

That's the bit that I'd really like to have TS macros for!

My own approach has been to loosely parse input as something like D-expressions, transform that, and pipe the result into the TS compiler. Crude but effective. The trickiest part is extending the tsserver support, but even this is doable.

Lisp user here. How come they can’t change their parser?

The describe it as "an unsanctioned hack on top of typescript", so I think the point is they don't want to change the typescript parser itself, but provide this as an add-on.

The implementation of the parser is deeply, deeply baked in to the whole of the MS TypeScript compiler and tooling libraries. It's not feasible to, e.g., add new AST nodes, or change or augment the grammar producing them, and there's nothing like S- or D-expressions to allow for read/parse separation like in lisps. (Which is why I added something D-expression like.)

Practically speaking, the TS team wants the language to resemble JS as much as possible and usually avoids completely new syntax, e.g. new expressions. For this reason they haven't added switch expressions even though technically it's probably not hard to do.

yeah, at this point, other than type annotations and other type related syntax (like type imports and type declarations), the team mostly view the other differences from Javascript as a historical mistake.

Ideally, a transpiler that does not want to type check the code, should merely need to know the syntax for type related code and strip them out. This is especially true if targeting a sufficiently recent version of ECMAScript, or if the transpiler will be doing its own transformation of newer features to older ones for backward compatibility.

In practice while many typescript features can be handled like that, even some that are not strictly type features (like method accessibility modifiers), some existing features like enums needs to generate new code, although at least for that one it is an easy pattern to generate.

This is especially true of any feature that cannot be made into a trivial re-write or strip rule. Namespaces and const enums are an example of features now discouraged, as simple rewriting engines cannot handle them. Indeed the whole "isolatedModules" compiler flag is specifically about making such separate simple transpilation possible, and its documentation details other scenarios that can be a problem.

That sounds great - is your code available anywhere so we can build on it?

Yes: https://git.syndicate-lang.org/syndicate-lang/syndicate-js/s...

The `syntax` subdirectory is (or should be!) pretty clean. The `compiler` subdirectory is, uh, functional. Please do get in touch if you have any questions.

Lean4 manages to pull off changing the parser on the fly at compile time. You can add new productions, add new syntax node types, and add new tokens. Then define macros or code to process the additional syntax. Here is a sample I found that adds a simple JSX-like syntax starting around line 93 and then uses it at line 169:

https://github.com/leanprover/lean4/blob/master/tests/playgr...

I believe most of the language is defined this way, although it is pre-compiled.

For more details see the lean4 metaprogramming book: https://github.com/arthurpaulino/lean4-metaprogramming-book

Yes, I've done this too, but I don't think it's feasible for the MS TypeScript implementation, which is implemented quite differently.

> Since WebAssembly can't directly interact with the dom (yet)

Going off at a tangent, is there a reason WebAssembly can't interact with the DOM? I would have thought that being able to do so would be kinda useful, particularly in a web page.

WebAssembly can interact with JavaScript, and JavaScript can access the DOM. This was a compromise to reduce attack surface and implementation effort, to get a version 1 out the door. Allowing direct DOM access would have required implementors to harden every individual DOM API to make sure the new call path doesn’t open any security holes. The WASM-to-JS layer is comparably tiny, and the JS-to-DOM layer is already there and (hopefully) secure.

But doesn't it defeat the point of using WebAssembly, since you have to pass everything through JavaScript anyway (and therefore it's at the speed of JavaScript)?

In cases where you care about performance, yes. Newer webassembly frameworks (rust from what i remember) are batching operations to the DOM for that exact reason.

But if WASM can call JS, and JS can then interact with the DOM, then doesn't tat allow WASM to exploit security holes in the DOM?

Codeflo answers it from the security perspective, but I’ll add a little to it in that WebAssembly doesn’t want to replace JavaScript. At least not currently, but it wasn’t always like that and there has been a long road of getting to this point with a lot of discussions on the topic. This may be dumbing it down a little, but basically it was decided that replacing JavaScript was unrealistic.

So for now you can think of it as a range of different things in the form of web development. You can view it as a sort of non-proprietary flash, where you can technically distribute your flash-like application to any browser. You can also view it like another form of virtual DOM (of sorts) the way Microsoft tries to do with Blazor, or you can view it as JavaScript “modules” where you do compute heavy stuff with C++ and implement it in your otherwise JavaScript frontend. But that’s just the web part, WebAssembly wants to be more than that, but you should go to their documentation for a better understanding of that.

It would be kind of useful to have WebAssembly interact with the DOM and I’m not completely convinced that the security would be a major issue, but it’s a massive undertaking, and it’s likely going to be sort of in vain. Because JavaScript is moving forward at a much higher pace than WebAssembly. One of the primary reasons you often see listed is that the DOM is slow, but that’s not really true. Because the adding or removing a DOM node is literally a few pointer swaps, which isn't much more than setting a property on an OOP language object. What is slow is "layout". When Javascript changes something and hands control back to the browser it invokes its CSS recalc, layout, repaint and finally recomposition algorithms to redraw the screen. The layout algorithm is quite complex and it's synchronous, which makes it stupidly easy to make thing slow. Imagine handling that, again, when we’re not even really intend on doing so in most frontend JavaScript. What I mean by this is that one or the reasons we use React (and by we I mean the global we) is that it’s much easier to not fuck-up with the virtual DOM, and while things like Blazor is valid alternatives, they aren’t remotely comparable even when backed by Microsoft.

So it’s really a question of “do we want to do all that work, when all that work is already being done better by someone else?”, and for now the answer is “no”. Which is disappointing to many, but probably not the many who were going to contribute anyway.

Excellent overview, thank you. WebAssembly is something I have seen mentioned for maybe 10 years but I never really took time to actually figure out what it is. I suppose I still don't truly know but for a 2 minute read, it cleared up a lot of my misunderstanding.

I don't think that's the consensus opinion, and suggesting that it is might be misleading to people who are less informed than you are.

You already mention Blazor, which would hugely profit from direct DOM access. Various React-like frameworks now exist for various languages that target WebAssembly, all of them would receive performance benefits from not having to go through a JavaScript layer all the time.

That's the community side. On the standards side, efforts to enable direct Web IDL bindings are still underway, they just take time. Some things have been postponed because the issues are more complex than originally anticipated. But this will eventually happen.

> I don't think that's the consensus opinion

Maybe, but wasn’t this the reason DOM replacement was removed from the roadmap?

The case he describes seems like a really bad example. All of that code could be rewritten to use properly composed functions - a library, for sure, but the complaint isn’t valid. That’s like writing procedural PHP with the database connection defined inline everywhere you use it. There is no need to do so if you can import modules!

Are there any actual cases where a macro system in TypeScript would be useful?

I think it could be useful for cases where codegen is used - creating ts file that operates on a list of files in directory, autogen for client/protocol based on some schema/declaration.

TS allows terms to depend on terms (normal JS functions), types to depend on terms (typeof operator, polymorphism, etc.), and types to depend on types (mapped types, inferred types, etc.), but not terms to depend on types. Macros could help there, the real world use case would be things like the classic: JSON.parse<{someField: string}>(...), in which a macro might allow the term to depend on the type (i.e. throw if the line data doesn't deserialize into the type).

This of course, goes against the "prime directive" of typescript and would never see the light of day in the core compiler, which is probably a good thing. But for a hobby project it'd be interesting.

In large scale TS applications of my experience, there's always several places where you're writing out long terms exactly corresponding to types in some mechanical way, but there's no real infrastructure to help you there so it's just a manual process.

> In large scale TS applications of my experience, there's always several places where you're writing out long terms exactly corresponding to types in some mechanical way, but there's no real infrastructure to help you there so it's just a manual process.

I’ve been at this point several times, and for most of the time, the solution was to either use the satisfies operator or remove explicit type declarations in favour of what TS infers on its own, then refactoring the code. This resulted in a leaner, more elegant, and most importantly correctly typed code base. YMMV of course, and sometimes this just isn’t possible, but I believe macros to be way too dangerous for that fringe use case or hobby project - inexperienced developers would jump to introduce them everywhere immediately, guaranteed.

Although I have no idea how they managed to do so, someone actually made the parse<SomeType>(someVariable), possible in native typescript: https://deepkit.io/library/type

Not native typescript, they hack into the compiler. Their minimal `deserialize` example is also 250kB (!!!!!!) minified. Not really an option. https://docs.deepkit.io/english/runtime-types.html#runtime-t...

> Are there any actual cases where a macro system in TypeScript would be useful?

A lot of use cases involving decorators today (which are really rough for tree shaking).

Writing type-safe SQL queries.

Using JSONSchema.

Oh dude! is not the full crab but at least a big claw of it lol, congrats on the work is awesome, hope it gets through some more stable versions so we can build more on top of this

This may be a useful tool, but personally every one of the examples seems to illustrate what I would consider to be an anti-pattern in how the code is written.

As an aside I think the non-null assertion operator should be transpiled via Babel from

`userId!`

`if (!userId) throw new Error('Non-Null Assertion Failed "userId!"')`

I like it, but I think it would violate the rule that TS code is simply erased. Though there are already a few exceptions like enums, but those are out of favor.

Just use zod?

> knows that a huge chunk of why it's so excellent is the macro system

Macros are bad and you're bad if you use them. This has been essentially a de facto mantra since the early 90s (when I was a wee lad) and it's quite common to see people writing very long articles on why you shouldn't use them[1]. They are magic, so should be avoided.

There's a reason languages like C# or Go don't have them, and you should take heed of the very smart people that designed these languages. Even in Rust, they are controversial[2] (the most obvious reason is that your IDE is worthless when macros are involved). Metaprogramming (i.e. macros) adds extreme complexity (in some cases, pre-runtime-Turing-completeness) with many footguns, including the classic, and very "fun," double evaluation.

Here be dragons.

[1] https://www.linkedin.com/pulse/why-considered-bad-practice-u...

[2] https://www.reddit.com/r/rust/comments/taxfe3/what_are_the_p...

As is usually the case, it depends. Macros for “reducing boilerplate” or other syntactic reasons are poorly justified. While the resulting code might look cleaner, it’s often much harder to decipher (JS “decorators” suffer from the same issue). Macros can be invaluable for trading compile time for runtime performance, compensating for missing language features, and general metaprogramming… in a compiled language. It’s beyond me why would anybody think that macros are a good fit for interpreted JS.

> There's a reason languages like C# or Go don't have them,

Go has a quasi macro system though (go generate)

https://gradha.github.io/articles/2015/01/the-day-go-reinven...

I like the macro systems in Crystal and Haxe (reifed macros)

https://haxe.org/manual/macro-reification-expression.html

https://crystal-lang.org/reference/1.8/syntax_and_semantics/...

I think C would have benefited from such system at the AST level instead of text substitution.

Javascript is dynamic enough there is absolutely no need for Macros.

> Javascript is dynamic enough there is absolutely no need for Macros.

TypeScript macroses would be very useful for the same things codegenerators are useful: for example generating api client from swagger schema. Ofc you can do some trickery with metaprogramming, for example build and eval strings on fly and the result could be a dynamically generated function, but the point is: we want the process and the result to be typesafe and efficient, which is what is missing with hacky dynamic metaprogramming approaches.

A swagger schema looks like YAML with a finite set of types. I don't see why you'd need to use eval in that case when from one hand you already have the schema in a javascript object and the other hand a simple builder pattern would build your API directly from that schema. Absolutely no need for macros.

Generally speaking, because it would be very nice to have types without having to do offline code generation.

C# _does_ have a macro system. See https://dotnetfiddle.net/y3zFQ4 for example. This has a macro called Foo which, when calls, prints out the AST of the given expression. It could also manipulate part of the expression and evaluate it, if desired.

I'd argue Turing completeness is bad also, Rices theorem and all. But like with macros, sometimes you just need to accept and deal with the badness because the alternatives are worse (or undiscovered).

If magic was unnuancedly bad we'd all be using assembly. Rust-style macros are a tradeoff; you get constrained magic, localized to the spot where you said `!`. Presenting using them at all in Rust as 'controversial', with a Reddit thread that says nothing of the sort, is disingenuous; proper Rust usage is littered with macros (and the presentation of said 'mantra' would be news to any Lisp user).

Reply on news.ycombinator.com