I’m porting the TypeScript type checker tsc to Go
kdy1.dev
kdy1.dev
The hard part about this project isn't doing the port but keeping up with the constant changes from the TypeScript team, especially if the TypeScript team isn't invested in helping you out. Staying as close as possible to the source language like this will be a big win there.
One of the big benefits of a native language like Go aside from lack of a JIT is the ability to easily add shared-memory parallelism. I haven't studied the TypeScript compiler internals so I'm not sure how easy it is to parallelize their type checker. However, I assume that it would be relatively straightforward to parallelize the parsing of JavaScript files, so I'd hope for at least be a big speedup there.
I’m honestly not sure if there is any popular language where this isn’t true?
Shared-memory parallelism is necessary but not sufficient. Ideally you'd be able to only make some small changes in a few spots to greatly increase parallelism. I'm hopeful that TypeScript's AST parsing is a good candidate since ASTs can typically be constructed independently from the rest of the program state. Lowering and printing ASTs is also likely parallelizable. It would also be great if it was also possible to parallelize parts of the type checker somehow, but that sounds harder. I had some ideas about that in another comment. Anyway I think this means that it's likely that a port of the TypeScript compiler could realistically lead to significant speedups and may even lead to extreme speedups.
As far as language choice, it sounds like due to maintenance reasons the decision space is narrowed to native, garbage collected languages similar in object model to TypeScript but with parallelism primitives. Some candidates could be at least Go, JVM-based languages, or .NET-based languages. I think those can all be ahead-of-time compiled to a static binary? Another consideration is how easy/compact/fast/cross-platform binaries are. Go makes this trivial but I haven't used JVM or .NET AOT builds myself so I don't know how they compare in e.g. executable size or startup overhead. One advantage of picking Go is that esbuild has already demonstrated that Go is an effective choice for this problem domain, which may mean less work/research/unknowns. A disadvantage of Go here might be that the object model is different than TypeScript vs. something like Java or C#. But all else being equal the choice comes down to personal preference.
I'm not sure how well supported they are / haven't used them, but I think they enable this?
You could also try sending JS objects between workers instead of using SharedArrayBuffer. But that copies the memory, which can take a significant amount of time, and of course also excludes certain kinds of advanced parallelism that require shared memory (instead of copied memory). Another drawback of using the JS worker model like this is that having multiple JS VMs probably means fewer CPU cores available due to there being multiple independent garbage collectors. I observed an effect like this when I tried to parallelize esbuild's JavaScript plugins. Running them in separate workers seemed to cause the gains from parallelism to stop when there were half as many workers as CPUs. I'm guessing is because half of the CPUs are busy collecting garbage for the other half. Using a language like Go with real shared memory presumably makes it easier for fewer resources to be spent on garbage collection.
[1] https://github.com/parcel-bundler/parcel/tree/v2/packages/co...
TypeScript's compiler is an unholy, unsound mess, because JavaScript is an unholy, unsound mess. Sure the basics of type checking structs and variables is somewhat easy. But what about conditional types? What about weird generic constraints? What about indexing types? There's some really confusing stuff going on underneath the hood.
It's also a mess that *does not have a spec*. Even if you somehow manage to make a type checker that implements most of the features, there's no way to ensure it has 1 to 1 parity with TypeScript. Plus that parity can be broken at any point.
The only way I could see this happening is if the author somehow convinced Anders Hejlsberg or a significant portion of the TypeScript team to work on this new implementation. I'm not sure that's gonna happen.
I would personally be very happy to use a significantly slimmed down subset of TypeScript if it provided a 62x performance improvement. Slow type checking is currently the main drawback of TypeScript development in large projects.
I hadn't heard of this trick - how much improvement does it make? Seems like it might be good for readability, too?
The Typescript compiler performance wiki might also be of interest: https://github.com/microsoft/TypeScript/wiki/Performance
> I would personally be very happy to use a significantly slimmed down subset of TypeScript if it provided a 62x performance improvement. Slow type checking is currently the main drawback of TypeScript development in large projects.
We all think that until the slimmed down subset doesn't work with our favorite libraries. Sure, there may be people who use no dependencies, but I'd bet that there's sophisticated types in some very popular libraries that even low dependency programmers use.
edit: I see they are running it with 8 threads, in which case yes 50x is achievable. In any sizable codebase though, you should probably split into modules and run with some parallelism via e.g. `wsrun` or `turborepo`
tsc has in fact been designed with speed in mind (I've been following the project since before it moved to GitHub, which is when they redesigned it for speed). Going beyond 1 order of magnitude performance improvement while implementing the full feature set is highly unlikely.
I would bet on a port eventually being done and hitting at least 10X.
Isn’t that already available by running in “transpile only” and having the type checking happen in the background?
I'm just guessing but I think a lot of Typescript's really complex stuff is to support third party libraries with really complex types - at least they often refer to libraries they've improved support for when they discuss complex seeming additions. So I wonder how much of the JS ecosystem you'd have to cut off to get to such a subset. A particular example would be that I've often seen a chain of 20+ dependencies leading to Lodash, and I'd guess that Lodash is really complex to type. If you lose everything that's eventually dependent on something that's really complex to type, do you have much of NPM left?
Tanner Linsley (author of React Table, React Query, and other similar libs) has expressed similar concerns.
(I'm actually starting to work on a talk for the upcoming TS Congress conference about "lessons learned maintaining TS libs", and this will be one of the things I point to.)
It’s a form of linting, not semantics, so it doesn’t have to be exact.
So the only possibility I see is a webassembly alternative.
That means if this implementation passes all TypeScript's tests, it is as good as TypeScript.
Each new issue that is opened in either repo that shows discrepancy can be a new unit test added to the TypeScript codebase.
The only thing is it may not catch some type error, which JS already does none of and TS can skip wherever you want or even implicitly and people are fine with that.
If I get 98% of TS typechecking 10 times faster. It's a huge win.
Give the people a turing complete type system and they will write a complete application with it.
Presumably TypeScript has a test suite already that might serve as a starting point?
I think that's exactly why the author wrote this:
> Eventually, I realized that rewriting a massive project such as tsc would be extremely difficult to continue. But the performance gains were exciting. This led me to try another route: porting instead of a complete rewrite. So, I started looking at the TypeScript source code.
It will require ongoing work to keep up with changes to upstream, but by matching upstream's structure as closely as possible (given the language difference), the path to supporting these nasty little edge cases should be relatively clear.
As with so many terms in computing, there aren't universally agreed definitions. But in the author's post, I read "porting" as switching languages while changing as little as possible vs "complete rewrite" as redesigning/rearchitecting it along the way.
And in comparison, a "complete rewrite" would require a full understanding of the big picture design of the original code, come up with a perhaps different one in the rewrite, and keep the differences in your head as you go (or, forget about the original design entirely and solve it from scratch - and keep that in your head as you go). Far more work and risk, and it would have been a bad choice here.
More than once I've yelled at my screen, "HOW DO YOU KNOW THIS?!" about some inferred or narrowed type.
declare foo: never[] // it can only be an empty array
foo[1] // compiles
TypeScript is a nice type system, it's surprisingly expressive and flexible, like e.g. in Haskell I was surprised that there's this strange notion that for a given type you can only have one `Eq` or `Ord` instance for some type A.
But in TypeScript you can have as many Ord interfaces as you want. I may want to order some `User` type by last login, first login or whatever I want, but in Haskell you need to overcomplicate it and create a newtype.
This is a good thing - it stops you from using one Ord instance when putting As into a Map or Set, and then trying to pull them back out using another. Newtypes let you provide "additional" instances in a safe way.
I'll give you a trivial example I have some data type representing a TvShow, why should there be just one way to sort a collection of TvShow? Maybe I want to sort by year, maybe by length, maybe by category.
It's trivial to do in fp-ts, you can have as many Ord instances as you wish.
If your type doesn't have a natural ordering (as your TvShow doesn't) then you would use sortBy to supply an arbitrary compare function (just like what you would define in your Ord instance).
Ordering (and more specifically a total ordering) in mathematics is a set and a binary relation that has the transitive, irreflexive and connected properties.
Not even natural numbers have "uncontroversial natural ordering", as I guess even you can think of at least two different binary relations (<= and >=) that form two different orderings for N.
There's a lot of orderings more, here's one that sorts first odd and even numbers: "0 < 2 < 4 < 6 < ... < 1 < 3 < 5 < 7 < ...". And there's others like this.
https://en.wikipedia.org/wiki/Gray_code
Natural numbers in fact have (or better: form) an aleph one types of orderings. None is "special, uncontroversial or natural" just because we're used to think about the default <=.
The concept of ordering requires two things: the data type and the binary relation. It's a pair of things we can represent as (X, compare) but which compare you choose isn't implicit, natural or magical.
When talking about Haskell, it has the concept of type classes, where types and their behavior are bundled together. That's a design choice of Haskell which has pros and cons, but there's no mathematical foundation for a data type to have one preferred ordering.
Ocaml, Scala and most pure fp libraries I know do not have the limitations of Haskell when it comes to the concepts of ordering and equality, in fact in most of those when asking for an ordering ask you for a data type and a compare function. Exactly what the mathematical definition requires you to.
Not a typeclass where those are bundled together for some design decision.
Hope I clarified you that there's no such thing as "uncontroversial natural obvious" ordering, because there's no such things in mathematics.
declare foo: never[0]; declare foo: []; // fixed-length array (tuple) without any itemsAlthough i don’t know if TypeScript asserts that x[i] is T | undefined if x is T[] and the length is non-trivial. In strict mode this would be really useful as you can just do x[i]!, but it requires you to think a bit about index-out-of-bounds or at least throw on the undefined case.
There's a flag for that: https://www.typescriptlang.org/tsconfig#noUncheckedIndexedAc...
https://www.typescriptlang.org/play?#code/CYUwxgNghgTiAEYD2A...
const a = foo[1]
should result in a bring of type `never|undefined` or `undefined` depending on flow analysis
I don't think that's accurate. Typescript only has one instance of Ord per type and it's inconveniently implemented through the method toString.
To usefully sort anything other than strings in Typescript, you are using a function where you provide your own comparator at runtime. That's more analogous to Haskell's `sortBy :: (a -> a -> Ordering) -> [a] -> [a]` or `sortOn :: Ord b => (a -> b) -> [a] -> [a]` which don't put any type bounds on the type you're sorting.
const x: Thing = exprA;
const y: Thing = exprB;
const z = [x, y];
const z: Thing[] = [exprA, exprB];
Doesn't
const z = [exprA, exprB] as const;
type GnarlyThingTuple = typeof z;
TS is sometimes weird because it is a superset of JS and has to be compatible.
This is true of many programming languages projects. It's not uncommon for someone to write a language in a weekend, and then marvel at how their very simple toy language is faster than C! And look how quickly it compiles! Thus begins a several years-long journey, where with every feature added the runtime and compiler get slower and slower and slower.
Saving a minute every cycle sounds very attractive, but when a person spends a morning hacking and then finds out the normal tool chain doesn't accept their work when the optimized one did, they will quickly distrust it in the future. It'll also make them very reluctant to share it with their team.
The only reason my project was successful was we ultimately reached a set of compromises with the team that owned the woefully underspecified tool chain and upstreamed large swaths of changes. I would probably never do something like that again.
You need a test suit for this not a spec.
This problem has numerous solutions in Rust - (A)Rc, RefCell etc. I don't understand how this caused the author of the article to stop considering Rust.
tsc was designed expecting GC and it makes perfect sense to port it to another fast GC language.
I can't think of any reason why a program must have an ill-formed ownership model (unlike, say, the fact that many interesting programs must have an embedded dynamic type system (tagged unions etc) to do their thing).
Now. The Rust language and existing tooling doesn't give you any help in figuring out the ownership model, but I can easily imagine a language and tools that would help with that.
This isn't meant to refute anything you've said (you said "[existing] non-GC languages are bad in general" not that non-GC languages cannot be good) - just add clarity.
It reminds me of a programming assignment I had 20 years ago. We had to take our teacher's old FORTRAN program and "convert" it to Java or C++. I knew both equally well at the time - and I picked C++ because it had goto whereas Java didn't.
This meant that I didn't really have to unravel the meaning of the program. Whereas with Java, I would have to.
Compilers tend to use a bit thornier data structures and graphs than you might see in other realms. It is very convenient to just represent these graphs as objects connected by pointers, and know that you don't have to worry about objects that become unreachable.
For example, the values in an SSA-based IR are all full of pointers to other values. Each value represents the computation to produce that value, like "x12 / 4". A strength-reduction pass goes in and mutates this node, replacing it with "x12 >> 2" (if such a change is valid). It knows the change is valid because it follows pointers from the node to its inputs. The change is available to all upstream values immediately because it is mutated. These changes may make other values unreachable, at which point, you don't care about them. You only care about the values which are reachable from computations with side effects. These nodes definitely have cycles and IMO, the notion of weak references between them doesn't make sense.
I'm sure there's a way to do that in Rust. Please, nobody try to explain how. I could figure it out. The thing is... it's not obvious how to do it in Rust, and figuring out how to "explain ownership to the Rust compiler" without creating Rc<RefCell<T>> cycle leaks is a non-trivial task that sucks away time I could be spending writing the compiler.
This reminds me of a paper I saw go by years back about writing a compiler in Haskell. Pure Haskell code doesn't let you mutate objects and doesn't let you do pointer comparisons, so it creates problems if you're trying to implement an algorithm which is conceptually pure (at a high level, a function with inputs and outputs) but where you're taking the implementation from a paper which constructs the output iteratively, piece by piece. The problem is that your data structures have to be annotated to show which parts are mutable (IORef, STRef, or TVar), but "which parts are mutable" may have a different answer depending on whether you are inside some function transforming or constructing a data structure.
It was obvious to me, reading the paper, that while I love functional purity, and functional purity solves all sorts of problems and makes it easier to write code in general, it sometimes gets in the way of getting shit done. The same is true of Rust's notion of ownership... super useful when it lets me write fast, efficient code without having to think about ownership (when it just works and gets out of the way), and super annoying when you have to figure out how to explain your program to Rust.
However, I still stand by my point - Rust might be bad at graphs, but I believe that a well-designed language with a borrow checker (maybe something closer to Lobster, which automatically inserts RC cells when you try to multiply mutable borrow[1] - something obviously more correct than what Rust does, I'm not sure how they messed that one up) wouldn't necessarily have to be.
[1] https://aardappel.github.io/lobster/memory_management.html
And I think it could/should work also with Rc<> but for some reason, rust authors decided it's important to write everything as if it was multi-threaded even if it's single-threaded now (so it can be easily turned to multi-threaded later).
And that's cool but it adds a lot of burden to developers. I am probably missing something, there is likely a good reason for this, as always, but it doesn't change the point, rust is super-complicated for hobby stuff and you have to think about irrelevant things, hence for me, it is low-level language (with generics and macros and very cool builtin unit testing but it's still low-level)
And by irrelevant I don't mean mem-safety, that IS important for me, I rather mean worrying about race-conditions in a single-threaded code :)
It sounds like a more ergonomic system that has the same technical shortcomings.
TSC is not an interpreter, but its type system is Turing complete, so I wouldn't be surprised if it has some similar issues.
Which tooling are you talking about?
The Rust compiler is both dog-slow and massive (both from a source and binaries perspective), and doesn't have a working incremental compilation mode yet, or support in-process hot-patching. There's no Rust REPL (hacks like papyrus don't count). Poor structural editing support. Integration with various editors/IDEs is lacking (e.g. there's no support for reporting possible performance issues with code constructs in any editor that I'm aware of, nor is there in-editor per-function disassembly or LLVM IR inspection, or borrow-checker lifetime annotation, no extract-function refactoring).
> support in-process hot-patching
I assume that we are comparing this with say C++ compilers. I think that Visual Studio added this recently but idk if saying this is a valid criticism of the compiler.
> There's no Rust REPL
Are you comparing it with C++ compilers?
The Rust tooling is so much better than C++ tooling that it's not even funny. You are right, there are places where this could be improved but just cargo makes my life so so so much easier than any other package manager.
Relative to Rust when doing a cold-compile, maybe, but certainly not relative to other languages like C, sane C++, Go, or Common Lisp.
I'm comparing Rust tooling with existing tooling in other languages in general, that is feasible to implement.
Common Lisp has multiple compilers (SBCL, Allegro, Clasp) that compile code nearly instantly (within a second for my modest 2kloc codebases, and individual functions in milliseconds), have REPLs (that also do native-code compilation under the hood using the compiler proper - no hacks), and support hotpatching.
Visual Studio implements in-process hot-patching, too, so between C++'s highly static nature, and the fact that SBCL is run by a tiny group of unpaid contributors, there's no reason for the Rust compiler to not have it.
Meanwhile, GHC (compiling Haskell, a statically-typed language) has a REPL, so it's clear that being dynamically-typed is not a requirement.
From a user's perspective, having either long compilation times or lacking a REPL is OK, but not having either is unacceptable.
Having a good package manager is necessary for having good tooling, but not sufficient.
But if the author’s more familiar with Go, performance in Go is likely good enough
I found the following article interesting on the topic : https://aminb.gitbooks.io/rust-for-c/content/graphs/
1. You have a graph struct that has ownership of all nodes (in Rc containers).
2. Nodes have Weak references to connected nodes
One way you could get this to work at the cost of some safety (depending on how you do it) would be to use arena style allocation. Obviously there are ways to do this safely in Rust, but it is probably easier to do it unsafely for a port.
However, if you just use a high performance language with a GC like Go/C#/Java, the port will be easier and you won't have to worry about doing too many allocations.
Hopefully the whole toolchain will be embeddable, too. Being able to do configuration-in-code and produce an executable that does the whole build step in milliseconds would be cool.
For everything else I rather enjoy the productivity of automatic memory management.
That being said, porting this kind of codebase to Rust would indeed be really rough—definitely possible, but by no means pleasant. I like to think the platonic ideal Rust, which we're hopefully getting closer and closer to over time, is capable of elegantly handling cyclic data structures and the like even while encouraging a single-ownership model for the 95% of cases where that's the right design. I don't think the Rust we have today is there yet.
Better than other dynamic languages, definitely. That is about it.
Plenty of languages that offer AOT compilation and automatically managed memory to chose from.
> tsc uses a lot of shared mutability and many parts depend on garbage collection. Even though I’m an advocate and believer in Rust, it doesn’t feel like the right tool for the job here. Using Rust required me to use unsafe too much for this specific project.
> tsc depends on shared mutability and has a cyclical mutable reference. Rust is designed to prevent this behavior. Having two references to the same data (shared mutability) is undefined behavior in Rust.
I'd be curious to hear more. In particular:
* Two references to the same data: were cells too awkward?
* Actual garbage collection: my first instinct would be to use an arena for each compilation, destroying the whole thing at once rather than doing reference counting or garbage collection. I gather that's not cool in the age of language servers and incremental compilation, though. Does tsc support those?
As I understand it, in this case, Rust would be able to have a line to line port from JavaScript.
I do still think your overall point holds. Why not use a library in the Rust ecosystem to help with this problem e.g. some GC<T> type.
Help me see it?
I can easily see how getting rid of the cyclical structure crosses from "line-to-line port" into "rewrite" territory. I'm not saying you're wrong, but I'm having more trouble seeing how "wrap some types in cells and pass along a lifetime and arena parameter" crosses that threshold. It might not be feasible at all, though, if tsc compilations are long-lived for language server + incremental compilation support.
> Why not use a library in the Rust ecosystem to help with this problem e.g. some GC<T> type.
iirc I've seen some experimental GC library mentioned. I think something like that is at least off the beaten path enough that I can see how the author might decide it's better to just use Go instead.
> Several reasons:
> It's written in Go and compiles to native code.
> Parallelism is used heavily.
> Everything in esbuild is written from scratch.
> Memory is used efficienty.
Using another language will solve 1 and will maybe solve 2. From this FAQ too:
> For example, many bundlers use the official TypeScript compiler as a parser. But it was built to serve the goals of the TypeScript compiler team and they do not have performance as a top priority. Their code makes pretty heavy use of megamorphic object shapes and unnecessary dynamic property accesses (both well-known JavaScript speed bumps).
tsc itself seems like it could be optimized. By how much, I have no idea. But it sounds like it would be easier than writing something from scratch. esbuild, while a great tool, is limited in scope. Typechecking Typescript on the other hand is a moving target, and a fast one at that. All in all, I'm not convinced by this approach compared to "regular performance work" on tsc itself. I think people talk about that because it's easier to start your project than to change tsc. In that case, the problem sound political rather than technical.
There's the x62 speed from the article, but there's no info on what features are supported. I feel like this is something where the devil is in the details.
I'm happy being proven wrong on that subject, I fear that we're going increase the number of tools in the frontend ecosystem for no good reasons, and make every a bit harder again.
A direct port of tsc to a "faster" language doesn't guarantee a faster type checker. Sure it might be a bit faster, but my guess is that the large performance wins come from a different design that takes performance into consideration from the start.
Maybe the thought process is that a 1:1 rewrite will handle compatibility and feature completeness? I suppose you could optimise from there?
I'm making a Typescript type-checker in Rust: https://news.ycombinator.com/item?id=30033119
In those cases, https://github.com/fitzgen/bumpalo works amazingly as an arena. You can pretty much forget about reference counting and have direct references everywhere in your graph. The disadvantage is that it's hard to modify your tree without leaving memory around.
We use it extensively in http://github.com/dioxusLabs/dioxus and don't need to worry about Rc anywhere in the graph/diffing code.
I presume that if the bottleneck was just number of loop iterations, V8 would already be about as fast as C or any other language?
Is it simply that accessing/mutating javascript data structures like objects, arrays, set, etc. is slow? Are those not fully optimizable by V8 for some reason? Or is it something else like the GC being slow?
Given that gopherjs[1] can compile Go to JavaScript and in some cases thereby achieve better performance than the native Go compiler[2], it's not obvious to me that a JavaScript program ported mechanically to Go is likely to be any faster than the original. It seems to me that the effort might be better applied to optimising the performance of tsc itself.
[1] https://github.com/gopherjs/gopherjs [2] https://medium.com/gopherjs/surprises-in-gopherjs-performanc...
Every language in wide use today is still evolving and, if anything, it seems that the pace of language change is generally increasing.
They are good willed but TypeScript doesn't have an official spec, it's an implementation-defined language. It's not C, it's not JavaScript, it's whatever each release wants it to be. This makes it very difficult to follow every version and evolution, you're consistently chasing whatever has been merged on release.
Also on "it's whatever each release wants it to be" - that is hell on earth if you're looking to target consistent software processes and it only gets worse the more you scale (both solution and team).
I get the value of Typescript - hell I've used it extensively myself... it's just that I can't stop feeling like this is all an anti-pattern: trying to make a loosely-typed language a typed language (if you can call it that?). =/
I'm happy to see this, as honestly I don't really understand why you would compile TS without type checking.
I don't think I'm remotely alone in doing this but I'll share from my perspective. I have two modes while writing TypeScript stuff: 1) prototyping something where I don't want to spend time writing types (or maybe I write some types but I don't care if they are correct or line up yet) but I still want to have JavaScript generated so I can test the thing out manually and 2) getting types correct as part of long-term maintenance and edge-case removal.
There are fast TypeScript -> JavaScript generators today like swc and esbuild. tsc is not fast. So I can speed up "1)" from above if I use swc and esbuild and only use tsc for "2)" from above.
As people write faster implementations of tsc-the-type-checker though I'll be happy to use that instead and just not use tsc anymore.
It doesn't compile to JS, but this is one of the canonical use cases for Stanza 's (1) "optional" type system
https://datastation.multiprocess.io/blog/2021-11-13-benchmar...
It also makes it easier to determine what behavior is worth testing and which ones are already covered by your types, if you tend to write tests last. If instead you're used to Test-Driven Development you can do types->tests->code.
But no I don't prefer it to what I do in TypeScript.
If I really need to call tsc explicitly, the few seconds that I have to wait, versus the minutes on some other languages, won't make me lose sleep over it.
When it comes to starting up a development server & building the client, there's a huge cost to repeatedly typechecking the same 1000+ files. By cutting out typechecking & only compiling, I can reduce the webpack client build from 40 seconds to 3 seconds (using sucrase).
I maintain a fairly large TS code base (nodejs backend). I think a full rebuild is 1:30 on my relatively recent laptop (i7-8650U, lots of RAM). But in practice I always use tsc -w and compile is mostly instant after editing a file (I do have to wait whenever I launch the command after I boot, but after that, it's fast enough).
tsc now support incremental compilation too, though I haven't played with it too much as I'm happy with watch mode.
Personally I only have a small number of projects that take more than 10-20 seconds to compile, but those ones are painful. I should probably do the same with -w for those.
If it's not in your CI/CD scripts, it's not actually getting checked.
> I can reduce the webpack client build... to 3 seconds
This is about 10x longer than any interaction with a computer should be.
You can make a similar argument and say you must write in a sound type system language, and TS typesystem is unsound.
Not great... I'm ashamed but it's just me on this project atm.
edit: misread parent originally
tsc --noEmit duration is 2/3 of a normal build on our fairly large code base, so that can definitely speeds things up (for CI for example)
Another possibility is to use something like GraalVM native-image to compile Typescript's type-checker into a binary. This might offer a speedup over its execution on V8.
To give another example of where this came up: they were discussing using Zig for the Ruby JIT, but despite really liking Zig the team decided to go with Rust because of its maturity: https://bugs.ruby-lang.org/issues/18481#note-21
That's because you're looking at it purely from only one point of view - the number of available features in the language itself.
What's unique about Go is not so much about the language itself as the approach of its authors to creating it. Go hasn't meaningfully changed in the 10 years of its existence. That's not an accident - the bar to add anything is extremely high. A feature needs to be punching way above its cost to even be considered.
If you look around the programming language space, you'll see the exact opposite attitudes, features get added into languages willy-nilly. Not much point in having different languages when they just trade features until they become indistinguishable from each other.
There are only three other languages that I know of that at least sort-of follow a similar philosophy to Go: 1. C 2. Zig 3. Elixir
You aren't going to be writing general-purpose code in C, so that's out. Zig is about a decade away from being a contender. Elixir is a valid choice, albeit much more niche and unfamiliar to the average programmer than Go is.
So if you want a language that'll be there for you in the long haul, that will not invalidate what you've learned about it every 3 years or so, you're left with Go.
Java has been going with the last-mover advantage of only ever incorporating a new feature when it proved to be useful by other languages. That has been going on for 25 years, so it is definitely queen of this category. Not sure why it is seldom appreciated, but it is also a very small language (compare the number of keywords it has by c# for example), and it was even named “blue-collar” for a reason. So I still don’t see Go anything new even from this perspective.
The compiler is blazing fast and you get small fast statically linked executables by default. That's very appealing to me even though the language itself is a bit spartan.
Writing a parser for the purpose would achieve performance closer to the other implementations.
The main pattern esbuild uses is an interface with a single dummy method to denote a union type like this: https://github.com/evanw/esbuild/blob/34899aaa1d76acd3b4adc5.... It's used several times and is basically esbuild's core data structure. I'd like to be able to optimize this pattern. Specifically I'd like to avoid each reference to a union type taking up two whole pointers in memory (Go represents interfaces as a pointer to the object and a separate pointer to the method table).
I'm only using the method table as a tag for the tagged union, so ideally it'd be a single byte or something even less expensive like part of the pointer. I don't think generics can help with this? But Go doesn't let you do fancy stuff like this so I'm just leaving it be. A low-level language like C++/Rust could do better here, but that comes at the cost of a significant decrease in productivity, so I'm ok with this trade-off.
I’m a crystal fan though, would love to see more tooling built with it.
Sure but this goes against "Don't communicate by sharing memory", no?
Yes, but that's advice rather than something Go requires. It's reasonable to say it's outweighed by the desire to avoid significantly restructuring the existing logic. The author is explicitly choosing to match the upstream program's structure to limit the project scope, and whether that structure is good or not is almost besides the point.
btw: especially in the Rust community, shared mutability doesn't always mean shared between threads/tasks. It's also a concern in single-threaded/non-concurrent code due to aliasing. This case does actually seem to be involving multithreading given that the author mentioned "using 8 threads".
* Rust references are strict. &mut references guarantee that nothing else can access the backing memory while the &mut is valid. & references guarantee that nothing (including &mut by the rule above, or unsafe code with raw pointers by programmer's pinkie swear) can change the backing memory while the &mut is valid. This allows the compiler to perform more optimizations behind the scene, like assuming a value loaded into a register is still up-to-date, even if there's been a function call between.
* Rust raw pointers make no guarantees (but require the programmer to type "unsafe" to use).
* C pointers are IMHO weirder. iirc, two pointers of the same type may alias (unless you use the rare "restrict" keyword), but two pointers of different type can't, unless one is "char*".
* AFAIK, Go pointers are at least as permissive as C's. They might not make any aliasing guarantees at all.
* Other languages vary...
Strange reasoning.
Sure, if you absolutely wish it to, in an `unsafe` block, you can define two writeable pointers to the same piece of data. I don't think I've ever seen anybody write such code in Rust (except perhaps when linking to a C library that required it?), nor have I ever felt the need to do so.
Now, it is true that Rust is designed to discourage writing cyclic data structures. I can imagine that trying to port (rather than redesign) an algorithm that deeply requires cyclic data structures, that will make your life miserable.
I'm just trying to decypher that cryptic sentence from the OP.
It does tend to imply the new tsc isn't going to be able to take much more advantage of concurrency than the current one, which may put a limit on what performance gains it can expect. I don't know if the current tsc is able to do anything concurrently. (Here, "concurrently" doesn't merely mean "able to have multiple events in flight and thus do multiple tasks at the same 'time'" but truly "doing work on multiple CPUs simultaneously".) Go is generally faster than Node, but it'll be hard to guess by how much in this case.
A Rust port would be harder but would probably lead on in a direction to improve the concurrency. However, that may be an infeasible amount of work, not because of Rust per se but because a port is intrinsically easier than a port and a rearchitecting.
CPUs have no problems with mutability. JS is slow because of dynamic typing, dispatch, flexible object layout, and an object model that requires lots and lots of heap allocation and pointer chasing.
But moreover, my takeaway was that the author is making a distinction between a complete rewrite and a port, where the former seems to be any rewrite that just tries to do the same thing (but may be architecturally very different), while the latter seems to be an approach where you attempt to directly adapt the original source code into a new language on a statement-by-statement basis. The latter would likely require the target language to support the programming paradigms which are used heavily in the original source code.
Or you could, you know, use the JS typechecker already built with large scale performance in mind (and in native language) - Flow.
I know Flow hasn’t had the bandwith to invest in supporting open source, and Typescript pretty much “won”, and there is now a large amount of library definitions without Flow support, but: If your codebase is large enough to suffer from TS’s architecture, you might have the resources to deal with the hurdles of adopting Flow. Flow still has some better designs such as nullables (?T).
That said, Flow is now being heavily optimized for large, strictly typed codebase, which means there is now no typechecker geared towards checking JS with minimal type annotations. I think that would still be useful for the prototyping stage of a project.