Why Is Esbuild Fast? | Hacker News Reader

Why Is Esbuild Fast?

esbuild.github.io

320 pointsmef166 comments

In posts talking about how someone made something fast, I see the same idea repeat time after time: it has linear time complexity, has great cache locality, and saturates all available processors.

It's annoying to me how much time I had to spend outside of school before I realized that this is the real recipe for fast code. In particular, even code that is nominally O(n^2) by standard measures is still a dis-economy of scale: the bigger the problem the slower it gets. In school, there's a premium given for finding a polynomial-time solution, but in practice every job where I've been tasked with making something fast, only linear time complexity was good enough.

In practice these linear algorithms tend to also be completely data parallel and have great cache locality. Not only do these properties make the code fast the second you harness them, but then tend to be exceedingly simple architectures to maintain over time.

For some other reading I really like on software performance in the real world:

- https://blog.nelhage.com/post/why-sorbet-is-fast/

- https://blog.nelhage.com/post/reflections-on-performance/

> but in practice every job where I've been tasked with making something fast, only linear time complexity was good enough.

But of course we remember from our CS school that some things simply cannot be done in linear time. I’m not sure what happens when your employer tells you to sort a big list and insists that only linear time complexity is good enough.

To that I find that the answer is usually: find ways to avoid needing to sort the list! This is more viable for many problems than you might otherwise think.

There’s also vastly faster ways of sorting lists of bounded numbers (or tuples), the most linear being: cut straws of the exact right length labelled with an Id, bang them on a table so the ends without the ids all line up, read the ids back in order.

I only said it took linear time, I didn’t say it was fast...

> some things simply cannot be done in linear time

Boy am I glad I didn't take CS

For example, a comparison-based sort can't do better than O(n log n) time in general. More information here:

https://en.wikipedia.org/wiki/Comparison_sort

That's smart. I didn't take Physics and now I can travel faster than EM waves in a vacuum.

"I'm cool because I never learned some things"

"webshit pride"

It was a lighthearted comment because the concept went over my head.

Anything you didn't learn in Christian Science courses at university, you can find and learn more at your local Christian Science Reading Room! Most cities and towns in the USA have one, and even many airports for people on the go. If you're flying through SFO, you can visit the one in terminal 2!

Radix sort exists

Good luck using that for a big list of strings!

If we are talking about linear in number of characters, an easy answer is building a trie and traversing it left to right. (Cache-friendly variation: https://en.wikipedia.org/wiki/Burstsort)

Sure, but it has the classic problems of the memory/storage required for the sort (minimum double all your keys), and a classic math issue relating to it's speed being O(k*N) and since k = number of bits/digits, N can only be as large as the number of bits/digits allow for.

This means quicksort and the like end up faster in practice for any "sort this list" operations. Radix/bucket sorting can be SUPER awesome when what you need is a single level of bucketing at all rather than exact order.

Sure, but it has the classic problems of the memory/storage required for the sort

No problem, just create a file of the necessary size on disk if you don't have enough RAM available.

Today's comment was brought to you by a large number and the letters big-O and c... where c, a large number, is the hidden constant of proportionality waiting to bite you every time you mention big-O. :-)

Huh?

If we're reading from disk, then we DEFINITELY don't want radix, as then it's almost guaranteed that all time will be dominated by loading pages rather than cpu cache locality being leveraged by linear time algorithms - which is what this entire discussion is about.

He was sarcastic

Radix sort is still O(n logn). More precisely, it's O(w * n) where w is the size in bits of the data being sorted, but then it follows that w scales with log n, since n distinct items requires log n bits and hence you arrive at n logn.

At one job we used to joke "i don't care if it's right, as long as it's fast"

Of course, it was only half a joke. For some pages, we really did want data quickly, so we were willing to accept stale data or somewhat inaccurate data, it that helped. For other pages/sections, accuracy was more important so caching wasn't possible or invalidation was critical.

But product design is always about tradeoffs between what features are desirable, what features are possible, what features are easy to build, and how features interact. Features that are possible but not easy to build can sometimes be replaced with features that are easy to build and close enough.

Maybe they expect you to ask how many elements are out of their place.

The answer is you get stuck maintaining a cache and playing whack a mole with whatever performance metric execs care about this month

Ah, yes, the good ol' Stalin Sort!

Quasilinear is still fine too.

> in practice every job where I've been tasked with making something fast, only linear time complexity was good enough.

Presumably you weren't tasked with making something fast, were linear time complexity was overkill. In other words, quadratic complexity may have been fine in many cases, but you weren't tasked to make those fast.

I'm not saying you claimed otherwise, but somebody might misread you as saying "in practice, only linear time complexity is ever good enough".

To quote a recent post: quadratic time is good enough to land to production, and bad enough to fail in production.

That's a pretty good quote!

It's also worth noting that if your size is bounded a simpler quadratic algorithm might be more maintainable overtime than a linear one (if it's easier to grok). Say max(n) = 25, it doesn't really matter if it's linear (say 4N = 100) or quadratic (N^2 = 625). In practice they're both fast enough (even though one is 6x faster) and not worth optimizing but if it can grown unbounded that's when the BigO runtimes start to matter. For something like ESBuild your input size grows larger and larger as your codebase increases and that's when you start to see slowness creep in.

Esbuild looks amazing. It seems to have been written almost exclusively by one guy (evanw) during 2020. Just the main JS parser file [0] is 12k lines. According to the Github stats he has committed 280k lines, that's almost 1000 lines per day every day since he started. Amazingly productive.

[0]: https://github.com/evanw/esbuild/blob/master/internal/js_par...

Not to be the one who is the one, but 1k lines per day sounds like the opposite of productive. If you still have to add/remove such large swaths of code, you're surely doing something weird.

Still a great achievement to produce a bundler and minifier as a one-person team, but not sure we should use the amount of code line changes as a measure for productivity.

Not weird, it's early, messy development. Perfectly normal (for me anyway)

> …1k lines per day sounds like the opposite of productive.

Relevant Macintosh folklore: https://www.folklore.org/StoryView.py?story=Negative_2000_Li...

Productivity might be the wrong word. Discipline, grit, perseverance?

I've written parsers/lexers/code generators by hand for much smaller toy languages with no thought about performance at all and those were huge undertakings for me. I was just trying to convey my impressedness with some numbers that we can relate to somewhat.

But yeah, he has a "net" loc contribution of 284-148 = 136k lines of code. Assuming the code base is resonable (which I don't doubt), it's a pretty big project which he has built, which I'm impressed by.

If you read the linked page, it was a very intentional decision:

> Everything in esbuild is written from scratch.

> There are a lot of performance benefits with writing everything yourself instead of using 3rd-party libraries. You can have performance in mind from the beginning, you can make sure everything uses consistent data structures to avoid expensive conversions, and you can make wide architectural changes whenever necessary. The drawback is of course that it's a lot of work.

> For example, many bundlers use the official TypeScript compiler as a parser. But it was built to serve the goals of the TypeScript compiler team and they do not have performance as a top priority. Their code makes pretty heavy use of megamorphic object shapes and unnecessary dynamic property accesses (both well-known JavaScript speed bumps). And the TypeScript parser appears to still run the type checker even when type checking is disabled. None of these are an issue with esbuild's custom TypeScript parser.

1k lines per day sounds pretty reasonable when writing your own TypeScript parser, among other components

He's also likely referencing existing implementations and so likely can create more code faster than having to think of everything from scratch.

He's obviously changing things to be more optimal as he goes, not transcribing.

> 1k lines per day sounds like the opposite of productive

Having built a compiler before, this seems normal, especially since the parser is written manually (as opposed to generated). Parsers are always a lot of lines of code.

Not all of them are the most complex, and in fact a lot will just be defining all the different nodes of your AST and branching on them, which, as blocks do, tend to inflate code a lot. I wouldn't expect a JavaScript parser to be any more compact though, or the development (especially in a codebase with few contributors) to be any less "thrashy".

> that's almost 1000 lines per day every day since he started. Amazingly productive.

I know what you mean, but I also hate using lines of code per day to measure productiveness, and I hate it when managers in corporations look at number of lines added in PRs to measure how productive a dev is.

Eh, it's a useful proxy. 1000 lines a day isn't better than 100 lines a day but both tell you that it's being actively worked on.

It is actually a good way to estimate the effort and complexity of the problem if you know how incredibly competent the programmer is.

I don't know about anybody else but for me a good approximation is the time interval between the first line of code and the critical line of code. The better my solution the shorter it often is. In particular, 'shorter' in the sense that future functionality doesn't have to constantly acknowledge that code, creating a tax on new features that builds until the project dies.

I was going to say "start of coding" but some tasks are just so boring that I may procrastinate on starting it. That also indicates an element of difficulty (in giving a shit), but not what we're trying to measure here.

I discovered the Mikado Method quite on my own as a consequence of trying to tackle increasingly complex cases of 'large-scale' or 'top-down' refactoring before accepting that as the oxyest of morons. I was sold when I realized that when you finally discover the crux of the problem, the 'Rome' that all roads lead to, between half and 80% of the code you just wrote isn't strictly necessary. Throwing it all out wholesale is how you avoid code hoarding. Yes, once in a while you go out to the proverbial trash can to pull something back out that you actually did need, but the tabula rasa aspect is profoundly useful psychologically, especially for that 'tax' bit I mentioned above.

There is always some function in the middle of the call tree, that if you add or change the meaning of an argument then the feature or bug fix becomes a logical conclusion of that new semantic. Either the code above it or below it hardly needs to change, decoupling the feature from all but a handful of functions/concerns.

A bad programmer will happily plow through adding a new argument to fifteen method calls and then propagating that change to a hundred call sites. And that code will be buggy as hell because their coworkers have already written them off. If it's too much code for others to review, you can be sure there are bugs in it. And if you didn't listen the last ten times people told you to stop writing so goddamned much code, you aren't going to listen this time, either. So have at it, sport. We'll just make sure you get the blame for the bugs, until someone in management wakes up. If they don't, then they see that the sections of code people "won't touch" but you will are getting bigger, and mistake your ownership of this code for a sign of prowess instead of a sign that you are inspiring apathy in others.

Just to put the 280k lines into perspective, deletion rate seems close to addition rate[0]; measuring the number of changes made to code is also a poor proxy for productivity, but is at least not quite as bad as the pure "lines added" metric.

[0] https://github.com/evanw/esbuild/graphs/code-frequency

Name a better duo than Go and massive LoC counts.

Seems like esbuild and its users are getting a lot of value out of those LoC counts.

Yup, it was literally designed for large-scale software development. Good article https://talks.golang.org/2012/splash.article

I think Go is a genuinely fantastic experiment testing the idea that it's actually easier to write and modify long, boiler-platy, but painfully explicit code.

The syntax was designed to be "just right":

https://talks.golang.org/2012/goforc.slide#10

https://talks.golang.org/2012/goforc.slide#11

https://talks.golang.org/2012/goforc.slide#12

I don't think that's even up for debate. As far as I'm concerned they have definitely met their goal.

I find it similar to the fact that, while I am thankful that the operator's manual for my toaster was written at a 4th grade reading level, I find it unlikely that anyone has that task in mind when they say they want to be a writer when they grow up.

Look at how good the release notes are: https://github.com/evanw/esbuild/releases

Some serious work there.

Seeing someone do almost daily work on a personal project, no matter the commit size, is something i find awe inspiring

> Just the main JS parser file [0] is 12k lines

While that is impressive, I think this is also an indication of a problem: the grammar is becoming unwieldy. For example, even for someone as prolific as Evan, he must decide between feature depth (e.g. bundle splitting) and feature breadth (e.g. implementing the equivalent to @babel/preset-flow, which is used by both flow and hegel[0] type systems)[1]

Esbuild supporting Typescript and JSX is undoubtedly a byproduct of these grammar extensions having become popular. But supporting extra grammar extensions does add to complexity, sometimes in non-trivial ways. In Typescript, for example, you can import types using runtime import syntax, making it ambiguous whether running side-effects from a library is intentional or not. This gets problematic once you consider treeshaking and not-really-standard things like conditional resolution via package.json's `browser` field.

It gets even more fun when you realize that module resolution isn't even specified, meaning that as far as a bundler is concerned, something like `import 'lodash'` means completely different things for a browser vs a project installed via npm vs one installed via yarn v2...

[0] https://hegel.js.org/

[1] https://github.com/evanw/esbuild/issues/79

> It gets even more fun when you realize that module resolution isn't even specified, meaning that as far as a bundler is concerned, something like `import 'lodash'` means completely different things for a browser vs a project installed via npm vs one installed via yarn v2...

Ugh, why does it have to be different between npm and yarn v2?

I'm an occasional JavaScripter and yetserday wrote an automation with Puppeteer.

The first line I wrote was

    import puppeteer from 'puppeteer'

only for Node to fail on that. I thought that syntaxt was supported, but I had to change it to

    const puppeteer = require('puppeteer');

I've no idea if it's a problem with my Node version (12) or what, but it was a real "Huh?" moment.

Modules weren't production-grade until Node version 14. Before that, it used CommonJS modules ("require" instead of "import").

It's because Node was invented before EcmaScript modules existed. CommonJS worked well and supporting both was awkward, so the transition took a while.

FWIW, that's not an `npm` vs `yarn` thing. That's a "Node has only recently added support for ES module syntax and it's still mostly experimental" thing:

- https://nodejs.org/api/esm.html

- https://nodejs.medium.com/node-js-version-14-available-now-8...

Its supported pretty well now but you need to explicitly switch ES module support on in nodejs.

The easiest way is via the "type": "module" flag in the package.json file for your project.

It's an odd thing. The way people talk, you'd think NodeJS and JavaScript are synonymous. Turns out, NodeJS is this generation's JScript, including shoddy support for the language proper, and it includes vendor-specific extensions that you're expected to use instead. It has somehow though managed to escape any of the criticism that gets thrown at other projects pilloried for stuff like this.

Node came out way before any of these features were invented, and the ES committee knowingly broke compatibility when designing for the browser. Modules are supported since v14 but you have to opt in.

They didn't break compatibility. NodeJS never attempted to be compatible with the Web. The design work that TC-39 and the browser makers did is also much better than what the NodeJS community puts out. And it's a design that was in ECMAScript 6. Six years later, and you still have to opt in. The TypeScript team got it right, working in a way that NodeJS should have been taken lessons from.

The design work that TC-39 and the browser makers did is also much better than what the NodeJS community puts out

Like the dumb broken Maps and Sets?

Can you elaborate what you’re referring to?

How are they broken?

Yeah, technically better, at the cost of huge complexity.

CommonJS was easy to implement anywhere and dead simple. We only need static analyzers and tree-shaking optimizations because the JS ecosystem went wild on dependencies. I honestly think we were better off 10 years ago.

> NodeJS never attempted to be compatible with the Web

It goes the other way around? Node caved in and support modules now, what has been done in tc-39 for web modules to be compatible with node (considering it predates all of it)?

There is no “JS community vs Node community” btw.

> Ugh, why does it have to be different between npm and yarn v2?

Mainly just because yarn v2 went off book[0]. npm vs yarn v1 is a more sane wold.

[0] A perspective on Yarn v2 from the creator of Yarn v1: https://twitter.com/sebmck/status/1300664946645069830

evanw appears to be Evan Wallace, the cofounder and CTO of Figma (!?)

Yup, that's the one! He's also the author of http://madebyevan.com/. I remember seeing that WebGL water experiment many years ago and being blown away. That FSM Designer came in handy for some classes where I had to draw FSMs too.

I'm baffled and confused that he manages to run Figma and work on esbuild at the same time.

Same, hence the (!?). lol

Yup, impressive.

SWC, another competitor project written in rust is also handled exclusively by a single person. Half a million lines of code and they have even built a type checker for typescript which is not included.

https://github.com/swc-project/swc/graphs/contributors

Evan Wallace (the author) is also the co-founder and CTO of Figma. Absolutely remarkable engineer. His co-founder Dylan is an investor in our company, and I can’t seem to get in touch with Evan at all because he’s always super heads down on something meaningful (such as esbuild). Mad respect for him.

How does he do Figma CTO-ing and at the same time building esbuild? That’s mad hours.

Also one of the things I love about Figma is their obsession on performance. It feels such a joy to use. Runs circles around any Adobe product. It’s written in a browser. Figma and vscode are my inspiration for browser based tools.

>How does he do Figma CTO-ing and at the same time building esbuild? That’s mad hours.

My guess is an alignment between purpose, fun, and skillset. I would say (not for the sake of bragging but for qualifying the legitimacy of my position) - I probably pull similar hours across CTO-ing, investing, and learning biochemistry. Then again, I'm commenting on HN and Evan isn't, so maybe not. :P

I switched from google closure to esbuild and it is amazing how fast and well it works. It even generates smaller code in my use case (200k lines app). Also the creator is really nice. He implemented two features I requested right away, which surprised me a lot. What a great project!

I submitted a couple PRs to esbuild and also found the author to be kind and helpful.

I've seen other interactions in the issue tracker and the author has been nothing short of responsive and understanding. This is a project worth contributing money to.

Out of curiosity, was that with closure's simple or advanced optimizations?

Simple optimizations, Advanced always broke my code.

Important distinctions when comparing Closure Compiler's performance to other tools:

1. Regarding bundle size: Optimization level used (SIMPLE optimizations yield output that is at least expected from a minifier these days)

2. Regarding compilation speed:

* Whether the JavaScript-only compiler was used over the JVM compiler: besides being slower, a number of optimizations are not available to the former.

* In many conditions a pre-heated JVM (or Nailgun), or using the NPM-distributed native compiler binary yields faster compilation.

* Which compiler flags have been tuned; e.g. such that the parsing of browser externs is bypassed during compilation for Node.js-targeted bundles.

Relevant discussion: https://github.com/evanw/esbuild/issues/425

Similar experience here, extremely rapid turnaround time on my request for extended symlink support https://github.com/evanw/esbuild/issues/781

This project is a prototypical example of the power of open source. An incredible contribution to the JavaScript ecosystem.

I just serve my ESM modules directly from the filesystem. It works well, builds in literally 0 seconds.

You also don't need a transpiler like Babel if you drop support for IE11. Almost all ES6 features– such as classes, proxies, arrow functions, and async are supported by modern browsers and their older versions. As long as you avoid features like static, private, and ??=, you get instant compile times.

It's not 2015. You don't need a transpiler to write modern JavaScript. As of writing, the latest version of Safari is 14 and the latest version of Chrome is 88. These features fully work in Safari 13 and Chrome 70, which have less than 1% of marketshare.

Alas, many of us still need to support IE11. We alas get a lot of value out of TypeScript, which needs to be transpiled. <1 second with esbuild is pretty good!

Here's my suggestion if you want faster builds:

In development, use an index.html that loads it directly from the filesystem, using a single-pass annotation remover such as https://github.com/alangpierce/sucrase A single-pass transpiler doesn't build an AST, instead just looping over everything once, which will surely make it faster than esbuild.

In production, use a bundler like snowpack.

What you're saying there is: in development, use sucrase. In production, use sucrase.

And saying it quite a lot in this thread.

Sorry, I meant to say use snowpack.

You can annotate your code with JSDoc comments and get the full value of TypeScript while just writing JavaScript without any transpilation step[1]. It won’t help you with IE11 support though.

1: https://www.typescriptlang.org/docs/handbook/intro-to-js-ts....

That's a helpful feature but it's by no means a replacement for Typescript. You do not get the full value with just inline comments, in fact that usage is extremely limited.

1. How do you handle npm dependencies that don't expose ES modules?

2. How do you handle the performance impact of your browser doing multiple sequantial requests because it doesn't know your whole dependecy tree?

> 1. How do you handle npm dependencies that don't expose ES modules?

See my comment above.

> 2. How do you handle the performance impact of your browser doing multiple sequantial requests because it doesn't know your whole dependecy tree?

Put all your dependencies in the HTML file, so it fetches them all in parallel. So instead of just loading the root application file and forcing the browser to resolve dependencies:

    <script type="module" src="/scripts/app.js" defer></script>

Load the dependencies like this:

    <script type="module" src="/scripts/lib/gui.js" defer></script>
    <script type="module" src="/scripts/lib/guielements.js" defer></script>
    <script type="module" src="/scripts/lib/network.js" defer></script>
    <script type="module" src="/scripts/lib/input.js" defer></script>
    <script type="module" src="/scripts/lib/rendering.js" defer></script>
    <script type="module" src="/scripts/lib/sockets.js" defer></script>
    <script type="module" src="/scripts/lib/world.js" defer></script>
    <script type="module" src="/scripts/lib/ui/classupgrades.js" defer></script>
    <script type="module" src="/scripts/lib/ui/toasts.js" defer></script>
    <script type="module" src="/scripts/lib/netstore/minimap.js" defer></script>

    <script type="module" src="/scripts/data/configuration.js" defer></script>
    <script type="module" src="/scripts/app.js" defer></script>

You can see a demonstration of this technique on my production site, a complex JavaScript SPA that loads everything (including dynamic content) in less than one second: http://vnav.io

For a small performance boost, you can load smaller files first via <link rel=preload>.

Hi, I tried accessing your site, but seems like it's stuck in this weird reload loop. It never full loads.

I'm on Ubuntu 20.04 using the latest Firefox, if that helps.

EDIT: Seems to load fine in Chromium.

You probably have HTTPS Everywhere with EASE mode on.

Why wouldn't the site have HTTPS enabled?

An artistic or deeply philosophical choice by the author that, in order to view their work, you must permit yourself to be man-in-the-middled?

Mixed Content: HTTPS sites cannot connect to HTTP resources. The game needs to dynamically scale without expensive load balancers in front. Until I get dynamic DNS and HTTPS working, the site will not have HTTPS.

Thanks, the issue was Firefox's built-in "Enable HTTPS-Only Mode".

Thanks for the response, but IMO both of those solutions are strictly worse than having a build step that takes care of it.

Also, there are tools that only bundle npm deps, like Snowpack or Vite. I think those would be perfect for your use case.

So your solution to build systems has absolutely none of the benefits that build system do in fact bring, or the ecosystem of a language that you might consider terrible but actually has a decent amount of quality re-useable code. But this works for your very specific use-case?

This is terrible advice.

> Put all your dependencies in the HTML file, so it fetches them all in parallel.

> ...

> You can see a demonstration of this technique on my production site, a complex JavaScript SPA that loads everything (including dynamic content) in less than one second

But it doesn't work like this in practice! There is a limit to the number of concurrent connections a browser will make, Chrome for example is <=10 IIRC. You can see this in the waterfall:

`rendering.js`, the last `.js` in the queue, stalled for 300ms.

Additionally, each round trip is dependent on:

- The user's latency

- Your server's response time

So with each connection there is overhead. You would be much better served concatenating these files.

`util.js` took 510ms, of which 288ms was spent stalled (i.e. waiting for a connection) after which spent 220ms waiting (time-to-first-byte).

Furthermore, Lighthouse gives your page a performance score of 60, which isn't great. Key metrics:

- 1.7 seconds to paint (which is okay)

- 7.7 seconds to interactive (which is terrible)

Finally, why on earth are you redirecting to HTTP after serving an HTTPS response? This makes your page load even slower:

    if (window.location.protocol === "https:") {
        window.location.href = 'http:' + window.location.href.slice().slice(6);
    }

> But it doesn't work like this in practice! There is a limit to the number of concurrent connections a browser will make, Chrome for example is <=10 IIRC. You can see this in the waterfall:

In HTTP/2, which our site will switch to soon, the concurrent limit is 100. The world's changed, change with it, and throw away those bundlers.

Either way, you can compare to our direct competitors: https://diep.io https://arras.io

> Finally, why on earth are you redirecting to HTTP after serving an HTTPS response?

This isn't really related to the argument at hand. My app is currently on HTTP for reasons downstream.

I would say the primary motivators at this point have shifted from browser compatibility to syntaxes like TypeScript and JSX that will (likely) never be supported natively

> I would say the primary motivators at this point have shifted from browser compatibility to syntaxes like TypeScript and JSX that will (likely) never be supported natively

I suggest using a single-pass transpiler like Surcase[0] that doesn't translate to IE11-compatible syntax, and just loops through the string once, avoiding generating an AST– making it much faster– removing TypeScript annotations and desugaring JSX.

If you additionally need to support IE11, you can use a development build with Surcase and a production build with a bundler like Snowpack. C/C++ developers have been doing things like this for ages: compiling files as objects during development, and compiling them into one big binary for production.

[0]: https://github.com/alangpierce/sucrase

Surcase is often slower than ESBuild since it is single-threaded. The benchmark posted on its README forces ESBuild to run single-threaded, and is thus a bit misleading.

If you have any third party dependencies you still need something like snowpack though, right?

I don't have any third-party dependencies. We all agree the JS ecosystem sucks.

When I do, I use a script tag, and put it before my ESM modules:

    <script src="/scripts/data/contractor.js"></script>

Use them via global variables.

I generally avoid third-party JavaScript modules for performance and maintainability reasons. When I do use them, I'll frequently vendorize them (put them in my source tree) and make ESM modules for them.

So you’re the one writing everything from scratch even though the wheel has already been invented and perfected and tested over years by many developers.

Also your website is serving dozens of cascading files instead of a single minified one.

Feels like 2010 to me.

It's the opposite.

Bundlers aren't the "wheels" here. The real "wheels" here are ESM modules, which have been available in every modern browser for a while. Bundlers are just pre-wheel, prehistoric, stopgap technology from the 2010s that we should be moving away from.

And there's no problem serving multiple files in 2020 because we have HTTP2 multiplexing now. In fact, it's probably more efficient than using bundling, because you can cache much better. Minification is also virtually unnecessary with Brotli, and not minifying has the added benefit of making the debugging experience much better.

And bundlers haven't been "perfected". Not even close. Webpack is without a doubt the worst piece of technology in my stack right now, together with Babel. Those two are terrible by themselves, but they also manage to "infect" other things elsewhere in my stack: for example, ESLint and Jest need Webpack/Babel plugins to work.

Basically everything you wrote is wrong. Loading multiple files is never more efficient because compression works on a file at a time in that case, plus subdependencies start loading only after their parents finish downloading. This is CSS’ @import cascades all over again.

The wheels I’m talking about are the actual modules that exist on npm used in production by millions daily. Everyone here can write some code that does something common like, say, left-pad, and then they screw it in some obvious way that was fixed in 2014 by the third user of that library.

Sure I agree that having no build is nice, but user performance is not comparable and you’re limited to your own code.

You seem to be still holding a lot of assumptions from 2010.

In practice compression is not a big issue if you use something with a pre-defined dictionary specialized for HTML/CSS/JS, which is Brotli. The advantages of bundling are not exactly too large. Sure you might not be deduplicating some identifiers, but the lion's share of code in practice is normal javascript/CSS/html.

Also, in the long term, caching and the removal of code added by bundling more than makes up for any losses from compression.

Also, with Multiplexing there's no problem with download multiple files sequentially, because you don't need TCP handshakes and HTTPS negotiation, which were the reason for bundling in 2010. In fact, multiplexing multiple files might be faster in some cases, because you're able to parallelise the downloads and the parsing, rather than having to download a big bundle before parsing it.

There are of course caveats to this, but neither bundling or separate assets are silver bullets.

Right, but that is basically saying we should all go back to something similar to before npm/bower/etc.

I love using plain ESM and I prefer to pull in stuff that only relies on that. But when I need graphing on a single page I'm not going to try to vendor Vega and I'm not going to pull it in on all pages. When I need xls parsing I don't want to vendor xls.js and maintain the diff myself.

So I have to use something like snowpack to make it work with my ESM system and dynamic imports.

If you don't have to require heavy libraries for certain pages then that's great and I encourage to use pure ESM without build systems for that, but you should also recognize that not all use-cases are ready for it.

Currently I use snowpack to handle dependencies but still run all my own code unbundled and unminified.

For Vega, I would use the pre-built minified ES5 version: https://github.com/vega/vega#es5-support

Add a <script src=> tag.

Access it by object destructuring a global variable. You can access ES5 stuff from ES6, just let ES5 run first.

> Right, but that is basically saying we should all go back to something similar to before npm/bower/etc.

That position isn't without good reason. Refer, for example, to the linked post, explaining the engineering decisions behind esbuild. ("It's Golang, not JS" is only part of the answer to "Why is esbuild fast?" The implications seem to be lost on many of esbuild's users, given the nature of esbuild itself and the job it's supposed to do.)

> We all agree the JS ecosystem sucks.

I mean there's a lot of garbage out there as with any ecosystem but saying "we all agree the JS ecosystem sucks" reeks of immaturity.

It's true though. 99% of the JS ecosystem is a security nightmare (transient dependencies + npm).

I love that TypeScript and esbuild both have 0 dependencies. For the first time I can have a build pipeline without transient dependencies.

If you're using straight ESM then you don't need any build tool a lot of the time. If I need any 3rd party module it's either include it in global using an ES5 library, or use https://www.skypack.dev/ which has a ESM ready built module of most npm packages that you can import directly in your code.

> You also don't need a transpiler like Babel if you drop support for IE11

I really dislike this line of thought: Babel isn't just a polyfill tool. It's an incredible way to enhance your code. Further, the future of javascript changes isn't static and shouldn't be.

This works IF you write 100% vanilla javascript. In larger projects a vanilla javascript approach usually falls short. I cant see a large project being written without a types in 2021. So you still need a build step, no matter if you are using Typescript, flow or some other typed version.

There's also a substantial writeup on the architecture of esbuild in the repo [1]. It is definitely worth a read if you are interested in some of the inner workings of esbuild or are curious how various bundler features are implemented. High-quality, in-depth architectural docs for complex projects like this are exceptionally rare.

[1]: https://github.com/evanw/esbuild/blob/master/docs/architectu...

Slightly tangential, but I really love the minimal but effective site design here. It is minimal, responsive and easy to read. Is this a theme or just the author being very effective and throwing together a documentation site?

It is also all custom: https://github.com/esbuild/esbuild.github.io. YAML files that get converted to plain static HTML pages. It's designed to work well without any JavaScript. Even the animated graphs on the home page are done without JavaScript.

Why all that instead of just a normal static site generator? Doesn't seem easily expandable.

I can confirm that esbuild is very fast - easily bundling my code a hundred times faster than webpack or rollup. And the minified esbuild bundles are only around 10% larger than what is produced by the JS-based bundlers. Considering it saves 30 seconds per build, this is more than an acceptable tradeoff.

It's good that this tool is shaking up the JS development ecosystem. It was getting stagnant (babel) and overly complicated (webpack). We can probably expect more tooling to migrate to golang and rust. I think the esbuild author accomplished his goal.

Esbuild is awesome! I tried it on my latest project and the speed has blown me away, even though it doesn't replace the TypeScript compiler for my usecase.

Its two biggest shortcomings are:

1. It can't compile const enums (TypeScript feature)

2. It can't compile down to ES5 (with some exceptions)

So I still need to run TypeScript + esbuild side-by-side. TypeScript compiles to ES5 and esbuild bundles the ES5 files.

Anyway, it is a massive improvement over TypeScript + webpack.

We really need it to support top level await, and the author doesn't seem to be that interested in supporting it at the moment. Hopefully he will come around soon.

Interesting, what is your use case for that?

The use case is that it exists: It’s already been vetted and it’s part of the language. Not supporting it means you don’t support the entire language.

It's part of the language, but not so much thoroughly vetted. The Chrome/V8 implementation has bugs.

https://github.com/evanw/esbuild/issues/253#issuecomment-773...

I mean the feature itself has been vetted, i.e. it wouldn’t be in the language if enough people didn’t find it reasonably useful. TLA is a substantial feature implementation-wise so I expect TC39 to have thought “yeah that’s still worth it”

TLA was accepted in the specification before the module loading scenarios were fully baked. Probably best to use async IIFEs until the TLA spec bugs are sorted out and the fixes are subsequently implemented in the major browsers and released. Maybe a year from now.

The use case is not needing to wrap your code in an IIFE [1] when you would have a top-level await statement. It is useful in entry point of apps where you might need anything that uses async. It's also useful for having async modules. [2]

They're also very helpful for debugging. Using await something in the console is way more convenient than something.then().

[1] https://developer.mozilla.org/en-US/docs/Glossary/IIFE

[2] https://github.com/tc39/proposal-top-level-await/blob/HEAD/R...

I'm the author of esbuild. Rest assured that I am very interested in supporting it and am actively working on it. It's just unfortunately very complicated to correctly, especially without affecting any of the other existing features. There are some recent updates at the bottom of this thread: https://github.com/evanw/esbuild/issues/253.

I just want to say thanks for you work!

And good job on finding those top level await spec bugs.

Hey, thanks for getting back Evan. Thanks so much for all your hard work. I will be following closely.

Curious to how you set this up without going crazy (and CI working)? Just a line in package.json that runs tsc, then another one that runs esbuild on the output?

It's a pretty simple setup for now.

For tsc it's a single line in package.json (+tsconfig with outdir: "dist").

For esbuild I am running the following Node script (saved as .mjs):

    import esbuild from 'esbuild';

    let args = process.argv.slice(2);

    await esbuild.build({
        watch: args.includes('-w'),
        entryPoints: [
            'dist/entry1.js',
            'dist/entry2.js',
            'dist/entry3.js',
        ],
        bundle: true,
        target: 'es5',
        outdir: 'bundles/',
    });

I wish the Angular team put some work into integrating. They are instead doubling down on webpack which is frustrating. After they back stepped regarding the bazel build system I thought that all was lost. Then this beauty came out. Unfortunately it's not as interesting as the incremental compiler.

It's still early days for esbuild. If you follow the commit history there are still some weird bugs. I'd give it a few months to get these things ironed out (along with wider usage) and it'll be good to go.

That would make my day. I have a 64GB RAM PC because I work on 2 or 3 angular apps at a time and since my PC is the fastest I also use it for CI builds on occasion. It's a nightmare to wait 20 minutes for an optimized prod build. And having to upgrade everyone's RAM so they can contribute to the front end is horrible. We are past 10GB of RAM required to start the dev server.

10GB!! How big is this project!? Could you split it up somehow?

We actually just worked on splitting it up into libraries. The shared library might actually make the ending build larger due to it being potentially duplicated (I'm not certain about this until I test it). Unfortunately that has to go on hold temporarily. It is a large Healthcare application in the therapy space. The ending build is actually fine, but the tooling eats RAM like something else.

Little by little I take large third party libraries and rewrite them on my own to drop 5MB includes. I'm sure that is part of it. Devextreme and Syncfusion are libraries I would strongly recommend avoiding.

Syncfusion added 15MB to my bundle size :( I like their components but they aren't lightweight by any means.

Incidentally, a new version of the Bazel rules_nodejs, which somewhat confusingly has both browser and no JavaScript support, includes an esbuild rule.

As for Angular CLI which relies heavily on web pack, it has years of grinding tweaking bug fixing work getting all the right behaviors for all the cases. It's going to take a lot to re-create that in a non-breaking way using something different under the hood.

Vite 2.0 (a build tool like Webpack or Rollup, and powered by Esbuild) was just released earlier today. It's seriously awesome.

I first learned of Esbuild early 2020 when I complained on Twitter that JS build tools are awfully slow [1]. Most my (mainly Vue) projects use Webpack and starting the dev server takes typically anywhere between 5-50 seconds.

With Vite, cold devserver start in a mid-sized project on my 2015 MBP is ~2-5 seconds, but vast majority of the starts happen almost instantly as the deps are cached. Hot module reloading and most changes is also very fast and feel instantaneous (where in Webpack projects they would take anywhere between 1-20 seconds). This has HUGE affect in developing experience as you can just keep writing and almost never need to wait for any changes. (The one exception being TailWind/PostCSS builds that take few seconds every time the config or main import changes.)

I absolutely love Esbuild and Vite. Been using them daily for few months now, and I'm currently in the process of converting all my old vue-cli based projects to Vite using a project template I made that has TypeScript, Tailwind, and e2e tests w/ GitLab and GitHub CI configured [2]. If you work with React or Vue and Webpack and haven't yet tested out Vite or any of the other Esbuild based build tools, definitely do check them out!

[1]: https://twitter.com/uninen/status/1230673711910576130 [2]: https://github.com/Uninen/vite-ts-tailwind-starter

Three things need to be pointed out:

1. You can support IE11 without Babel (I got rid of it many years ago already): Typescript has transpilation, and you can add polyfills

2. As much as I like JSDoc-augmented vanilla JS, it doesn't catch nearly as much as real TS

3. Webapps are made of more than just scripts, loaders are even the big reason why Webpack got popular initially. Luckily, Webpack 5 can automatically recognize assets referenced in "new URL()" calls, so you don't need types for images or shaders anymore.

Esbuild is certainly fast but I could never figure out how to get it to preserve some symbols during minification (for when I want to expose some class methods or properties publicly to external consumers)

Open an issue if you haven't. He is very responsive and thoughtful considering he is the cofounder of Figma a company which is rapidly growing.

esbuild has keepNames options to preserve that

Indeed. The relevant documentation: https://esbuild.github.io/api/#keep-names

I wonder if window['exposedObject'] = {... would work.

I've really wanted to use Esbuild in my projects but I haven't figured out a great way to get the desired functionality I want:

I'd like to be able to transpile my code but keep the same directory structure in the output. For example, I have an index.js file that exports all my components and I'd like to just create a "build" folder output with the same exact file structure, but with all the files exported from index.js having been transpiled.

Does anyone know if that is supported?

Esbuild does have a JS transform API, so you could write a command line tool that walks the file system (Babel, ESLint, Prettier, etc. all do this so you have examples), transforms each file, and writes the code back out. Snowpack's build command also does this.

Hey, I was trying to do this same thing for a TypeScript project I was bootstrapping. Digging around the issue tracker, I recall the maintainer stating that this is currently not in the scope of esbuild, and that swc[0] might be better suited for this.

Personally, I wrote a crude JS script using the esbuild transform API + node-watch to compile .ts files and copy files/folders during development. This is all very recent, so I can't ensure that it's bulletproof quite yet.

[0] https://swc.rs/

I recently switched to using esbuild [0] and certainly is fast for minifying js and even css!

For my site I basically only do minifying for js, and bundling+minifying for css, and it all gets done in 0.2 seconds. Right now the longest time in my deploy process is actually just doing a git fetch (with depth=1).

[0]: https://twitter.com/thexpaw/status/1358124690347339778

Speaking of fast, that web page loaded instantaneously for me. Gives me confidence in the product, despite being technically unrelated.

Go has shared memory between threads while JavaScript has to serialize data between threads. Both Go and JavaScript have parallel garbage collectors, but Go's heap is shared between all threads while JavaScript has a separate heap per JavaScript thread. Untrue when using sharedArrayBuffer

> Every time you run your bundler, the JavaScript VM is seeing your bundler's code for the first time without any optimization hints.

Not necessarily so: https://v8.dev/blog/code-caching

For the uneducated like me: if I am writing a NodeJS application in Typescript, can I use this instead of the `tsc` Typescript compiler CLI and get faster compile times?

Yes for compiling, but without typechecking. You'll still need tsc for that.

Does esbuild work with font files that comes with css as well?

can esbuild used to do quick development builds with some dev server, like serve node pkg? or just for prod builds? if so, how the pipeline looks like?

Yep https://esbuild.github.io/api/#serve

TIL a new word: megamorphic

swc (https://github.com/swc-project/swc) is a rust equivalent to babel and has a bundler under development. Not sure how far along it is.

That would be an interesting comparison of performance and features.

SWC looks cool, but esbuild is more mature I think. The bugs in SWC's changelog make me pretty wary of using it in production just yet.

swc is more ambitious, it wants to re-implement all of tsc type checking, not only transpiling: https://github.com/swc-project/swc/issues/571

it's fast because it doesnt do the thing that everyone else does to be useful which is type checking

This is dishonest. In esbuild's benchmarks, webpack, rollup, and parcel do not do typechecking either.

I like the idea of having type checking moved out of the watch path, or at least be in a second window watching, so building is faster. But you get your type checking errors too.

You also have the editor finding issues for you, which is where I want that caught.

Yes this seems like an excellent performance boost to development. Parallelize type checking and compilation. We all have these many-core machines now, and so many tools that single thread the world unnecessarily.

You may want to check out this webpack plugin [1], though I'm not sure how much it could get you there.

[1] https://github.com/TypeStrong/fork-ts-checker-webpack-plugin...

Nope. This is wrong. It clearly mentions that its only a Javascript and CSS bundler and nothing much. It doesnt try to solve every problem in the JS world. None of the other bundlers do type checking by themselves. They are always through plugins.

Reply on news.ycombinator.com