Hacker News Reader

Comment by lhnz | Hacker News Reader

  > The transition from CommonJS to ES modules has been slow and full of terrors. 
  > Bun supports both module systems, all the time. No need to worry about file
  > extensions, .js vs .cjs vs .mjs, or including "type": "module" in your 
  > package.json.
  >
  > You can even use import and require(), in the same file. It just works.

This is the highlight for me. The Node.js ecosystem is more-or-less completely broken otherwise. This might save it.

(I think that the most impressive thing about Bun isn't its performance but the pragmatic, developer-friendly choices that have been consistently made by Jarred.)

View on HN

This problem is very old at this point. I'm doubtful that Bun has "solved it" any more than other tools. They've just selected a different set of trade-offs that will result in a different set of packages not working. Is there a reason to think they've cracked some code that other tools were unable to?

The compiler is the bundler is the runtime. The reason it’s such a problem on Node is because the tools are separate and often conceptually at odds.

No, it's a problem because the formats are fundamentally incompatible and it's impossible to infer the author's intent. A package with no imports/exports or requires could be either ESM or CJS and no heuristic can solve for that reality.

How common are packages with no imports, exports, requires, or "module.exports="?

I imagine that's quite rare, because such a package would only be imported or required for its side effects; and because that package cannot import or require any others, could you safely just assume that if it was `require()`ed use CJS, and if it was `import`ed, use ESM?

Rare things still happen a lot at the scale of npm. And due to transitive dependencies it winds up affecting a lot of actual users. And this is just one example of an incompatibility, there are many more. Bun didn't discover a silver bullet here that dozens of other tools in the past missed on.

Do you have an empirical basis for that? I'm curious because on one hand, Bun is advertising they are solving a major challenge for library authors, really betting quite heavily on that being persuasive for adoption.

The risk for Bun's adoption is that they're wrong, of course.

The risk for Node's usage is that they're right, and library authors begin urging users to use Bun because it's 100x easier than making a useful library with Node. (This will happen very slowly, but things that happen very slowly can start happening all at once very quickly.)

> And this is just one example of an incompatibility, there are many more

Such as? If you could provide 5-10 of your many examples and why you think that Bun's approach doesn't solve them it would be very helpful.

Yeah, I don't believe GP. Their assertions that Bun hasn't solved this (or that the heuristic used is inadequate) are basically based on speculation without any evidence.

Jarred even came here and agree with them. It's nothing controversial. Just nuance. No need to flame. DYOR.

Assuming it has no imports or requires, and no references to meta, it’s a bare JavaScript file, and whether you treat it as esm or cjs shouldn’t matter, no?

JS has two top level grammars, one defaults to loose mode and the other defaults to strict mode, among other nuances.

Possibly the most devious nuance is whether the spec's appendix B applies, which affects whether html comment syntax is valid (yes, this is a thing). The html comment token can therefore be parsed either as a comment or as a series of operators depending on the grammar being used.

Effectively, this means it's possible to craft a program that does different things in CJS vs ESM mode.

Maybe the trade-offs are better?

https://twitter.com/threepointone/status/1698271991927648643

In Bun, you can use import and require in the same file. There's three steps for what makes this work:

1) We patched JavaScriptCore to load ES Modules synchronously when they don't use top-level await. The main tradeoff here is that requiring an ES Module which uses top-level await is unsupported (Bun throws an exception when you try to do this)

2) To support require() inside ES Modules, we add `import.meta.require` and have a transpiler integration to convert calls to module.require() or require() into import.meta.require(). This either loads the file as an ES Module or as a CommonJS module, depending on what the transpiler says it is

3) We rely on certain heuristics to decide whether a script is an ES Module or a CommonJS module. If they use certain sloppy mode or CommonJS-only features, it becomes CommonJS. If they use ESM features or it's somewhat ambiguous, it becomes an ES Module.

I’ve often wondered why node doesn’t do 1 in this list. It would solve 95% of the problems, I’m sure.

For 1), what about requiring an ES module that doesn’t itself use top-level await, but imports another ES module that uses top-level await?

That throws since we cannot make it synchronous. It works by reading the internal state of the Promise and getting the value out of it if the Promise is fulfilled.

While I appreciate the convenience that will result from this excellent engineering, I'm concerned that this could result in a lot of mixed module files that now can only work with Bun.

Is there an option to require each file to use either CJS or ESM features, but not both? If not, may I suggest adding that option?

Seems like a case for ESLint and I agree with you.

Mixed require and import (the latter in both its static and dynamic variants) is already possible with Webpack and Vite.

I hate it and would complain about any code introducing this in any kind of review.

It's great for integration, full stop.

This is an inconvenience where tradeoffs really show and demand to study the behavioral/semantic differences.

Funnily enough, I still find Bun interesting.

Regarding import.meta., I already love Vite's dynamic import transpilation features. They aim to work almost transparently and often do (I am not even talking about the glob feature here).

That being said, I'm unsure if Bun's way of bridging the module ecosystem gap using import.meta will be as useful.

If you ask me, frontend code should ban Node/CJS/require entirely or transpile it into static or dynamic ESM imports; respectively.

Any case where this is not possible unambiguously is a case for human intervention and code rewrite/conversion.

>If you ask me, frontend code should ban Node/CJS/require entirely or transpile it into static or dynamic ESM imports; respectively.

>Any case where this is not possible unambiguously is a case for human intervention and code rewrite/conversion.

Good god, yes

Why go to such lengths to support a legacy module system?

All three patches mentioned will lead to modules that work only in bun and not work in node/deno/browsers, further fragmentating the ecosystem.

It's a legacy module system with a lot of code written in it. Without mixed support, you can't rewrite code incrementally to use the ES modules. As a consequence, many projects still stay with CommonJS, since the migrating everythin in one go is a big hurdle.

In a sense, an opinionated collection of tools _is_ a solution.

Yeah, definitely recommend you try it before discounting... I too have been super jaded with the current Node import ecosystem (esp. as an author), but Bun has basically become something of a silver bullet, with some small exceptions.

If I want, I can literally run a TS file directly with mixed import/require (not that you would ever) directly, without a package.json, tsconfig, dependencies, etc.

In my experience, it's basically Node, minus all the setup/config/build headaches.

This is what I pushed Node to support when they proposed their split syntax and I'm very happy to see Bun adopt this. It is the "correct" way as far as I'm concerned.

Node's split ecosystem has really hurt the entire language and runtime.

Is there a quick summary of the problem somewhere? I haven't worked on frontend stuff in nearly a decade.

From a package maintainer standpoint, trying to support both ESM and CJS in the same published package is a _nightmare_.

I wrote an extensive post a few weeks ago detailing all the pain that I've dealt with and problems I've run into this year trying to modernize the Redux packages to fully support ESM and CJS:

- https://blog.isquaredsoftware.com/2023/08/esm-modernization-...

I've gotten a lot of feedback from other maintainers saying they've run into similar issues.

Also had a couple podcast discussions following up on that topic:

- https://syntax.fm/show/661/supper-club-shipping-esm-with-mar...

- https://changelog.com/jsparty/290

That blog post is exactly what I was looking for, thanks!

Oh my god, thank you for this. I expected a bad situation, but I underestimated the badness by several orders of magnitude. Makes me want to never touch JS again.

Still not clear to me what ESM exactly fixes. CJS works pretty well for me. The amount of pain ESM gives in unit tests is enough to ignore ESM. Maybe Bun will make a difference here.

By being declarative instead of imperative:

- The dependencies of a module can be scraped without running the code

- The subset of declarations used in an import can be known

This allows modules to be loaded in parallel, and also allows for things like tree-shaking. The biggest benefits are for web browsers, but all around it's a more constrained way to express dependencies, which allows the bundler/runtime to take more liberties.

CJS does not work in browsers. Bundlers fake it well enough you might not feel that pain regularly, but ESM was designed for use in browsers and CJS wasn't.

You could get those benefits through special syntax that chose to not deviate from commonjs semantics.

We are in this mess because tc39 explicitly chose to not give a damn about nodejs ecosystem.

None of the secondary benefits of ESM to commonjs semantics incompatibility (like recursive imports, top level awaits etc) are IMHO worth the ecosystem disruption.

Commonjs is synchronous. If you used it in the browser, it would block the main thread each time you required a module.

That's bad.

ESM has dual syntax for static and dynamic imports too.

What I am saying is that

import Foo from "foo"

could be sematically equivalent to

const Foo = require("foo")

and

const Foo = await import("foo")

could be semantically equivalent to

const Foo = await Promise.resolve(() => require("foo"))

if tc39 designed ESM import spec with CJS compat in mind.

It is fully possible to introduce additional syntax while not breaking the ecosystem.

CJS makes an assumption that all of the main body of a module is synchronous including its imports and that the main body of all of its imports also ran synchronously prior to the main body of that module. A lot of CJS modules are built around side-effects during module loading, built on assumptions from the way that Node especially synchronously loads everything, that break in any attempt to make the process asynchronous. It's not just a matter of syntax sugar, it's a trouble of bad assumptions and presumptions in the CJS format itself that browsers have no way to paper over (but bundlers can fake by putting everything in the same file). Loading a new file is always an asynchronous operation in a browser. CJS has never supported that. Those problems were known at the time when Node picked CJS. (AMD was the competing format that worked in browsers. AMD had its own problems and was a pain to work in without tools like Typescript, but it was compatible with browsers. CJS never was. ESM imports are backward compatible with AMD imports but because AMD is mostly a dead format in 2023 no browser today actually implements the shims to support that, but they were on the table for a long time and can still be ponyfilled if anyone is crazy enough to have an AMD codebase in 2023 that didn't switch to a CJS-based hodge podge in Node at some point in the last decade and that they can't all-at-once migrate to ESM. Which in my experience mostly just describes what's left of Esri's over-reliance on Dojo's AMD loader for ArcGIS JS and just about nothing else in the wild in 2023.) This mess is all Node's fault and it should have never picked CJS in the first place, that was always the losing horse.

(Hypothetically there might have been a place and time in CJS history to force CJS require() to always return a promise and module.exports to always take a function that returns a promise and that hypothetical version might have been compatible enough to import directly from ESM. Nobody actually wanted that hypothetical "ACJS" so it never existed. ACJS could have at least met browsers half-way and might have made the transition less overall painful.)

Nobody is advocating for synchronously importable modules in browsers. Also, people had already written support utilities like require1k [1] to support cjs in browsers. We don't necessarily need a full bundler - we only need a module analysis step between parse and execute which a native js engine is quite well positioned to facilitate.

Having said that, the fact remains that vast majority of pain points that people who actually need to maintain isomorphic libraries face today have nothing to do with synchronous vs asynchronous nature of cjs and esm modules.

It is the myriad pointless nuances like default export, namespace imports etc. that are sources of biggest headaches in day to day work. In cjs there is a simple model that a module exports an object and while importing you import that object. Instead now we have a situation where we need to deal with:

    export default { 
        foo() {...} 
    }

is not same as:

    export function foo() { ... }

and

    import Foo from "foo"

is something different from

    import * as Foo from "foo"

Plus the additional complexities introduced by import bindings being live etc. are just annoyances that one has to deal with over and over again every time module interop is involved.

[1] https://github.com/Stuk/require1k

Jesus, I thought python packaging sucked, but wow.

I maintain a lot of packages consumed by thousands of engineers internally at my job.

We don’t have super rigid standards in many ways, however that experience has lead me to wonder why so many engineers are struggling with dual support or cjs and ESM.

That said, none of those packages are redux level of downloads, and imagine you hit more edge cases than I do in that

Edit: originally stated hundreds. Turns out we’ve grown enough that I can say thousands now

I'm having trouble understanding why ESM was needed in the first place. I've always found the require system to be very elegant. What benefits does the module syntax provide?

I forked and customized an old Javascript templating engine and have published it as npm packages. Should I spend effort migrating to the new way?

Just my 2 cents/anecdata in response to your tweet about what library authors have to put up with, hoping it helps some of the doomers replying:

> Build artifact formats (ESM, CJS, UMD)

I don't know anyone that ever used AMD/UMD. CJS is loadable in ESM, and tsup makes the two of them trivial.

> > Matrixed with: dev/prod/NODE_ENV builds

Not every library has this issue.

> Bundled or individual .js per source

Not a new issue, this has always been a toss up (I prefer .js per source)

> exports setup

Yup this is a pain, especially because I prefer multiple .js files, and I wish tsup would help here.

> Webpack 4 limits

I'm not sure what you mean by webpack not understanding optional chaining syntax, we use webpack 4 and use it all the time. Might be a benefit of using typescript.

And tsup helps with the .js requirement issue.

> TS moduleResolution options

Again it looks like tsup solves this.

> User environments

When is that ever not an issue? Very fair for maintainers to say "nope, sorry, that's too esoteric an environment"

> TS typedef output (bundled? individual? .d.ts, or .d.mts?)

Should match your .js file output 1:1 (.cjs/.mjs et al)

---

All that to say, if you use typescript and don't do anything fancy, you've got a pretty compatible library with just `tsc`

And if you're looking to do something cross compatible, tsup will help a lot.

> From a package maintainer standpoint

Like why is this the standpoint that trumps everything else (eg stability of the ecosystem)? Have a tool to solve the problem and be done with it. It's not like with esm you can publish packages free of tooling anyway.

I never said it "trumps everything else".

I'm saying that I've spent much of this year dealing with all the pain around this, and linked an article on my own experiences, and that other package maintainers agree that this matches their experiences.

If you can build a tool that magically solves all these problems, great! Please let me know when that's available :)

(FWIW Bun looks like a genuine improvement on the _consuming_ side of things, but that doesn't help on the _publishing_ side of things.)

Every few years I end up having to write a bunch of JavaScript-- and recently it was time again for me to dig in. The `import` vs `require` thing caused me HOURS of pain. The modern JS ecosystem is truly painful if you're not using a react app or some off-the-shelf solution-- and I am not, I'm writing a small library. I honestly can't even fathom how difficult and maddening it has to be for folks just getting started.

The fact Bun solves that problem is more than enough of a reason for me to give it a try, I'm honestly fucking stoked to try it out, and I never thought I'd have those feelings for yet another JS runtime or build tool. Big thanks to the Bun folks, I'm looking forward to gaining some sanity in my day.

I've been creating module projects as default for nearly a year now. It's been largely fine for the past 6 months since Typescript 5.0 released with the "bundler" resolution strategy.

Very rarely run into issues but have gotten around with pnpm patch and meta modifications in the past until projects have fixed issues..

My take is 2023 is the year of the module. The water is fine.

I don't usually comment on downvotes and I have just under 5k karma so it's uhh, just no thing to me typically..

It's also against the guidelines, but again 5k karma...

That said, kinda curious now downvoted into negative for sharing my, extensive, experience creating module type projects.. Hrrmmm.

So almost 10 years after modules were added to the spec, a few years after they've been available in most browsers, the past 6 months have been largely fine to use them for _new_ projects.

That doesn't paint a great picture.

When I say fine I mean I've had no complaints with any aspect of my workflow since TypeScript released the bundler resolution strategy, which mostly just means you no longer need to specify extensions when importing..

I don't know about the past 10 years as I haven't tried to use modules for the past 10 years. I've been using them for the past year on new AND existing projects with Vite+React, Vite+Storybook, SWC+Fastify, Rollup and library projects, and etc.

Unless somebody has to work in the past I'm sure what relevance that has on using modules in 2023.

agree. i ought to be a javascript expert by now but nothing makes me feel dumber than navigating all the type:module and .js/cjs/mjs/ts stuff. for a production app of course you should be clear about what you are writing. but for prototyping Bun's permissiveness lets me write code first and worry about the rest later.

> for a production app of course you should be clear about what you are writing

Agree in sentiment, but I don't think this even applies to the module/import mess. My experience has been that the difficulty is in using different packages that made different choices, and making them interop with your own code. I have still yet to make top-level await work in a Typescript codebase where I need to import nontrivial packages. I don't think you should ever need to worry about the specific way your modules are imported, not in prototyping or in production.

I've been writing JavaScript for a long... long... time and I've kinda zoned out the last few years with regards to ESM vs CSJ

I only know that I perfer writing import over require as those semantics and syntax make more sense to me.

For the benefit of the future reader, I’m guessing you meant “CJS”, short for CommonJS.

From my experience, the feeling that "Node.js ecosystem is more-or-less completely broken" comes only when you're trying to create package supporting both CommonJS and ESM (see https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3...). Just rip the ban-aid and convert all your code to ESM and you'll be much happier (otherwise you experience something like https://blog.isquaredsoftware.com/2023/08/esm-modernization-...).

"just convert all your code to..."

I remember hearing that once from the Python devs...

I don't use Python, but AFAIK the differences between Python 2 and 3 and the effort needed to upgrade are higher then ESM vs CommonJS. But that's not relevant anyway; I'm not saying the upgrading to ESM-only is easy, I'm saying in the end, it's easier than trying to support both module formats.

It's anecdotal but I switched to ESM in Node a couple of years ago. Never experienced any problem.

My stack is usually Fastify with official plugins and a PG client.

yes, definitely and completely broken. require and import differentiation the most retarded changes, broke my build. spent sometime to understand the substantial difference, turned out it's nothing too substantial.

Seems like a bad idea to support both. Encourages bad behavior and half-ass code. I've had no problem moving codebases to es6 without a ton of effort.

Maybe at most, hide the capability behind an execution flag.

The problem isn't your code. It's the dependencies of your dependencies.

You may not be aware that node created a situation where ESM maximizes compatibility when consuming libraries and CJS maximizes compatibility when producing libraries. Its a circular incentive paradigm. I suspect you are consuming libraries using "import" and everything "just works". Well, under the hood you are likely importing many libraries that use CJS "module.exports" because that is what libraries use to ensure their library can be used with both import/ESM AND require/CJS.

The only solution to this problem is to upgrade require work for importing ESM. Once that is implemented ESM will become the natural choice for those publishing libraries. Until that happens library authors are going to continue to use CJS so their libraries are available to everyone.

The last version of Node that only supported CJS libraries has already long faded out of LTS. In my opinion, more libraries should just rip the band-aid off and entirely stop publishing CJS. Sure that will upset some downstream dependencies, but semver major tag it, and many of the downstream dependencies need the kick in the pants to move to ESM anyway.

Node.js v12 doesn't have proper ESM support and went out of LTS < 6 months ago. Have some patience!

Node 12 went out of LTS 16 months ago. 4 months ago was Node 14

Also, Node 12 supported ESM somewhat fine, you just had to use the --experimental-modules flag for Node. If you still have to support Node 12 in 2023 for some odd reason, you probably shouldn't have a problem enabling an experimental flag to support more modern libraries.

Node 10 was truly the last LTS that had zero ESM support, and that support ended 2 and a half years ago.

Reply on news.ycombinator.com