Dum: An NPM scripts runner written in Rust | Hacker News Reader

Dum: An NPM scripts runner written in Rust

github.com

112 pointsvvoyer90 comments

I'm just dumbfounded that HN guys became so negative! Literally every comment is so negative :( Please guys, this guy did an awesome job, the painful slowness of npm -- I gotta endure it everyday, this guy is literally giving me a sigh of relief. I'm gonna use it over npm and even contribute if necessary.

If "written in Rust" is a clickbait, then most first page HN titles are clickbaity too otherwise why will you click on them? To me and most people, "Written in Rust" or even "Written in Go" reads like "it's fast" and that's definitely a feature!

I kind of disagree, given that I wrote a comment that perhaps questioned the project somewhat. A lot of comments are pretty positive, at least of the time of this writing. I started out with "cool project". Even if I use node every day like yourself the startup time for npm commands is so fast compared to other stuff I use like transpiling typescript to javascript and the worst offender of all: Babel.

The babel transpiling takes about 5 seconds compared to the 200ms of npm scripts startup time. Since I haven't felt this pain of npm startup times myself as a pain I must be able to state that without being interpreted as negative, don't you agree?

Of course, speed increases are positive over all but this requires me to install something just to gain 200ms from whenever I run an npm command which isn't that often. I am sure there's a lot of people out there who this benefits a lot but I haven't personally experienced this is my years of doing node development.

There have been performance-focused alternatives to NPM for some time: Yarn and PNPM. They offer other benefits as well, and both have become pretty popular (Yarn moreso I believe). There are performance-focused alternatives for a lot of things in the Node/JS ecosystem. TypeScript tsc: ESBuild. Babel: SWC. NVM: Volta. All popular to varying degrees, with people happy to install them for their performance gains.

Sure, npm run isn’t my biggest bottleneck. But it’s noticeable, and becomes annoying a couple of layers of indirection deep… which isn’t that uncommon, especially in monorepos. Shaving 200ms off might not sound like much, but it really depends how often you’re taking that 200ms hit.

It's a good direction, like swc which replaces babel, webpack and esbuild, and is also written in rust. I feel all these parts needs to be done in a language like rust, not just some parts. I wish the whole "npm install" thingy aka the package manager which is painfully slow, needs to be written in Rust next. That would be awesome!

My comment was praising this person, and I got downvoted for it.

They've contributed so much work, and I think it's really cool that they're always diving in to solve problems and take a crack at alternative solutions. Mad props.

HN is extremely JavaScript-fatigued. Feels like all JavaScript-related novelty has been getting downvoted lately unless it's coming from a small set of "trusted" JS authorities (Next.js maintainers, Kent Dodds, etc.)

I'm not sure why all the shade for this project.

NPM is painfully slow, everyone's harping on the 200ms, for larger projects that number is much higher.

I'll be using this and happy it exists.

Because for some people NPM is the hill they decided to die on :)

> NPM is painfully slow

Agree on the negativity (surprising and completely unwarranted), but a nitpick: npm install is slow (this doesn't replace that). NPM scripts are generally very fast, unless the implementation of your actual script is slow (in which case it will be similarly slow with Dum).

(the Dum GH has benchmarks showing perf. gains for scripts, but the % improvement is only significant because the script doesn't do anything - the gains would be static for real-world use-cases, making the % gain negligible)

All that said, there are other very valid reasons to use Dum - limited, and containing a lot of caveats, but very valid all the same.

> Instead of waiting 200ms for your npm client to start, it will start immediately.

That seems like a pretty big performance problem on Node's part.

Wouldn't it be better if Node solved this, than people having to use external package managers that may not be so well-integrated into the ecosystem? Thirdly would people even bother using an external package manager because of a 200ms delay? Is that considered acceptable at this point?

You presuppose NPM is well integrated with the ecosystem /snark

It would be better, but it might also be borderline impossible or at least extremely difficult. Since Node is open source, anyone is free to try to solve. Maybe it is trivial!

In the meantime this is a practical, acceptable solution.

> Thirdly would people even bother using an external package manager

What do you mean? This tool is not a package manager, it is just a script runner.

> It would be better, but it might also be borderline impossible or at least extremely difficult.

Out of curiosity, what makes you say that?

> What do you mean? This tool is not a package manager, it is just a script runner.

Bah, sorry. I had PNPM in mind when I wrote that (which I highly recommend). PNPM doesn't explicitly focus on startup time, but it does majorly improve performance as it caches packages.

node has a costly and intertwined init itself, where the startup cost is 50-100ms.

On top of that, `npm` similarly has a high startup cost adding to the original node cost.

Node apps are very slow to start to run due to synchronous `require` eager-loading all the module files, so there is some re-architecting necessary to get a performant startup time

yarn v1 is probably the best of the bunch with startup lag close to pure node (100-150ms)

yarn v2 and up can be extremely bad, 1s-3s with medium to large repos, especially with PnP enabled which AFAIK forces eager resolution.

pnpm similarly has had troubles starting up quickly. It used to be even slower as it passed the command to `npm` in a suboptimal way. Nowadays its about the same speed as `npm` as it uses `npm` for `run`.

https://github.com/nodejs/node/issues/35711

Node's working on it.

Interesting, I had seen previous work on this and thought they’d already squeezed the available perf juice out of it. Nice to see there’s still more room for improvement on this front. And especially exposing more to userland, I definitely have use cases for that!

It's not solely a node problem. The benchmarked task is starting up a node subprocess, that's what most of the 50ms in the Rust execution is. So npm is adding an additional 150ms + whatever the Rust code takes, beyond shelling out to start a node subprocess.

I assume when you say "Node" you mean NPM?

Just spent a few minutes checking NPM's startup code. For a total run of ~234ms, it spends:

~50ms loading a version-parsing lib, loading JSON metadata containing Node binary version and checking node binaries against some hard-coded compatibility data. Dum doesn't seem to do any such sanity checks, which may be fine if you manually ensure your environment is correctly setup.

~25ms initiating logging (& exception handling). This seems like it could be improved.

~40ms loading config & 2 external dependencies (`which` and `graceful-fs`) - not sure if the config is the reason for slowness. The dependencies bring functionality Dum doesn't seem to need as much as NPM does. I guess the load-perf of the dependencies could be improved, unless it's the config loading causing this.

~80ms checking for updates!!! Absolutely no need to do this on every run - this could be throttled somehow at the very least.

The remaining ~40ms is actually running the script

---

Update: After looking more closely, it appears the update checking code doesn't actually run every time - it's taking 80ms just to load that code (without doing the actual check - the check is cached)

Honest question, what use case have people run into with non scripts that a 300ms startup time is painful for? Mine have always been simple and run not very often, so it’s barely mattered to me, but I’m sure I’m missing some obvious uses that others have

Maybe for stuff that runs over and over again and needs every microsecond it can get. But if that's a requirement, nodejs seems like a pretty bad choice anyway.

In the past, tools were slow, so everyone was forced to use "watch" modes in their tools, so startup time didn't matter much.

Nowadays, we have esbuild and a whole suite of other tools that run in <10ms, so a slow startup time is dreadful.

Ah that’s fair. Most of the tooling I’m running is still honestly pretty slow (CRA being the main set, Jest being the other), and they both do the “watch” trick.

I've always used --watch just so my stuff would re-run without me having to manually do it...? That watch modes were to get around slow startup, and not a usability feature, is new information to me.

That's a question designed to confirm priors, depending on perspective it's not necessarily an honest one. It's better to think about changes like this in terms of what becomes possible with overheads removed ("what becomes more convenient to do as an npm script if startup is much faster?") or comparable examples from other areas ("what happened to the web when it no longer became necessary to wait 300ms+ for application logic to respond to a click?") etc.

Sure, that’s what I was trying to get at but worded badly. What am I missing that this would help with, and that NPM scripts are the right choice for them?

Git hooks as someone mentioned below are a good one.

It absolutely was an honest question, and nitpicking on my intent to imply I think this project is silly seems a little unnecessary: I don’t, and really do want to know what it could help with, as none of my use cases need it right now. I’d like to keep any eye out for when they would though.

For me, speedup enables a lot of much simpler ways to do the stuff that I already do.

Run tests and linter on-save. Have vim with LSP and other tooling, startup fast enough to close and re-open the entire editor, rather than background (<CTRL>-z) or alt-tab. Just do a <CTRL>-c<CTRL-p><enter> to reload any changes in a running dev server, rather than tooling that attempts to listen to io-events and reload the code magically.

There are loads of small things that become a lot simpler when kill&start is just as fast as "internal reload".

It adds up. 300ms is pretty good, but that's just starting up a node process, then you get a long process of parsing and executing the script itself.

Which initially is not too bad, but the bigger the codebase gets and the more often you run it, it adds up. 300ms per invocation is something that a lot of CI providers would love to save.

Relevant XKCD: https://xkcd.com/1205/

That doesn't apply here: all these scripts do is shell out to some other binary. Any performance cost of using node or other binary is going to be exactly the same. It's 200-300ms regardless of what the underlying script does.

git hooks

Especially when you have 6 git hooks, and they're all run one by one in "pre-commit", which is written in Python, another whole slow interpreted ecosystem. I do not miss my last job.

That’s a good one I hadn’t considered. Definitely going to keep this tool in mind if we ever need to run some hooks from npm scripts.

This one might help: https://github.com/evilmartians/lefthook

Anything that takes 300ms to start up is not suitable for anything but server applications (and even then...) or daemons. Nothing living in userspace should be that slow, IMO.

You could make this actually useful by making it read a .env file to set the environment, instead of what people always do, which is:

> "test": "NODE_ENV=test mocha --reporter spec"

(and doesn't work across platforms like windows)

Or added functionality like being able to pipe the output to multiple places (eg. terminal and log file).

That would offer perhaps a slightly more compelling use-case than saving 200ms; I don't care about that much, I honestly doubt anyone else does either.

The spawn time for an npm compared to the runtime for an npm task are orders of magnitude different, surely, for any non trivial task.

"NODE_ENV=test mocha --reporter spec" actually works on Windows when using recent versions of Yarn. They implemented a minimal shell parser for this and similarly basic features.

A lot of people use cross-env[0] for that in NPM. You basically turn this:

   "start": "NODE_ENV=production node ./index.js"

To...

   "start": "cross-env NODE_ENV=PRODUCTION node ./index.js"

I'm glad Yarn supports this natively, though. It seems silly to rely on an external package for something so basic (especially with the security vulnerabilities of the ecosystem as a whole).

[0]: https://www.npmjs.com/package/cross-env

> NOTICE: cross-env still works well, but is in maintenance mode.

Which is why I recently kicked all the cross-env out of my nodejs projects and replaced it with Ruby's dotenv[0] (in projects that already used ruby alongside) and zenv[1], in Rust for projects that don't.

Basically

    "start": "zenv --file .env.production -- node ./index.js"

[0] https://github.com/bkeepers/dotenv [1] https://lib.rs/crates/zenv (readenv is an alternative, but that doesn't allow overriding the .env file)

My first thought seeing this was “I wonder if they know about bun”, was happy to see it’s an inspiration! And just in general happy to see more performance-focused tooling projects in the ecosystem. Good stuff.

Rust based tools are so cool. I have used various cli tools and they are just awesome. One example is fnm which is fast and makes it easy to switch node js versions.

Seems cool, but I have never felt that the 200ms wait is any way an issue for me. Also just a funny note but Dum means stupid in Swedish. :D

It also seems to mean stupid in AMOGUS.

https://old.reddit.com/r/AmongUs/comments/iyg093/is_it_on_pu...

Considering all the "--watch" and "watch" options and scripts that come with many default tools (jest, react-boilerplate etc), it is considered a problem to many.

For me it is a serious nag. And I come from Ruby, where, with Rails, startup time is abysmal. So I am used to something.

Considering all the "--watch" and "watch" options and scripts that come with many default tools (jest, react-boilerplate etc), it is considered a problem to many.

Don't people use the watch options so they don't need to interact with the app to have it rerun things? I don't think speeding up how long scripts take to start will change that.

In my case, it hooks into my editor or IDE. E.g. I don't need to have a "--watch" feature in my test-suite, because my Editor/IDE can fire the tests just fine.

Same for (re-)starting webservices and such. And when not available or applicable in my editor, GNUtools, bash and unix offer plenty of tools to easily do this for me. E.g. https://linux.die.net/man/1/inotifywait to fire a command (e.g. restart a service) on changing files.

I presumed these `--watch` where there to solve the slow-booting issues only (like in Rails) but maybe they are there because someone didn't know their OS and DE could do it for them already and decided to build it into the suite or tool instead of using existing tooling?

It also means stupid in English :)

Maybe if you pronounce it but isn't it spelled with a b? Dumb, not dum?

"You dum-dum" is a common slang.

TIL. Thanks!

You should look up what "git" means in English :)

Glad to never have really used scripts in package.json, just bash scripts directly calling `./node_modules/.bin/whatever`

i've been loving egoist/dum as a complement to antfu/ni, but god forbid i ever remember how to update a cargo package. egoist if you're here, have you considered publishing binaries to npm?

Another project by Egoist.

This is one of those people who just constantly pump out so much open source code. I remember looking through their profile when I was using EME years ago (2017?), wishing I had the time to do half of what they were tackling.

Slightly off topic: So many projects and posts boast “written in Rust”. I haven’t been on HN that long—was there a point in time where “written in C++” was a similar “selling point”?

From a quick glance, it looks like this could have been implemented in any native language. I like Rust as much as the next guy, but using it to get clicks when Rust specifically over another native language doesn’t make a difference feels a bit clickbaity.

Personally I rather like that convention (and similarly for Go). Often I will follow links that specify one of these two languages since I primarily work in Go and Rust, and it's useful to add them to my mental map (or favourites folder) of what is available such that if I need something similar I can find it again.

Libraries or tools written in C++ interest me less personally, because they're more work for me to integrate into whatever I'm typically doing, but I can see that someone working in C++ might like such things to be labelled the same way for the same end.

I've seen myriad "written in C/Python/Java/JavaScript/TypeScript/Lua" posts over the years. You're not imagining things, though. The world waited 30 years (from the 1985 when C++ appeared to 2015 when Rust 1.0 was released) for a new, viable language in the systems space (read: without garbage collection). That's a really long time, and the excitement around the language is well deserved.

> for a new, viable language in the systems space (read: without garbage collection).

*Angry Oberon noises*

Stuart Smalley, as Oberon, looking into the mirror giving his daily affirmation: "I'm viable enough and, doggone it, people like me."

    it looks like this could have been implemented in any native language

I use this 30 line bash script, with a dependency on jq. It works for me, and is pretty much instant.

https://pastebin.com/raw/cnDk9zs4

Hey I like this. Can you make a powershell one?!

> was there a point in time where “written in C++” was a similar “selling point”?

> using it to get clicks when Rust specifically over another native language doesn’t make a difference feels a bit clickbaity.

- every single article mentioning rust in any way has this complaint in the comments

- I think that much of the reason people mention it is not a cynical ploy for clicks (though that exists for sure), but denotes that the program is a modern program that is likely to work in a certain way? It's hard to specify what that way is, I should try and write something about it, but I think people who complain about this are missing that it's kind of a cultural signifier more than a literal statement about the language it's written in.

For me "written in Rust" signals three things:

* It is probably fast.

* It is almost certainly very easy to install.

* If anything is missing or broken I can probably fix it myself.

>I haven’t been on HN that long—was there a point in time where “written in C++” was a similar “selling point”?

There are a few projects that tout being written in C++, but it has never been a regular "selling point" in HN, because HN was a thing almost 2 decades after C++ was the "new hotness".

So, we've seen it for Node, Rust, and Go more.

>I like Rust as much as the next guy, but using it to get clicks when Rust specifically over another native language doesn’t make a difference feels a bit clickbaity.

You think this person spend weeks writing this for "clicks"? Unless the author posted it, they might not even know it was posted on HN (it's often the case with posted projects: someone else finds them and posts them).

They had a pet project in mind, they liked Rust and wanted to use it, they did.

> it looks like this could have been implemented in any native language.

from the readme: "How: This is written in Rust! (Or any compile-to-native language)."

There was probably a time when C++ had the kind of enthusiasm that Rust does now, but as C++ was released in ~1984, I don't know that you can quite compare the two.

It's not a ploy for clicks. Programmers spend a lot of time working with programming languages and so the choice of language is an interesting thing to mention, no matter what it might be.

https://hn.algolia.com/?query=written+in+c%2B%2B

https://hn.algolia.com/?query=written+in+python

https://hn.algolia.com/?query=written+in+go

https://hn.algolia.com/?query=written+in+rust

EDIT: There's one exception. "Written in bash" is and will always be clickbait, fight me

Wrong! "Written in COBOL" is the true clickbait.

"Written is bash" always means it's either unimpressive or absolutely horrifying.

It's done for other languages, as others here have mentioned, but it happens more for Rust lately. A project being advertised as being written in JS or Java or C# does have a signal about how easy it is for the reader to contribute (if they know the language), to integrate it with other projects they use (network effects) or what their operation might be (can I build a single binary and deploy everywhere? what is the observability and debugging tooling like?).

Rust answers some of those implicit questions in a way that is acceptable to a lot of us.

Last I heard was from https://www.scylladb.com/ Selling point being C++ version of cassandra

> was there a point in time where “written in C++” was a similar “selling point”?

Notepad++ maybe?

> Slightly off topic: So many projects and posts boast “written in Rust”. I haven’t been on HN that long—was there a point in time where “written in C++” was a similar “selling point”?

"Written in Rust" is the new "Sent from my iPhone".

Written In Rust is great because Rust is one of those Fun And Shiny languages that people want to write in just to write in it, like Elixir or Node (topical!), except it's not going to pull an Electron and involve a giant VM and a bundle system that includes a user-mode Xbox controller driver by default, but rather is faster in its first draft than you could possibly squeeze out of one of those other languages. Rust captures the 'systems they don't teach you about in college' vibe without products written in it slightly decreasing the usability of the developer ecosystem.

There was a time where anything with JavaScript in the title automatically got into the front page. And now it's Rust. Rust is now in it's golden age. The question is if all the enterprise "Java" will switch over from JS to Rust, and demand that Rust change it's syntax and methodologies so that the "JS" developers can understand it.

you don't need NPM to run bash scripts, or js scripts

it doesn't matter that you have the thing re-written in Rust, because there is no reason for such thing to exist in the first place.

we're calling out 'npm run dev' now???

I've seen an occasional `npm run bash ...` and `npm run docker ...`. I presumed that was to have tooling like .env hook into it and setup the env. But it might be stupid copy-paste errors, as well.

Isn't the point of `npm run`/`npx` to run stuff in the local `node_modules` folder?

It's much more convenient to type `npx something` than `./node_modules/something/bin/something`, for example. Especially so when folder names are not obvious.

Just put your scripts in bash functions and then all of them in a scripts file, with enhanced $PATH at the top, or use make. In fact most of the more sophisticated projects use this approach.

package.json is such a sloppy and arbitrary place for your scripts. I don't know how the creator of node looks at the mountain of things that are all in package.json and thinks that's a fine design. You are a package manager, stick to it.

Reply on news.ycombinator.com