If you're applying idiomatic Nix to modern JS, I wouldn't be surprised - modern JS tends to involve installing thousands of packages by just combining them into a directory, but Nix doesn't want you to edit existing directories. So your dependency graph turns into a build-dependency graph, with each JS package requiring a full build of everything it depends on, and the Nix build system is presumably not optimized for fast turnaround times on five-line JS modules that don't even have a compilation step.

(Personally, I wouldn't try to apply idiomatic Nix to modern JS - I'd apply it to the major components like "my web server" and "my database library" and have "all the JS I depend on" as one big Nix package. That's not really a claim about Nix, I wouldn't try to apply idiomatic, say, Debian packaging to modern JS either. In both cases I'd still get about 90% of the benefit of using Nix/Debian as a delivery mechanism.)

The other thing that could be slow is if you start compiling all your dependencies, including gcc and node.js, from scratch. While there's some security benefit in doing so, the reality is that just about nobody actually does that. You'd want to set things up to use the precompiled packages, or at least set up your own cache server and have compiled binaries you trust but only do it once.