Building Node.js applications without dependencies

I manage a 4year old 'vanilla' typescript app at work. It doesnt't really do anything 'wild', looking at it you would think 'that could be built with anything/nothing'. But if you check node-modules, it contains ~1500 npm packages!

Your second thought might be 'oh, its because they let themselves use a huge amount of random gizmos!'.

Nope.. the actual root dependencies are relatively small in number, maybe 30-40 core dependencies.

The problem is, that the prevailing attitude in the ecosystem is to 'just use dependencies', for anything, like lpad. As 'I know, we'll just use the creditcard my daddy gave me'.

As long as your daddy is rich, you won't see a problem.

If your daddy isnt rich, you will notice that every 10 days as you run npm-build, it will proclaim 'hooray, there are now again 10-15 high-medium security in your 1500 dependencies'.

Combine this need to npm update continuously, with frontends happy disregard for typechecking and backwards compatibility, and you have a recipe for the plague,which we are currently living :-(

I used to not have issues with Java or .net, but with npm it’s very painful; I need to pin everything as updates, even minor ones, break whole slews of things. And that’s with dependencies of dependencies of dependencies. If you are not a consulting outfit (I suspect consultancy outfits love the js ecosystem; every day something breaks so that’s another few $150/hr hours), but have a company with 100+ applications installed that actively make or help you make your bottom line, you want them to run in peace for years or decades. If no new features or security issues arrive, you don’t want to spend $0.01 on them as what’s the point of that?

And that’s where this new fangled crap falls down: we have Django and .net apps running that are 10 years old, php and java ones 15-20 years old that all survived without much or any work. We have apps in js/typescript from 1 year ago that have bucketloads of deprecated, no longer supported, mandatory update warnings. It’s a pita so we are looking to move back to php or/and Django.

I can see that in a startup with a dedicated team, this would be no issue; you are on top of all and working daily.

Note as well that for some products (the more regulated of our products; financial/health/gov) have strict dependencies we all have to vet; in our experience of the past 4 years, it’s not a hallucination that npms change so often without any useful new features or fixes; just some useless dep updates that sometimes break things and weren’t needed, at all (you do not always need the latest version of everything ya know); I suspect (but maybe someone here knows) is that npms often update because it gives the authors more kudos somehow (as in; if posting many videos on TikTok ups your engagement; does that work for npm updates and new useless npms too?).

The thing about dependencies (regardless of ecosystem) is that it’s easy for them to spiral out of control, making it difficult to have a solid grip on all the things going on under the hood. Over time this can be a real problem because it makes tracking down breakages and fixing them that much more difficult, especially when dealing with deep fractals of dependencies.

In short, they’re fine in moderation but each added is another liability. As such, it can be beneficial to take a judicious approach and only add them when the value they bring outweighs that liability.

Because with big enough project you have 2 ways:

You update dependencies as a full time job and it bites you because something breaks big way. But most likely you have some help to fix it and as you are on top you might need to put work to fix it but yeah still full time job.

You neglect updating dependencies and it bites you and you don’t even expect it just because you have to update something you did not expect will need it and you are now scrambling to do anything because you are messed up.

Besides I like writing my own code more than dealing with some lib shenanigans.

The black hole that is node_modules? Like, it’s fine for my personal projects, but every time I install the project I implicitly trust some 2000 authors to not have added “rm -rf /“ to their installation scripts.

Ideally I wouldn’t have to worry about that.

If you upgrade regularly, maintain sufficient business-logic testing to catch breakage, and regularly audit your dependencies for those which are no longer maintained so that they can be ripped out and replaced with maintained alternatives, then you're fine.

If you skip any one of those, then your full-time job is longer about shipping features/value but just trying to get the damned thing to work.

The maintenance required, and often the poor development choices for something that can be really basic, or tailored to what you actually need.

That being said, I’m not against dependencies, but I do think you should think about them quite a lot. I can’t imagine not using Typescript, eslint or ts-jest as an example. Similarly I can’t imagine interacting with many Microsoft APIs without using their SDKs (even as painful as that can be at times). Or using not using React(or similar) for the frontend.

But we needed an OData client, and there were some available through NPM, but they were terrible. You could use them, but you wouldn’t have a good time. So we wrote our own, which is on NPM actually, but also a really good example of dangerous packages as we made a lot of breaking changes through the different versions as we were building it. If you had blindly used it, you would have had to put in a ton of maintenance. Something which isn’t really exclusive to the Node package environment. We have a C# application that uses RestSharp instead of the Microsoft SDKs (for God knows what reason) and it’s a few versions behind by now because they made some rather big changes that nobody wanted to spend time on. On the flip side, we’ve also used Dapper as an ORM and been very happy with that. So often it’s also a question of using dependencies where and when it makes sense.

But blindly going into them is a maintenance nightmare. The less usage packages see, and the fewer maintainers, and so on, also has a big impact on whether you can trust them or not. We update quite a few of our packages automatically after a one-two week period, and others we only update manually after vetting the code. So it’s always a resource question.

I learnt this lesson dealing with Windows DLLs back in the VB days. More dependencies == less sanity.

I've written Node/React web applications that break completely after a few months of not being updated, and after a year or more they need refactoring to remove deprecated/dead dependencies. And they always come with half a dozen security warnings that are impossible to get rid of.

I've seen the default Go linter aggregator have to drop half the linters because no-one updated them to use generics. It's annoying, but not so much that I want to dive in and fix someone else's linter.

I've spent hours learning the interface for ${useful library}, only to find they changed it in the latest release and they're not maintaining the old version.

And I'm certain that the massive use of dependencies will cause a huge security problem this decade. Best account I've read of this: https://medium.com/hackernoon/im-harvesting-credit-card-numb...

I use as few dependencies as possible these days.

Installing a few basic packages in a Node project can quickly result in 1000s of packages. That's 1000s of packages you need to maintain (update), 1000s of packages that can be or become compromised, or get deleted. 1000s of packages that will spam "this package is deprecated" messages in your logs or just break because NodeJS decides that importing and requiring files should work differently again for some reason.

1000s of packages with backwards-incompatible changes that seems to happen every few years in NodeJS land. Some packages have backwards-incompatible changes and you don't even know because you're not using the package yourself and suddenly the behavior of your code changes in some subtle way you'll find out about a few weeks after upgrading.

I think the problem is that NodeJS's standard library is either too barebones or nobody wants to use it, since there are libraries for all kinds of basic stuff and everyone ends up using different libraries (or different versions of the same library) for the same thing. HTTP clients, array/string manipulation and date/time handling are some notable examples.

For me, personally, my "hatred" developed while working on a team that shipped software living in other people's process spaces.

Classical examples of this are metrics clients, tracing clients, loggers, HTTP frameworks, routers, etc. Tools that are hard/impossible to spin out into a dedicated microservice but generic enough to warrant trying to share that code across many services.

These dependencies can have measurable impact on the user experience.

Some teams just toss the code over the wall so "it's someone else's problem." But I'd take a "every instance of my code in a production server is a production deployment" approach.

Being able to test your changes against real production traffic, knowing when you've made a change that is going to break services, owning that, being able to roll it back yourself. All of that is stuff I want in my dependency manager.

A dependency is more than something down inside a node_modules directory. It's a code path that executes alongside, or even directly handles incoming customer requests.

Trying to track down and upgrade my library/framework/w.e. when it's deployed across 100,000+ instances for 10,000+ services is... not fun.

Even less fun is trying to surgically update _only_ my code and not rebuild the entire world pulling in a huge pile of changes along with my changes.

Package management failed me when I tried to model deployments as a collection of stakeholders shipping code to production.

This isn't just an npm problem. This sucks in pretty much every package manager I've worked with. Don't get me started on apt... unpack an ar archive into a global namespace, run some Rube Goldberg scripts, and hope for the best.

The Nix paper gives me hope, but I never got around to using Nix in production before I moved on from large-scale deployments.

I think it's also a question of education. You can read these sorts of blog posts and tinker with this type of development without using it at work. A lot of web developers are confused about basic things like the request cycle and I think the use of magic easy frameworks is partly responsible (also this is stuff you don't learn in CS school). Seeing how things work without dependencies or frameworks really helps.

Different combinations of versions break in novel ways. They each come with their own build constraints. Some of them have bugs that go unfixed for years. It's find to submit a PR, but sometimes they're not accepting any. And it's a transitive dependency anyway, so using my fork wouldn't even help. And so on.

Neither.

I've also used pretty much every package manager out there, and this may hurt some, but I've always loved the npm ecosystem the most.

You might as well assume (c) whenever you use a library provided free of charge. Be prepared to either fix the bugs yourself or make sure there are alternatives you can use as a replacement. Even if the author is responsive now, there's no guarantee you'll get help once your code is in production a few months later. The library might even be archived at that point.

Lets say you've used FFTS, the Fastest Fourier Transform in the South [0]. Performance is spectacular, but now it's no longer supported and you find a bug. The code is impressive: Thousands of lines of code that dynamically generate SIMD instructions. But it's impossible to understand. You have chosen a library you cannot repair yourself. This is fine when there's alternatives, and there are for FFTs, and they are easy to swap out. If you can't easily swap out one library with another, you better make sure that you can repair it yourself.

And I hope people don't harass or "harass" library authors into merging pull requests...

[0] https://github.com/anthonix/ffts

We patch our dependencies.

This adds a diff to your repo, not an entire fork of the dependency.

A package manager like pnpm will install the package as usual, then apply your patch over the top.

https://pnpm.io/cli/patch

On a reasonably large project, you accumulate sufficient dependencies that you end up doing all of that.

(author here) Anything that's available to you, without configuration, after installing nodejs and starting the application. That includes the whole NodeJS standard library and maybe standard (or extremely common) utilities available on the server (curl, git, ls, find, sed, etc).

I still have to find a way to store data and I really don't want to create my own database, but I haven't found a database in the standard library.

Std is not a dependency.

Your comment got me curious if it’s at all possible to build a Node web service without importing any Node modules (ie with just the global/module-local namespace). Glancing at the globals docs, I think you’d be limited to WASM, binary extensions (like N-API), or maaaaaaybe doing something horrifically hacky with CJS require.

> The article is interesting, though a little messy in its structure.

Thanks :) I tend to write messy. I'll give future blogs some more revisions to reduce the mess.

> Isn't the easiest way to do this to make a request to the URL via the browser and see for yourself?

In the applications I'm usually working with, there are many pages, and in many cases they do things differently based on the current context (permissions, roles, state of the business, date)

I generally test every route at least twice; one happy flow, and one unhappy flow (trying to submit a form to trigger validation errors, or trying to visit a page you should have no access to), but generally there are many tests per route.

I have routes that have literally 1000s of lines of tests for every "real world" case imaginable. Additionally, every bug that's found gets reproduced with a test before it's fixed.

The whole web application is usually just gathering params and converting them to domain logic calls, and taking the return value and converting these to a HTML document as a response.

The domain logic is fully tested, and the web controller is tested separately by some very basic tests (a simple GET to see if the form fields are present, a POST to see if errors are rendered, and to see if there's no error in the template). Occasionally I will replace the domain logic modules by mocks if the domain logic modules are complex or slow, to keep tests isolated.

Because the domain logic modules are tested thoroughly AND the web controllers that invoke the domain logic modules are also tested, most pages (that are backed by modules representing the domain logic) are tested multiple times in different ways. It is not uncommon for me to have widely more LOC for tests than actual code.

There's currently 2300 tests in my largest NodeJS project and the whole suite runs in about 30 seconds, but most modules can be tested in less than a second. It is only the "main" module that's slow because it's actually starting the whole application and being tested by actual over-the-network HTTP requests using fetch while the application is also doing real API calls in the background to external services which are mocked in other tests.

I think he just elided the imports of node's new-ish built-in `node:test` module: https://nodejs.org/api/test.html