Ask HN: Why are JavaScript dependencies so messy? | Hacker News Reader

Ask HN: Why are JavaScript dependencies so messy?

59 pointsgraderjs83 comments

I love JS, but every once in a while a new bundler comes along that "solves everything". And it works, for a while. then it breaks. Why? Why are there so many edge cases? I don't understand it. We only have a few module types (AMD, CommonJS, ES modules), with a few types of import and export syntax. How hard can it be to get it always right?

Like parcel. It worked. For a while. And now if you check the GitHub there's 690 open issues, and I had issues today getting it to work when running after an 'npm i' done in v17 or v18, yet it's fine to run in v{16,17,18} if 'npm i' is done in v16.

And snowpack: v0 (or 1) worked great, but the next version broke so many things (compared to the prior version) that I need to keep the dep version locked to the earliest ones for packages where I use that. Tho I guess that's more of an API problem.

What I'm really talking about is: why can't we just have a bundler that works always and everywhere (and I don't want to 'wait for' deno)?

Why would parcel start to get bugs...how hard can it be??? :...(

View on HN

I think there are a few things here that are kind of unique to the js/web platform.

- JS doesn’t have a proper standard library and so you tend to have many more dependancies in a project than with other languages.

- Because of that you then have a very large tree of dependancies, often with incompatible version requirements of the same package.

- Due to the nature of needing to “bundle” your code for distribution on a “slow” network, a lot of optimisations (tree shaking, code splitting, async module loading) are required to minimise your bundle size.

- You say “We only have a few module types”, that’s two more that every other language. And they are incomparable in subtle ways.

Those and some other issues are completely unique to the web platform, they are some difficult problems to solve.

However, I think the current generation of tooling has finally got there. Vite, with its esbuild and Rollup backend is bloody brilliant. If you use a framework with official Vite support it just works.

Why doesn't Javascript have a standard library? As in, why has one never been made and shipped with the browser?

Oh, they ship stuff akin to one, but that's the web APIs, not a “standard” library. As in it's not really standard.

Javascript has a standard library. What Javascript doesn't have is a robust kitchen-sink library akin to other more complex general purpose programming languages, because Javascript was designed to be used in the context of a browser to script webpages, and its actual standard library reflects that.

Unsurprisingly, because we've apparently decided Javascript needs to replace all existing languages in all current and future forms of application and system development, its actual standard library is no longer sufficient for many peoples' requirements. But they would rather claim the language "has no standard library" in order to justify the insane complexity of the JS ecosystem than admit that perhaps it just isn't the right tool for every job.

Second "Vite (esbuild) just works". I would also recommend start using bun.sh on smaller projects/scripts so you have first hand knowledge of how ready to production it is. Jarred works in an insane mode, so I wouldn't be surprised if it turns get mass adaption before deno does.

There have been many attempts at standard libraries that are always extremely opinionated and often introduce performance penalties and an additional layer of defects. That has nothing to do with the tremendous number of dependencies that would exist anyways if you aren’t extremely aggressive about dependency management.

I'm sure I'll get downvoted for this one but quite frankly it's where the PHP developers all slithered off to. It was inevitable.

I avoid to the point of refuse to work with any JavaScript platform at all. They are a universal shit show.

Edit: honestly I stand by this comment. I literally have spent hours this year dealing with fucked up messes in nodejs, package security issues after repos were hacked and code made it into NPM that displayed banners on commercial sites. My comment is toxic yes but quite frankly both communities are as well (PHP and NodeJS) and utterly earned that ire and discontent. The fact people lean on the stack and produce stuff that handles critical aspects of people's lives is utterly frightening. So yeah toxic, I don't care. Feel free to be annoyed. It's still a turd, just been rolled in glitter.

Not sure about the developers, but PHP and Javascript have a problem in common:

PHP has some sort of standard library but it's not intuitive (the function naming is a mess), sometimes outdated, wrong or broken (but the broken behavior has now ossified and will never be removed for backwards-compatibility reasons).

JS has basically no standard library at all.

Both of these issues mean everyone has to use third-party packages to make up for that, of which there are tons, each with their own issues, breaking changes, upgrades, etc.

PHP changes behaviour of core functions all the time now (and subsequently breaks old programs with every release). In every release stuff is deprecated and deprecated stuff removed. Look at https://www.php.net/manual/de/function.implode.php for a telling example and check the 8.1 changelog at https://www.php.net/releases/8.1/en.php for a list of bc breaking changes.

Only about two decades too late.

Don’t move the goalposts.

I never set a goalpost, so certainly didn't move it... but one very reasonable goalpost I might have set would have been "Don't let your language stagnate in a hole of bad practices on top of bad design for a decade or more while the internet fills up with content memorializing and perpetuating that state."

I'm not a PHP hater, and I'm the first to admit that what I've seen of recent work on the language is a breath of fresh air. Even in its "fractal of bad design" days, the language had its merits and has paid my bills before. But I'm also the first to admit it absolutely dug its own way into that aforementioned hole, and personally I think it is now way too late to climb back out to a point of regaining significant mindshare.

JavaScript has a ton built in but developers get in a habit of using libraries that introduce their own api and polyfills on top. I blame awful browser support of years past. Today most new features are available pretty quickly across all the major versions.

> quite frankly it's where the PHP developers all slithered off to

PHP isn't my favorite language but using phrasing like "slither" to describe an entire community of developers is frankly shitty. Every language has its problems including whatever you use.

Unless Python, then slither is fine.

Dammit. You have my upvote.

I always say the reason PHP is still so hated is because rather than arguing about it on the internet to defend the language, PHP programmers are going home to their spouse, kids, and hobbies at the end of the work day.

That's the Visual Basic guys who do that.

The PHP developers spend most of their time outside smoking.

I spent most of 2005-2013 doing contract work getting people out of near business ending buckets of shit thanks to the PHP community's inability to produce a product that was fit for purpose and the awful staff contracting building on top of the wobbly pillars of excrement. Slither is exactly the word because they were the first people to slither away and point fingers elsewhere when something was wrong. Now ten years after that I am in regular contact with people actively getting rid of it and it's the same problems. And here I am now on Sunday evening eviscerating another failed nodejs stack left by the same slithering fiends.

Oh no it's exactly right. Slithering is the word.

I don't get anywhere near this level of sheer incompetence on a daily basis in the Python, .Net or Go ecosystems. Nor do I have problems sleeping wondering what shit show am I going to face on a Monday morning with them.

8 years of contracting work and this bad of an outlook on the community, maybe you didn’t charge enough?

I was being paid large amounts of danger money to undo damage.

You sound like a fun guy. Enjoy.

> I'm sure I'll get downvoted for this one...

Rightfully so. Your comment is utterly toxic.

Those might not be the same people, but for many people JS is the door to the world of software development - the same way PHP was (also for me).

The irony of this comment is that Composer & packagist is one of the sanest and most well-thought-through package management systems. PHP actually fares better than the vast majority of language ecosystems on the specific topic being discussed in this comment thread.

Although Composer was incredibly slow. No idea if that’s still true, but I believe it was partially a consequence of its approach to dependency constraints, which is what made it actually quite robust.

Composer 2 isn't bad, I have had comical situations in the past where it got very confused and ended up gobbling 32GB of RAM before crashing, but that hasn't happened in a very long time.

PHP's dependency management is actually quite nice. The community developed a standard [0], a client [1] and repository [2] that pretty much deal with everything I've had to do with PHP dependencies without issues.

[0] https://www.php-fig.org/psr/psr-4/

[1] https://getcomposer.org/

[2] https://packagist.org/

So you can use nice tools to bake a poop cake :)

That's not particularly pointed at PHP that. There is a big problem in this industry about where we get our software from and how it's built and managed but that's well outside the topic of this thread. Fundamentally everything is fundamentally broken with dependency management, trust and competence.

JS is retracing some of the same steps of Python. They are both a fun language with not a lot of constraints and a lot of capability.

What this does is create a large community of enthusiastic beginners led by few (prone to follow trends by authoritative members) so you get some experts but a lot of beginners and beginners love libraries, especially ones written by the expert contingent. The JS stdlib is not as strong as it could be, nor is python, so libraries in some cases make a lot of sense but for beginners it is easy to rely on libraries for everything.

Eventually some of these beginners turn into skilled programmers and either stick with just JS or Python, and then the other subset moves on and tries other languages that address some of the perceived 'shortcomings' of JS/Python. What you find is that a lot of the ecosystem becomes that beginner crew and a repeated brain drain of the other 70% who move on.

> The JS stdlib is not as strong as it could be, nor is python

Comparing the js stdlib with python one tells me you never used python.

Also the theory that experts move on while noobs stay is just a theory. I'm going to assume you personally moved to rust and need to call yourself an expert.

Seems people aren't liking this.

However the average python project has 25 dependencies and the average js has 174 (source https://snyk.io/reports/open-source-security/) so perhaps js and python do not have equivalent stdlib?

Not sure how good average is. For small-ish projects my experience is more like Python having like 5 of them, with a handful transitive each, if at all and in JS with anything besides a hello world it's n^n^n.

The amount of stuff I've put into production with just flask + requests + dbsomething comes to mind, although I don't remember how many transitives those have.

I could never understand why people insist on using requests rather than the stdlib (which works just fine).

When async became a thing, I remember hitting some bug in aiohttp so I wrote myself a tiny (just doing the bit I needed) HTTP async client, which I figured was easier than wrestling aiohttp into compliance.

These days aiohttp is quite nice for async, and for sync I just use the stdlib.

My async client, if it can be of interest https://github.com/ltworf/localslackirc/blob/master/slackcli...

aiohttp is too new for the time I'm talking about. And no, I would not say that the stdlib was fine and requests was absolutley worth using in Python 2 :)

Dependencies is different from bundling. Javascript's dependency management is an absolute dream, and you can have a newbie up and running with a consistent environment very quickly. Compare that to Python, which still doesn't have a declarative package manifest format (leading to extremely slow/inconsistent package resolution), or a mature "lockfile" for deterministic installs. npm has both of these out of the box. You can even assert what versions of npm or node you need in your package.json to make sure you're using known-working versions.

EDIT: Turns out Python just finalized a pyproject.toml format in 2021. Of course, this doesn't really help much until every package out there migrates. npm has used the package.json format pretty much since day one. And there is still no standard lockfile, other than dumping the output of `pip freeze`.

> Compare that to Python, which still doesn't have a declarative package manifest format

It does, with pyproject.toml [1]. It even had two [2], but setup.cfg is deprecated because it was specific to setuptools.

[1] https://setuptools.pypa.io/en/latest/userguide/pyproject_con...

[2] https://setuptools.pypa.io/en/latest/userguide/declarative_c...

About the situation in python... Have you tried using "pip freeze > requirements.txt" to get a deterministic set of requirements? Or am I thinking too simplistic?

Yeah, that's unfortunately not good enough. Ideally, you want to store the graph of dependencies so that you can understand why each package is being installed, with useful metadata like the hash of package to ensure you're getting the same code.

Just because Python's story here isn't good doesn't mean JS's is automatically good.

Your environment may be consistent at this point in time but it's still a horrible mess.

How so? OP's gripes were mostly around bundling, but the dependency management in JS is pretty solid.

Because it's a language and ecosystem built by mostly designers and unicorns not backend programmers who rely more on logic and functionality rather than colors and rainbows. My 2c.

I tried to compile a rust project the other day and I realised I had to download hundreds of dependencies. I guess strong typing helps but dependency hell is just common for any moderately popular platform.

With both Rust and node, it isn’t easy to do the things that should be easy. So developers will take the path of least resistance to make it easy. And both Rust and node make their package manager their project manager so everyone is an “npm i” or a “cargo add” away from making their lives easier.

I'd say that's better then the C way, where every projects Implements their own version of a linked list, json parser and so on, no?

C hides all the dependancies. Rather than seeing the 100 sub dependancies, they just tell you to download the binary of libthing where the sub dependancies are prebaked in so it looks like you have 4 dependancies.

Maybe on windows. Certainly not on linux.

To be fair a lot of the JS developers that wanted something better moved to Rust, and brought a lot of the same ethos with them. Anybody who's looked at the import list to their favorite Rust app has probably said to themselves "woah its just like a node app". Not saying it's a bad thing, I'm more of a stdlib guy but I see how it grew to be the way it is.

I hated NPM... then I started trying to use some Python projects.

Now I like NPM.

Yes!

I feel like so much criticism of NPM comes from either people who've only ever used NPM, or people who haven't used NPM and just saw a big node_modules once.

A common one in work is Java devs criticising the huge node_modules folder but not realising that .m2 exists as their IDE handles it for them "by magic".

To be fair I prefer Maven, but probably more for the culture. In Node you generally ask for the latest version and hope it’s still available/the project still builds when you come back to it. pom files are more clunky but they do tie you down more

This isn't really true though. Npm supports a full selection of semver specifiers, including pinned versions & Maven versions plugin handles ranges allowing using the latest just as npm (pretty sure maven also does this natively). There's no difference in what each manager does here: any community difference is going to be incidental convention.

Npm also uses package-lock files which tie your install down strictly, in case that's what you're referring to?

I mean yeah, you’re right. But in practise npm allows you to wildcard version numbers. And when you come across old npm projects, you’re likely dealing with people that have wildcarded their dependency versions.

Sure if you start from scratch you can adopt idioms that work for you, but in general you’re more likely to get into dependency messes with npm than maven from my experience largely because of the wide use of patterns instead of locked down versions being pulled in.

If it's an old npm project without a lockfile, then yes, community convention will scupper you there. Lockfiles have fixed that.

How is python package management a problem? spin up a new venv, pip install whatever, away you go. I don't do a lot of python work so maybe I'm not hitting the boundaries, but it's never been anything less than pleasant to work with in my experience.

JavaScript can go directly to hell.

Dependencies are hard because dependencies are hard.

There’s no easy dependency management system.

Bash - the most interoperable language in existence - contains its own version of malloc because some systems used to need that (back 30 years ago).

JS has an extremely uncertain runtime. Is it in a browser? Engine? PDF?

Because JavaScript itself is nothing but messy and a lot of new programmers start with it and somehow learn it and follow in the same vein

Same with PHP: its bad reputation was due the proliferation of terrible, insecure and unmaintainable code written by first time programmers (or should I say webmasters) in the early 2000s.

These days its Javascript and the horde of fresh faced web designers and React engineers that try and reinvent the wheel. I don't think the quality of JS engineers on average is that low, programming culture has improved a lot, but without real world experience one tends to both overengineer and underestimate the problem. The JS world is very brittle because of it, and because of a frankly terrible language.

1. backward compat. most still use commonjs over ecmascript. some even use amd bundles.

2. design by committee. most large projects have users that have edge cases. devs sometimes create messy apis just to cater to a larger group of audience. notnecessarily bad, sometimes the product just grows, sometimes its for security reasons.

3. incompetence. sometimes you dont need a library. sometimes their implementations suck in terms of readability, security, space and time complexity, maintenance.

4. code golfing and not using linters. these 2 are pain in the ass.

Just don't say that you aren't fan of creating new app with 17 vulnerabilities out of the box.

Things change. Web development has a lot of incidental complexity that these tools help manage. Eventually tools are overcome by the complexity.

Parcel in particular is a clever wrapper on top of several other tools, so I’m sure when their dependencies break they pay the price.

I think some of them are also victims of their own success. Lots of these tools have a happy path, but then they gain more and more users who want various exceptions to the happy path.

It’s messy because it actually works. If JavaScript couldn’t scale indefinitely then it would never get to the point where it is messy.

npm introduced the concept of independent transitive dependencies. This is very unique when first introduced. Most other dependency managers require the importer to reconcile “conflicting dependencies”.

Dependencies are a messy, human, complicated problem.

What would an ideal dependency manager look like? It could read the source code of a library and the overall program, determine what sections of code / apis / interfaces are actually used, and perform that recursively for each interfaced library with the libraries it interfaces with.

Or the compiler / programming languages would require lots of annotations to aid this, possibly both human and compiler. Imports and interfaces would need to be very fine-grained.

Now add in operating system compatibility, versioning ...

And then what if library invocations aren't a static/explicit reference? What if they are "plugins" whose module name is in a configuration file and is dynamically loaded?

Oh, no small thing, what if your language doesn't even have any degree of strong typing, so analysis of apis cand api invocations can't easily determine the types of the objects, methods, signatures, parameters, data being used to be invoked...

Javascript has been in the worst situation for this: competing solutions, rapidly evolving language, rapid churn in frameworks and libraries and interfaces.

Javascript also didn't have a strong server-side / cli toolchain origin and centralized authority. It was a browser language repurposed by dozens of vendors for other purposes, almost all without regard for an overall ecosystem. It was subject to major corporation monopoly battles.

Javascript also has a "neckbeard" shortfall. It's a language ecosystem where those people with decades of experiences aren't generally present.

None of that is a good formula for effective package management at an ecosystem level.

Python is kinda similar, at least pip provides some centralization, but good lord does python suck for package management and dependency resolution. Python started as a traditional UNIX-y language for scripting alternatives, so that provides a more solid foundation.

But look at linux package management. Slow, huge, error prone, and ultimately requires containers to solve with reliability.

Yes it's terrible but then again alot of package managers and dependencies are, however i think the sheer volume in javascript, and the accessibility for anyone to post to npm has it's own set of problems, one thing i noticed is that interfaces constantly change in NPM packages compared to others i work with, its very hard to not ending up locking to a specific version because of API updates. If this is due to JS community or the ease of access to create packages, i dont know but it has something to do with it.

That's because JS with its dynamic typing is great for developing code quickly, but not so great for maintaining and evolving it over a longer time frame.

Then there's the lack of the standard library, which means your code has to depend on code of random people who may or may not know what they're doing.

Also, because it's the web where patching code is quick and easy, people don't seem too bothered with quality.

I've chosen mobile development myself and fortunately things are a lot more sane here.

Probably the only real reason I avoid JavaScript is this mess.

Maybe if you maintain an dependency you can feel important.

I feel like a few npm packages only exist to pad someone's cv.

To left pad their CV

Does that actually come from all the instructions of "build your own github profile" with projects in there. And npm package is the lowest hurdle outside cloned first tutorials...

The different module syntax is the problem. If we just had the standard import/export syntax from the beginning I think 99% of problems would be avoided.

Going to try and give a reasonably systematic answer to this:

Firstly: the title of the post is different to the content. The title asks why dependencies are a mess, the content cites problems with bundlers. These are entirely distinct topics. Bundlers don't handle dependencies and package managers (mostly) don't concern themselves with bundling. They're different problems and have surprisingly little in common.

So the above breaks it down to two questions:

1. Content: Is javascript bundling a mess?

2. Title: Are javascript dependencies a mess?

To answer question (1) let's first look at it comparatively. Are other language bundlers a mess? Obviously they don't have any as bundling is a client-side problem and javascript is the only client-side target. So it's hard to gauge this. Maybe bundling is a fundamentally hard problem that will always necessarily be a mess.

Subjective opinion: I think there are obvious ways bundling could be done better and it surprises me they haven't been done but speaking from a place of inaction I'm reluctant to be too critical here.

To answer question (2) let's look at it comparatively again: are javascript's package management and library ecosystem more or less of a mess than Java's, Python's, PHP's, Golang's, Rust's, etc.? Some may disagree here but I think the answer is objectively "no":

- The Ant/Maven/Gradle/Bazek/sbt ecosystem is absurd complexity, and I've seen plenty Java .m2 dirctories worse than node_modules. I think Java must only escape the criticism JS gets here because every Java dev I've met is so reliant on their IDE they don't even know .m2 exists (same goes for ObjectiveC/Swift)

- Python is both better and worse than Java. On one hand, the manager ecosystem is even more unapproachable: the setuptools easy_install pip pipx conda poetry venv pyenv nightmare isn't even helped by consistently demarcated config files in your repo root: good luck figuring out what to run depending on whether there's setup.cfg setup.py requirements.txt pyproject.toml etc. (if they even exist at all - pip not requiring project root dep config is why so many Python readmes just have long handwritten accounts of which deps you need to install manually). On the other hand, python's actual dependency tree handling errs on the oversimplified side. Have two transitive dependencies on separate incompatible versions of a single library? Tough shit, rewrite your code.

Go seems to be leaning too heavily on Git and falling into the trap of modelling their package management system on the implementation rather than modelling the implementation on user needs. It's not terrible though, but that's perhaps due to its age.

PHP is probably the least messy, and Rust - while making some mistakes - is new enough to be following some good modern examples. But overall NPM is by far one of the LEAST messy package management ecosystems.

Other languages do have bundlers and JS is not the only client side target. In the Java world this is called building an uber-jar and if you want to "tree shake" it you'd use ProGuard or R8. Conveniently, R8 uses the same config file syntax as ProGuard so there's a standard for defining what to keep/rename/etc. Or if compiling to native code, then GraalVM native image which deletes dead code and creates a single binary at the same time.

These tools seem to work better than the JS equivalents in my experience, or at least they are a lot more stable. It's probably because they're products of companies who want to maintain stability for their users, they aren't just projects that were written, uploaded to GitHub and then eventually abandoned. ProGuard has been maintained for over two decades, R8 was written for Android and is maintained by Google, GraalVM is a 10+ year project maintained by Oracle etc.

The Java dependencies situation is not so messy, I think. You've named a variety of different build tools but they all agree on where to find dependencies, how they are named, how to download them, and dependency graphs aren't quite as large or deep as they get in the JS world. Most projects use Maven or Gradle. Ant is obsolete, Bazel is really only used by Google or Google-adjacent firms and sbt is for a different language. Your ~/.m2/repository directory is a cache of every library version you ever downloaded, it's not the set of JARs used in a specific project.

The biggest problems JVM world modules have are:

1. The half-completed, not really successful attempt at modular JARs which encapsulate their internals. Adoption is slow, and it's not unheard of to find popular libraries with broken module metadata.

2. If you get version conflicts the build tool has to pick one version, usually the highest. The JVM can do isolation of dependency graphs, but the integration between build/deployment tools and those APIs isn't there. Fine if that library is backwards compatible. A mess if it's not.

3. The general prevalence of 'shading' (bundling) as a way to resolve such issues.

4. The near total reliance on Maven Central as the one big server. Of course most language ecosystems have this issue, but it used to be better with many dependencies sitting on other repositories. Over time those other repos went away and now it's all just Sonatype and Central.

That's a good insight on bundling and not one I had considered. I guess bundling for dynamic network delivery is so different in my mind to bundling for portability (& possibly one-time network delivery).

But while they're certainly different I'm sure there are many similar concerns. Just reinforces my instinct then I guess, that JS bundlers are ripe for improvement.

On your remarks around the Java package management ecosystem, I was aware of a fair amount what you point out, & the rest is helpfully elucidating, but it didn't seem to contradict my point that it's pretty messy?

I guess messy is a pretty subjective term. Messy relative to what? I've used a lot of language ecosystems over the years and this is one of the least messy. It isn't mess-free though, for sure.

Go's "package management"--I presume you mean the module/dependency management--has almost no reliance on Git. It has a couple of ways to "fetch" code, one of which is Git. Many people use the Git method because it's convenient. Go also supports zip files over HTTPS which is the recommended way, and in any case "fetch" methods can be added in the future and do not impact the module/dependency management at all.

I feel your pain on this. As an occasional NPM library author, I feel that Typescript can help with this, although NPM itself is not structured in a way that compliments Typescript: it’s awkward to write in Typescript, publish to javascript, then consumers fetch your library and go back to writing in Typescript.

I'd like Deno to take off, personally.

Letting the versions of your project dependencies float without control guarantees some headaches.

And, of course, the Fast & Furious dependencies development lifecycle has some side effects

> every once in a while a new bundler comes along

I think the answer is (at least partly) in the question.

The necessity of rewriting the world in javascript as richer browser-based apps took off and native browser plugins went out of style meant everyone and their cousin needed (or "needed" https://xkcd.com/927/) to implement some library or functionality from scratch themselves. On top of that, the npm default of publishing everything publicly meant that what once would have been an internal utility or wrapper became a de facto public api.

That behavior ingrained itself into the culture and community around the ecosystem far more than is good for it, imho.

Messy? I invite you to try some C

Reply on news.ycombinator.com