Unpopular opinion: you should copy/fork/DIY your dependencies for everything | Hacker News Reader

DIY your dependencies for everything

twitter.com

59 pointstosh40 comments

I don't think this is that unpopular of an opinion; you only have to be burned a couple of times to earn this one

You'd be surprised how many are in the industry who didn't - yet? - get burned or don't consider it good enough reason to change.

Can you share some examples where you got burned?

An ancient version of svelte/sapper did something like this to me. I was pinning my versions correctly. A year or 2 later, I needed to make a one line change to my project. I was on a new computer. Neither my computer nor the existing netlify autodeploy could successfully build my project. Luckily, it was tiny enough to remake in any fresh SPA framework in less than an hour.

Usually, this happens with any tutorial/blog that tells you to pip/npm install and doesn't specify an exact version or provide a pinned package.json. if it's a year old, it almost always won't work without estimating what version of X was likely published at the time of blog entry.

My side projects from 2012-2017 cannot be built or ran because of dependencies. My jsbin repo with lots of experiments cannot be ran anymore. But I have the sqlite database.

I forgot to pin dependencies when I was working. It would take a lot of trial and error and effort to get back to where I was. Or I have to rewrite the experiments.

Take lots of screenshots and screen casts to preserve your software!

A screenshots of multiplayer editor that allows transclusion that I can't build anymore.

https://github.com/samsquire/liveinterface?tab=readme-ov-fil...

Not unpopular at all. Vendoring is a fantastic idea. Most bigcos mandate it for anything running in production. Helps keep your supply chain tight and removes the fun surprise of production blowing up because of a backwards incompatible change in a downstream library.

It depends on the use case. Taking responsibility for your dependencies need not be that big a deal: if you trust the upstream then automate a pull. Then you're set up to turn it off and deal with issues at leisure when they arise.

There's tradeoffs, as there are in all things. It's not always the best idea; but neither is "lets just dynload it off a URL at runtime" neither.

Many dev shops have to use local clones of github repos, just because of availability / lag / latency problems.

Popular opinion, but often weighed against effort and risk.

The big places do this. The startups don’t because it’s not a large enough risk.

Vendoring saves dependency-hosts a buttload on bandwidth costs too. And makes you resilient against their outages. And any future shenanigans on your current dependencies, either for cost or code injection.

Vendor everything. It's better for everyone except your code host, which is easily changed if necessary.

You don't have to vendor for that. You can also setup a transparent proxy which will save all the dependencies you use in a "private repo". And if you update deps often, it will save a lot of space.

Entirely possible, but there's ample evidence that GIGANTIC amounts of people/projects/etc do not do that. Probably because everyone has to configure and run new opaque blobs of software per language / package system / etc.

Vendoring (generally) means it's already on disk and needs no config or separate processes to take advantage of. It's nearly foolproof.

People also don't generally vendor. Just saying it's an option - there's too many "vendor everything" voices around. I've got a 4.4GB repo which says "please don't, that was a bad idea".

I agree with this but more so the DIY part. I usually _do_ have dependencies in my projects, a handful of important frameworks but avoid the utility libraries as they have way too much surface area and its easy to roll them myself. But this is a deeply unpopular opinion in just about every company I've worked at.

To go into more detail: for a custom built web app I might use nextjs, react, express, passport, prisma, and even some others. Sounds like I'm using a lot of dependencies right? I kind of regret using prisma, and maybe I can avoid using react / nextjs in the future. But I will handroll minor data manipulation needs, or requirements for 'fake' data, tests, etc. My experience at work is that the exact same app at a big company would be slower, and would have about four times as many dependencies, a complicated monorepo, and more. And trying to stop it is utterly futile.

This might have been an unpopular opinion 10 years ago. But since then, we have supply-chain attacks, and we have left-pad. I think vendoring or mirroring all dependencies is starting to become standard practice nowadays.

At the very least, I think it's generally a good idea to pin all dependencies to the exact version.

Question for those in this thread who hold this view: Please share your stories, along with which language ecosystem you're working within.

(Cards on the table: I've occasionally vendored stuff for a variety of reasons - because we needed to apply a patch that upstream didn't want or didn't accept quickly enough, or because we were stuck on an old version and needed to backport a fix - I've also on occasion copy-pasted 50-100 lines of code for a specific bit of functionality to avoid bringing in a 5MB library. These are exceptions - in the general case I prefer to depend on upstream wherever possible.)

I was hired to revive a near death startup that got an infusion of money. The original team built the product with OpenEmbedded, pre-cursor to Yocto Linux.

The build no-longer worked because many of the links to the dependencies were dead, and the original team didn't cache the dependencies and put them someplace safe. I spent a couple weeks just tracking down packages, from probably not great sources, to get everything up and running again.

A.k.a. “vendoring”

Mirroring is another form, i.e. use GCR for external container images or GCS for yum/apt repos.

No really, you can setup nexus repo and keep things there. Vendoring is kinda bad for git repo, especially modules based ones.

Second this. If you put a proxy between any repo, you garuntee the lifetime of these projects vs huge repos of mostly static files.

That's an unpopular opinion? It's exactly how I've done things in my own companies for decades. In my opinion, to do it any other way involves taking on far too much risk.

My habit of minimizing external dependencies has let me avoid numerous serious problems.

Tell it to most javascript people and they'll look at you like you grew a second head.

I suspect it's popularity is inversely proportional to the average years of experience in the sector you work in.

As you progress in experience you typically build up enough skills to DIY deps when you need to, encounter situations in which you can get burned with not vendoring, and have your eyes opened to the rather middling quality of a lot of dependencies.

Idk how I feel about this. I think this is only an appropriate solution if you are 100% capable and can take complete ownership of patching said dependencies.

I get how there is risk associated with a supply chain attack, but what are you going to do when you don't understand a vulnerability and need to fix it?

Most problems aren't impossible to solve of course, but those who've been working with a codebase for a long time probably have a more intimate knowledge of how it works.

Better option is to proxy the package manager and then you can always upgrade and store those.

Its also acts to speed up testing and deployment

How does the act of copying/forking open your eyes to anything?

In my case, it's helped me to learn. It's a point of pride to me that my JS canvas library has no direct dependencies, but there's some things I can't get my head around. For those - in particular a pseudo-random number generator and a really fast gaussian blur filter - I copied code and learned by adapting it to work well with the rest of the library's ecosystem.

In other cases, I've taken inspiration from other libraries and implemented that functionality from scratch. The library's entire filters system was inspired by the way SVG allows users to build filters from building blocks. For the color system I should have reached for a well-maintained library (such as color.js) but instead chose to implement my own system using the CSS Color Module 4 spec as my guide. I've probably learned too much about color gamuts but if a question about the difference between D50 and D65 white points ever comes up in a pub quiz, I'll have the answer!

You've mostly described the DIY part which I've specifically excluded, as its connection to learning is obvious

> JS canvas library

Is that publicly available anywhere?

Oh, yes. I regularly spam HN with links to various parts of it. Check the submissions link in my profile.

This is what I do in Go. I vendor all of my modules. If it's not in my codebase, it's out of my control. I require control.

Sounds like your stack has a really bad Dependency management system.

For example, Java’s maven is stellar (grade too if you use Maven’s)

Basically if your in js, this means pushing all node_/modules to your repo.

Do this for every capitalist organization you depend on. Fork the organization's entire operation into a free automated software library. This transforms the org's capital investment into a zero cost production multiplier, delivering value while eliminating the need for capital and intellectual property.

OT are there some HN guidelines for posting single tweets? I get if it was something deep, but this appears some guys shower thought. I don't think that's what most people comes to HN for ...