Hacker News Reader

Comment by MaxBarraclough | Hacker News Reader

> I don't want that level of control over other people's projects, it's scary

How far do you take this though? The average GNU Linux distro ships with a whole pile of packages already installed, from a multitude of different authors.

View on HN

Packing is most of the point of a distro. They are specifically taking on that responsibility. They also have a better perspective to handle overall compatibility.

In a sense you've pointed out the alternative to having the programmer handling the packaging -- having some third party package and distribute. And this separation of responsibility turns out be almost always be a better solution. Distribution and coding are, after all, two full jobs (without 100% skill overlap). Plus, hopefully it indicates that at least two sets of eyeballs have at least glanced at the code (not the full desired many-eyeball outcome, but as good as we can expect sometimes).

Given that Debian (and its descendants...) packages a shitload of npm packages, it's a wide stretch to say there is more QA for these packages from the Debian side than there is from the npm side.

The one thing that Debian provides is that in the case there is a security issue, admins worldwide only need to do "apt update && apt upgrade" and they are safe, without having to check all of the software that runs on their servers (as long as said software comes from Debian, that is!).

I do think people would be served, generally, by being more aware of the fact that distros are not some doing some hardcore security vetting. But the alternative is just to use whatever was pushed up to NPM, right? In that case, Debian packager+NPM push > NPM push by definition, unless the Debian packager somehow provides negative QA, which seems unlikely. (Also, on the incredibly unlikely offchance that some Debian packager reads this comment -- your work is incredibly useful and I very much appreciate it, just trying to be realistic about what exactly is provided by your group!)

> unless the Debian packager somehow provides negative QA, which seems unlikely

It has happened before. Last time there was anything major was over a decade ago though.

https://lists.debian.org/debian-security-announce/2008/msg00...

Yes, but AFAIK, those are heavily tested or audited in some manner. That's different from including code written by randos in your app that they can remotely change at any time.

In practical terms, can they really be audited?

This is at least obvious DoS, I’m sure it’s easy to slip in an innocuous line that, dunno, ships your ssh keys to some rando server.

look at diffs?

Can you really say, with a straight face, that you inspect the diffs of your entire dependency closure every time you deploy an update? With the level of scrutiny required to detect a maliciously-obfuscated security exploit?

If you can, you're an infinitely more diligent developer than I am, that's for sure.

Fortunately the problem could become more tractable if something like SES / Endo takes off:

"Endo protects program integrity both in-process and in distributed systems. SES protects local integrity, defending an application against supply chain attacks: hacks that enter through upgrades to third-party dependencies. Endo does this by encouraging the Principle of Least Authority. ... Endo uses LavaMoat to automatically generate reviewable policies that determine what capabilities will be distributed to third party dependencies."

https://github.com/endojs/endo

Nice link, thanks! Good to see Mark Miller is still working in this space.

I look at gits diffs to see what's changed between versions, but that's for fun and not diligence.

> With the level of scrutiny required to detect a maliciously-obfuscated security exploit

Nope. Not paid to do that and I have not been given any such responsibility.

That said, I think the attack vector on this is very low.

Packages are rarely updated to the latest version.

We don't use a lot of packages.

We mostly use packages from trusted sources.

We use packages that are open source.

And frankly, not just diffs. You’d need to inspect the initial state, and any new dependency added. That’s potentially hundreds of thousands of LoC.

I actually do this, on occasion. (Not 100%, and not to a degree I'd say "yeah, I'd've caught this colors/fakers thing." But enough to say that I've seen a decent sample.)

There is literally, on average, no difference in average quality between commits on FOSS projects, and commits on projects we pay external entities for. Some paid projects are just crap code, and some FOSS code is extremely high quality.

I've had to roll-back / hard-pin dependencies from both low-quality FOSS & low-quality paid projects because of commits that — once you find & read them — are just bananas.

(I have no idea how to solve the root problem here, honestly.)

> I have no idea how to solve the root problem here

I'm not even sure what the problem is. If it's "updating dependencies introduces severe side effects" wich, I think, should be accounted for in the process

> Yes, but AFAIK, those are heavily tested or audited in some manner. That's different from including code written by randos in your app that they can remotely change at any time.

It seems like the problem here is more cultural than technical, specifically that the JavaScript community has fully embraced packages that are "written by randos" that are "wrappers around three-line Stack Overflow answers."

I use packages, but I wouldn't use any that are developed by a rando with no reputation or "institutional oversight." An important part of choosing to use one is evaluating the maintainers.

Does the JavaScript ecosystem have anything like Apache Commons? I'm guessing not, but it probably should.

Not just that they've fully embraced the "written by randos" but even worse: "as soon as the rando publishes an update or change, use it!" They seem to fully automate updates because packages are so poorly written (and frankly, it probably helps with revenue stream, if their client's websites occasionally break and need them to fix it.)

...and meanwhile NPM's idea of vetting packages is basically "YOLO, BRO!"

It's both technical and cultural.

Javascript is used on the front end. Front end devs obsess (or at least used to obsess) over download sized. So you'd have crazy stuff like custom builds of Underscore (https://underscorejs.org/) with just the functions you wanted. Think manual sandboxing, if that makes any sense. You could get a package of Underscore with just map, filter and reduceRight, if you wanted to.

Now, when Node came around, people wanted as much as possible to have the same libraries available on the front end, so the same obsession with size was carried over.

Ergo the micro-milli-nano-packages they make.

Now, about the technical solution to this. We have this, for well defined programming languages (read: statically typed ones, or dynamically typed ones with a clear structure).

It's a linker. Tech from the 1950s.

Link (include) just the stuff you want, "tree shake"/"remote dead code" whatever you don't.

https://www.joelonsoftware.com/2004/01/28/please-sir-may-i-h...

Java's largely to blame for this, Sun REALLY, REALLY hated stuff that could be hooked into any OS and wasn't portable, so they didn't provide a linker. Everything was supposed to be on their JVM and you were going to install their JVM everywhere (2 billion devices!!!) and to hell with small stuff or heaven forbid, including native libraries. Javascript followed (on top of the Java restrictions they added: dynamic, poorly defined language, that would have made linking with tree shaking really hard, anyway). .Net also followed.

Almost 3 decades later we're trying to undo that damage.

The problem with tree shaking has been twofold:

- JavaScript is a very dynamic language with dynamic property access and a few other features that make it hard to guarantee that the linker won't accidentally remove too much

- historically there was no standardized "module" format until ESM (ES modules) came up (with some time in between with few competing non-standardized proposals), so statically analyzing exports/imports was difficult; in frontend you'd long rely on just creating and reading global variables (i.e. side-effects).

Hence it's been "safer"/easier to create small packages.

But it's not only this. Once you put a mega-package in your repo, it's easy to gradually start relying more and more on the things it gives you. Even if it supported perfect tree shaking, you'd call one method here, one method there, and with each build your bundle size would balloon (which is not good if you could write one line of code while lib method's code is 1000 lines because it supports IE4 and 17 parameters).

Whereas when you rely on small packages, you need to make a conscious choice each time to pick another dependency.

You probably don't care about this on servers written in C++ or Java that much; but on frontend it's a big deal; hell, even when building native apps for Android/iOS you have size limits for the stores submission / limits for the number of methods (tech limitation in Android). Big companies invest crazy money to shrink their native bundle sizes (https://blog.pragmaticengineer.com/uber-app-rewrite-yolo/).

I think history (20+ years from now) will prove that for all but the smallest, almost toy, systems, dynamic typing from the 80s and 90s was a mistake.

The maintenance burdens these languages are creating will make Cobol look like a kiddie bike with training wheels next to monster trucks.

I think that dynamic languages played an important role in pushing for the development of mainstream static typing that didn't suck. ML's been around for a very long time, but there was seemingly little interest in pervasive type inference in languages actually used in industry until they had to compete with the concision of dynamic typing.

Actually building large systems in dynamic languages? Probably going to turn out to be a mistake though.

I really don’t think it is fair to blame it on Java. Having very little native dependency is a huge plus to an ecosystem (just look at what Java will be able to do with Loom thanks to the almost all-Java dependencies). Also, Java was particularly keen on downloading class files at runtime, so linking everything was not even possible.

And it is not even a difficult thing to fix without going the linker way: java’s modules essentially solve it (as well as javascript modules could/can) — just specify what is visible outside a package and both ecosystems can “tree-shake” non-used code (though I dislike this nonstandard term)

> Does the JavaScript ecosystem have anything like Apache Commons? I'm guessing not, but it probably should.

This isn't a Javascript problem, this is a node problem. Node is just a Javascript platform among many. The fact that the node community decided to go with all these "nano packages" has absolutely nothing to do with Javascript. Nothing forced Node, the distribution, to come with such a barebone standard library. Absolutely nothing... but the idea of being dependent of NPM which was orchestrated by NPM founders, that's how NPM, a private business made money and eventually sold to Microsoft.

There’s no reason people can’t keep local caches of these libs if it is a major concern. This seems like a non issue.

Stale libraries are more likely to contain known security vulnerabilities.

I know it's bad practice, but I just checkin vendor files/libs to source control. Makes auditing new releases of libraries a bit easier. Assuming they aren't binaries of course.

I don't recommend this approach

I haven't had issues yet, but it's considered bad practice for a reason. What headaches am I in store for?

You're committing something that is not a part of your source code into your version control system, assuming your source code is git, this is irrevocable without rewriting history.

It's mainly a nuisance. It takes up unnecessary space. Introduces possible annoying merge conflicts etc etc and it's not trivial to remove it.

As reference, I migrated repositories from TFVC to git. One team relies on checking in packages into source control, another one does so far less. One repo is significantly nimbler.

Checking packages into source control is making your VCS a package manager. Presumably you have one. Don't hammer nails with your screwdriver

The benefit of having your dependencies vendored is that you have everything needed to build your application without having to download stuff from the internet. You get to ensure what exactly makes it into your application. Yes, it will increase the repository size, but I don't see why merge conflicts would be a problem since you are just replacing a file with a new version.

I imagine your commit log would be polluted with commits just for changes in the packages.

You would have one anyways for changing the lock file or whatever. Changing your dependencies is something that you may want to be able to undo. It's useful to be able to go back to a known working version of your program with versions of your dependencies that you know work.

> you have everything needed to build your application without having to download stuff from the internet

that's using version control to act as a proxy. AFAIK, a lot of package managers already cache local copies

> You get to ensure what exactly makes it into your application

sorry but i don't follow

> I don't see why merge conflicts would be a problem since you are just replacing a file with a new version

Are you working alone?

>AFAIK, a lot of package managers already cache local copies

But this cache is usually not easily transferable to someone compared to them just cloning a repo.

>sorry but i don't follow

You have the source code to all of the dependencies in your application.

>Are you working alone?

How many forks of a dependency do you use? Just using the master branch and upgrading along that should be good enough for 99% of your dependency.

> But this cache is usually not easily transferable to someone compared to them just cloning a repo

right, so they need to download "stuff from the internet". it Doesn't matter much if that stuff is from a remote repo or hosted by a package repository. Except if it's architecture dependent, in which case you definitely don't want to share across architectures. Not to mention they may already have a viable copy in a proxy or cache

> You have the source code to all of the dependencies in your application

I'm afraid I still don't follow

> How many forks of a dependency do you use? Just using the master branch and upgrading along that should be good enough for 99% of your dependency

Well, if I was expecting things to not break I'd never follow upstream master for a dependency.

But the question pertained to merge conflicts. If several people track the same remote and check in dependencies into VCS I'd expect annoying merge conflicts

Or are we perhaps misunderstanding each other? I'm not sure I follow what you mean by forks. Releases are typically on different branches or tags

Git submodule documentation explicitly says[1] it's designed for adding 3rd party libraries to a project in this exact scenario.

[1] https://git-scm.com/book/en/v2/Git-Tools-Submodules#:~:text=....

I’ve been doing it witb Yarn 2 / pnp and it has been great so far.

You checkout a project and start it. No downloading required, and no 10000s of files.

I also like doing this, but with node, you have a massive tree of thousands of files. It's crazy and gross.

Yarn 2 pnp kinda fixes this

You’re free to update as you like. It’s entirely possible to audit your local packages.

Letting maintainers update your projects is a convenient feature. If it is a liability in your use case you can work around it. Yes it will take more effort, but your use case justifies it.

Reply on news.ycombinator.com