Hacker News Reader

Comment by jchw | Hacker News Reader

It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the lockfile and does not adhere to it when installed as a dependency. It will pick the newest version of dependencies that matches the spec in package.json.

Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version.

This avoids the node-ipc issue entirely, at least until you update the go.mod.

View on HN

This really depends on the specific package manager: if you're building an application in Rust, its lockfile will contain the full tree of dependencies, locked to a specific version.

I might be misunderstanding GP, but I think what they're saying is that when package A depends on package B, building package A will use B's lockfile. Assuming that's the case, I think this is generally not how Rust does things, as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate, although there's nothing stopping anyone from just removing that line. I think I remember reading in documentation somewhere that checking in Cargo.lock for libraries is discouraged (hence the policy), but I don't recall exactly where since it's been so long. (That being said, there's a pretty decent chance you were the one who wrote that documentation, so maybe you might remember!)

> I think this is generally not how Rust does things

That is correct.

> as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate

Even if it is included in the contents of the package, Cargo will not use it for the purpose of resolution.

The "don't check it in" thing is related, but not because it will be used if it's included. It's because of the opposite; that way new people who download your package to hack on it will get their own, possibly different Cargo.lock, so you end up testing more versions naturally. Some people dislike this recommendation and include theirs in the package, but that never affects resolution behavior.

Ah, good to know! I'm glad I asked

Yes, this is true; I think the article is definitely making the distinction vs NPM and not Rust.

The article seemed to go out of its way not to mention any specific package manager or ecosystem. So I think comparing to Rust is completely reasonable.

It didn’t mention one by name, but Rust hasn’t been subject to any widely publicized supply chain attacks. They do, however, mention left-pad by name. I think it can be implied that they really did just mean npm.

They definitely should have said that then. Simply referring to "lockfiles" paints with a very broad brush that includes a number of package managers that don't have the problems that NPM does.

The default way Go handles go.mod is fairly different than the default way Cargo handles Cargo.lock files, including for example with libraries.

Also, when the blog says:

> Moreover, when a dependency is added with go get, its transitive dependencies are added at the version specified in the dependency’s go.mod file, not at their latest versions, thanks to Minimal version selection.

I believe that is significantly different than default Cargo behavior and for example default 'pub' behavior for Flutter (though I know approximately nothing about Flutter package management beyond a cursory search just now ;-)

To my knowledge, both Cargo and Flutter 'pub' prefer the most recent / highest allowed version by default when asked to solve constraints, whereas Go does not.

Cargo: [1]

> When multiple packages specify a dependency for a common package, the resolver attempts to ensure that they use the same version of that common package, as long as they are within a SemVer compatibility range. It also attempts to use the greatest version currently available within that compatibility range.

Flutter 'pub': [2]

> For each package in the graph, pub looks at everything that depends on it. It gathers together all of their version constraints and tries to simultaneously solve them. (Basically, it intersects their ranges.) Then it looks at the actual versions that have been released for that package and selects the best (most recent) one that meets all of those constraints.

[1]: https://doc.rust-lang.org/cargo/reference/resolver.html

[2]: https://dart.dev/tools/pub/versioning#constraint-solving

In that same section, the blog describes the behavior of 'go install foo@latest' and contrasts it to how the default install "in some ecosystems bypass pinning."

That is also a difference in default behavior between Go and Cargo.

To install a 'foo' binary, 'go install foo@latest' gives you the latest version of foo, but the direct and indirect dependencies used are the versions listed in foo's go.mod or a dependency’s go.mod file (and not whatever the latest versions of those direct and indirect dependencies might be at the moment the install is invoked).

'cargo install foo' supports the optional --locked flag, but its not the default behavior: [1]

> By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute which versions of dependencies to use, possibly using newer versions that have been released since the package was published. The --locked flag can be used to force Cargo to use the packaged Cargo.lock file if it is available.

There are definitely pros and cons here, but to my knowledge it is not "just NPM" that is being contrasted in the blog.

Finally, I'm no world-class Rust expert, but I like using Cargo. I think Cargo is a fantastic tool that set the bar for package mangers, and it has done great things for the Rust community. But it is easier for communities to learn from each other with a base understanding of where & why different choices have been made, which is part of what is behind some of my comments around Go's behavior. ;-)

[1]: https://doc.rust-lang.org/cargo/commands/cargo-install.html

So, I believe that you're confusing two different cases here.

`cargo install` is not how you add a dependency, it's not like `npm install`. `cargo install` downloads the source for and installs a runnable binary program, like `npm install -g`. Cargo does not currently have an "add this dependency to my Cargo.toml" command built-in, but `cargo add` is coming from this purpose.

With `cargo install`, the default behavior is to completely recompute the dependency tree before doing the build. The `--locked` flag modifies that to use the lockfile included in the package to do that build instead (and in fact fail compilation if it does not exist). That lockfile will still be a full graph of all dependencies and transitive dependencies to build the binary, it doesn't like, recurse or use any lockfiles that exist in any of the dependencies' packages.

Hi Steve, first, thanks for weighing in here!

I might have misunderstood your comment, but in my GP comment I was indeed attempting to contrast 'go install foo@latest' with 'cargo install foo', which both install binaries. (I wasn't talking about 'go get bar@latest', which now is just for updating or adding dependencies to a project).

Also, I'm contrasting what happens by default at the moment either binary install command is run. My understanding is Cargo's (non-default) 'cargo install --locked foo' behavior is similar to the default behavior of 'go install foo@latest'. In other words, the default behavior is fairly different between 'cargo install foo' (without --locked) vs. 'go install foo@latest'.

I edited my GP comment to simplify the example to use 'foo' in both cases. Maybe that helps?

Ah yes, I did miss that, thank you / sorry :) Too many sub-threads around here!

I don't know go install's semantics well enough to know if that comparison is true or not, I'm just trying to make sure that Cargo's semantics are clear :)

This is, roughly, what "go install" does:

It will copy a binary to $GOBIN. If the binary is not built, it will be built from source. If the source is not available on the local system, it will be fetched.

During the build, any dependencies of the build target not available of the local system will be fetched.

I believe Cargo does still have less strictness over dep versions than Go modules, since it will never use a module newer than the one specified in any go.mod file. Lockfiles are generally not honored recursively, and I don’t think Cargo is different here? Hope I’m not spreading misinformation, though I couldn’t find any docs with a cursory glance.

I don’t want to make assertions that I’m less sure of, but I think NPM and Cargo are actually more similar than different here. They both specify exact versions in lock files, for all nested dependencies, but don’t honor the lock files present inside dependencies, instead calculating the nested deps from the constraints.

Cargo does not do "recursive lockfiles", that's correct.

This is also true of npm and yarn, as far as I can tell: package-lock.json and yarn.lock contain the exact version of every transitive dependency.

I always forget the exact semantics, but the parent's description of them as "recursive" is not the same as Cargo; Cargo determines the full tree and writes out its own lockfile, if dependencies happen to have a Cargo.lock inside the package, it's ignored, not used.

How does go deal with the diamond dependency issue when transitively you've a dep on the same package via two different paths?

As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.

I think you might have a small typo here:

> by choosing the lowest version specified in any package that depends on a given package

It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).

Yep, I regrettably described the system wrong. It’s the highest version specified, and thus minimum version possible.

That's a nice & concise way to describe it.

Minimalistic Version Selection might have been a better term for it.

That did confuse me a lot. Selecting the maximum makes a lot more sense, since that gets SemVer assumptions correct.

Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5?

Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints?

Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.

> Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5?

No, you end up with the highest explicitly required version. So 1.8 in that scenario, if I followed. (Requiring 1.5 is declaring support for "1.5 or higher". Requiring 1.8 is declaring support for "1.8 or higher". 1.8 satisfies both of those requirements).

> if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.

By default, you get 1.8 (for reasons outlined upthread and in the blog post & related links), but you have the option of getting the latest version of C at any time of your choosing (e.g., 'go get C@latest', or 'go get -u ./...' to get latest versions of all dependencies, and so on).

Also, you are using the word "pin". The way it works is that the top-level module in a build has the option to force a particular version of any direct or indirect dependency, but intermediate modules in a build cannot. So as the author of the top-level module, you could force a version of C if you needed to, but for example your dependency B cannot "pin" C in your build.

I'm fairly sure Go Modules does not support what you’re describing. It specifically avoided having a SAT solver (or something similar), unlike most package managers. You specify a minimum version, and that’s it. 1.8 would be selected because it is the highest minimum version out of the options 1.5 and 1.8 that the dependencies require. Unless you edit your go.mod file to require an even higher version, which is an option. Alternatively, you can always replace that transitive dependency with your own fork that fixes the problems, using a “replace” directive in your go.mod file.

If your dependencies are as broken as you’re describing, you’re in for a world of hurt no matter the solution. I also can't remember ever encountering that situation.

Well after 12 years of using ruby and bundler and maintaining a bundler-ish depsolver at work, and playing around a bit with cargo, I can say that it is becoming clearer as to why I don't grok go modules at all.

The lack of a depsolver is a curious choice...

I don't think my example was remotely "broken" at all, that's just another day doing software development.

> I don't think my example was remotely "broken" at all, that's just another day doing software development.

It's not the norm for me. If this is what you consider to be the norm, then this kind of statement doesn't make me feel any better about Ruby.

I will say that Bundler is one of the better package managers, but the existence of the constraint solver doesn't fix this problem -- Bundler doesn't allow you to have multiple versions of a single dependency. The problem is fundamentally the dependency not maintaining its compatibility guarantees, which I would definitely call "broken". Sometimes breakage is unavoidable, like with security fixes that you want to be available even to users of existing SemVer versions, but it should not be a common situation.

If I'm understanding you properly, recursive lockfiles means that if I depend on some chain of dependencies A->B->C->D->E, and E has a security vulnerability that they patch in a new version, I have to wait for A B C D and E to all update their lockfiles before the security vulnerability will be patched on my system?

That's not correct. You can unilaterally decide to update the version of E without waiting for anyone. Alternatively, if only C for example decides to update their required version of E, you would get that version of E if you updated your version of C (directly or indirectly), without needing to directly do anything with E yourself.

Slightly longer explanation of the mechanics here: https://news.ycombinator.com/item?id=30871730

The best complete explanation is probably here: https://research.swtch.com/vgo-principles

But wouldn't that have the same issue then? Developers decide to update their dependencies to patch any security vulnerabilities, and wind up adding installing node-ipc's malicious update

The difference is that it's an explicit choice instead of other package managers who'd happily install latest compromised versions of packages by default.

But if its a manual explicit choice, that adds more friction to these patches, and many developers may not update them at all. It's a trade off

A manual choice can be easily automated; an automatic choice can be difficult-to-impossible to de-automate.

It is trivial to manually upgrade dependencies in NPM. You just use `npm update <package>` with an optional version number if you want. And upgrading all dependencies of a Go package is also a single command. So honestly it seems like there is very little difference. My main point here is the trade-off. Either you reduce the friction for upgrades, and run the risk of malicious upgrades like node-ipc. Or you increase the friction, and run the risk of security vulnerabilities being unpatched in many projects.

I personally prefer the former. Encourage upgrades, but then NPM should also have a separate repository for community verified / trusted packages to reduce the chance of a random single developer damaging the entire ecosystem (left-pad, node-ipc, etc)

If I set up a new Node project I get the highest 'supported' version of whatever. If I add a new dependency I get the latest version of any transitive dependency I didn't already have. As far as I know that's impossible to disable. That's the automated upgrade I mean.

I see, in that case yes Go does have more tooling for being able to install the minimum vs the latest of all packages, using their `update` command if you want the latest. But it would also be trivial for Node to add a command to grab the minimum of all dependencies when installing new packages. They just haven't felt the need to add such a feature. Because again, it comes down to which side you want to encourage: installing minimum versions to prevent malicious updates, or installing latest to patch security vulnerabilities.

Reply on news.ycombinator.com