Hacker News Reader

> Or is there something I'm missing here?

There may be some platform-specific dependencies, but at least when building on Windows, I saw 67 dependencies downloaded.

One thing that may be skewing the count is that internally coreutils is packaged with one crate per command. There are about 100 commands, so you'll have seen at least that many separate crates building. They're listed here: https://github.com/uutils/coreutils/blob/6e8c901204934029c88...

My impression is that ecosystems where tracking dependencies is hard tend to discourage them, while ecosystems where tracking dependencies is easy encourage them. The coreutils crate depends on separate md5, sha1, sha2, and sha3 crates, if managing external dependencies was hard I imagine those would be bundled together. But it is easy, so it makes sense to organise the crates so that each just provides one specific piece of functionality. Think single responsibility principle applied to packaging.

Urgh, that's just a verification nightmare waiting to happen. Who is going to wade through all that mess, review every package, pin its version etc...? You need reproducability before you can get safety and security.

Cargo uses lockfiles by default to pin all dependencies of an application, so reproducibility shouldn't be much of an issue.

No one reads lockfiles. In python, people have successfully backdoored projects by sending a PR that upgrades a dependency and regenerates the lockfiles.

Somewhere in the 1MB lockfile patch, they pin a dependency to a version with a known remotely exploitable flaw (sometimes a zero day, sometimes not).

If you care about security, api stability and legal compliance, you should minimize the number of developers and packages you trust. My rule of thumb is that, for a team of ~10 developers, each package dependency adds a week or two of developer maintenance time each year or so.

Having a dozen packages where one will do, and having them do the same thing recursively means you need to spend 144 times longer dealing with such things.

Why is number of packages and not package size the relevant metric here? Especially when all the packages live in the same repository, there's practically no more overhead to review each package qua package, versus just a bunch of directories in a package monolith.

This "144 times longer" number seems pulled completely out of thin air.

Both are relevant. To their point, each additional maintainer is also a hidden dependency of each package, so the number of packages is relevant.

In the case in question the dependency count as crates is separate from the dependency count as git repos. In this case things that would normally be one big library where split in many different functional crates in the same repo.

Twelve packages with twelve dependencies each is 144. I’ve seen >10x this level of blow up on multiple products at multiple companies.

One company missed installing malware behind all the firewalls of most of the fortune 500 by less than a week.

We made a procedural change to the build process of a product that had been shipping for years.

The change hit our CI/CD 5 days before one of the big language package managers started spewing malware at us.

Package maintainer count bloat is a serious real world problem.

Ignoring it is negligent.

I'm not suggesting that we should stop reviewing dependencies. I'm pushing back against the idea that internally structuring a large repo or project via independent modules necessarily explodes the burden to review.

It may be the case that this preference for well modularized code leads to lots of disparate small packages in the ecosystem, but within the scope of a single project, I see no reason why modularization poses any intrinsic risk.

Actually you did miss something.

Thinks like `cargo tree | wc -l` or the number of dependencies cargo builds are not appropriate as they count _internal_ dependencies (from the same repo).

Given that it's split into many "sub-crates" in the same workspace this will make the numbers unnatural high.

Or with other words _most_ of the 417 "dependencies" for building the "unix" feature set are internal ones in the same repository. I.e. the project is just split up into many small parts.

Additionally many deps are reused between many internal deps.

Removing duplicate dependencies, dev-dependencies (e.g. testing tools) and build dependencies less then 100 deps are left.

Looking through them many are rust versions a C/C++ libs which are "available per default", like e.g. unix_socket, libc and similar.

Then some are thinks which come from rust deps being split up more, e.g. there are md5, sha1, sha2, sha3 as separate crate.

There are also a bunch of "internal" deps of deps e.g. backtrace-sys.

Anyway still a lot of deps but most are quite reasonable and could be "taken care of" in some way or another if they want to ship a distro based on this.

Most scripts don’t depend on the POSIX standards anyway, they depend on GNU or BSD programs. CPython’s configure script, for example, blows up if the system grep is BSD and not GNU. The whole C build ecosystem is duct tape anyway and things rarely build without tons of manual effort (figure out what versions of which dependencies need to be installed to which locations in the file tree and then maybe things will build, but none of it will be documented and good luck reverse engineering those Autotools/CMake scripts!), so citing concerns about breaking things seems really disingenuous.

I believe a main point of the article is that while linux might be fine with this state of affairs, OpenBSD is not and actually care about having a specific standard.

His statement was that there was "no attempt" to write POSIX tools. As a "statement of fact." Irrespective of how much work there is still to do for coreutils in terms of compliance, they are an attempt. They may be be completely unsuitable as a drop-in replacement right now, but that was not the "fact" that he asserted.

Rust dependencies are just compilation units. Do people complain when a coreutils-scale project ultimately depends on 400 .c (and/or .o) files?

To replace anything in OpenBSD, you need way more than POSIX-compliance -- you need OpenBSD-compliance. Which means every non-POSIX option has to be implemented down to edge cases, every departure from or controversial interpretation of POSIX (POSIX is vague in many places) has to behave exactly the same. Or you'll definitely break scripts.

These nuts don't understand OpenBSD's base doesn't map to GNU coreutils, it ships SEVERAL more tools. They are so GNU and Rust blind they don't get anything.

.ls -1 /bin/ /sbin/ /usr/bin/ /usr/sbin/ | sort -u | wc -l

Good luck reimplementing that.

I don't think you miss anything.

POSIX compliance is a headache which is not only hard to get right but also strongly limits your interfaces and internal tooling.

Many scripts still will run with non full compliance, but that doesn't help if you try to build for OpenBSD.

Also I'm not sure how they ended up with 400 deps for coreutils. I would have expected much less. But yes this means at least until better code signing and so on is the default this isn't at all appropriate for the base of any OS.

I just which the author would have written it a bit less mean in it's wording. (Probably was just annoyed, but sill).

The author is from the 1990s era of Linux / FOSS, before kindness was part of the culture, when the culture had a tough competitive streak where mistakes and humility were seen as signs of weakness and disrespected. High expectations wee tightly bound with low levels of patience. And many people were scared off from contributing.

> POSIX compliance is a headache which is not only hard to get right but also strongly limits your interfaces and internal tooling.

Platforms which are POSIX compliant usually offer extensions and then allow for selecting actual POSIX-compliant behavior. It's not a very strong limitation if you're already fairly close to that anyways.

Somewhat academic as I have no dog in the fight, but ...

> To actually achieve POSIX-compliance is not an easy task and requires lots of testing.

Why are the existing tests not enough for a new implementation?

Sweet summer child, imagining OpenBSD has comprehensive tests? This is UNIX, old-school. If it seems to work, it works.

> Is this really appropriate for a core system component to have this many dependencies?

Absolutely not. The entire OpenBSD base system can be built without an internet connection.

after you are done downloading all the source.

You can check that the source hasn’t been tampered with before starting the build, and know you’re building the right code base.

Builds that curl junk and use a dozen language specific package managers don’t have that property.

That is exactly the set of properties you get with Rust and Cargo: that the source hasn't been tampered with, and that you know you're building the right codebase. It would be deeply irresponsible to build a package management system without those properties.

True enough... but you still don't know what the dependencies do. E.g., one added ads. Another added use tracking. This could even be legitimately in line with the dependency's purpose.

Unless you're manually vetting all the code, 400 dependencies means trusting a lot of third parties.

Trusting a lot of third parties is almost as irresponsible as not bothering with checksumming, etc.

See npm and pip for examples. I suspect cargo will have its share of incidents over time, unless it is somehow curated by a small group that tests and maintains the packages, but if it is, then it’s comparable to apt for c/c++.

This is one of those things that's strictly true, but is kind of misleading because it doesn't capture the full picture. In particular, there is actually no guarantee that the source code you see on GitHub matches the source code you compile from crates.io. Now, most responsible maintainers tag each release and publish exactly that tag to crates.io. Which is good. But that's just a convention. There's nothing enforcing it.

If you want to actually review the source code, then you'd have to download the crate archive itself and review the code there. There is some tooling for this (notably, cargo-crev), but it's definitely not a normal part of development in the Rust open source ecosystem as it stands right now. For the most part, trust and reputation hold the system together. For example, imagine what would happen if someone found malicious code in one of my crates that I was duplicitous about (i.e., published it on crates.io but kept those code changes off of GitHub).

You're right.

I've heard from reliable sources that this is actively being worked on.

That assumes every user is a developer, expert in security assessment, knowledgeable in tricks like trusting trust, and all the languages used in the code base.

You can also build Rust and its ecosystem without an internet connection. It works the same way; you need to get a copy of everything first, but the build itself does not require the internet.

(This has been a hard constraint for effectively forever, because both Firefox and Linux distributions have this requirement.)

I understand that if you have all dependencies locally, you don't need to download more dependencies.

I understood NotAPlumber's remark as: openbsd has no third party dependencies. In other words: there is one party to trust, not 400.

If I misunderstood the BSD case, I'd love to hear it.