Build It Yourself
lucumr.pocoo.org
lucumr.pocoo.org
In C++, I control my dependencies. In Rust I get above 100 so quickly I just give up. In terms of security, let's be honest: I have no clue what I am shipping.
Also Rust does not have ABI compatibility and no culture of shared libraries (I guess it wouldn't be practical anyway given the number of libraries one needs). But that just destroys the OS package distribution model: when I choose a Linux distribution, I choose to trust those who build it. Say Canonical has a security team that tries to minimize the security issues in the packages that Ubuntu provides. Rust feels a lot more like Python in that sense, where anyone could push anything to PyPi.
It's signed by a maintainer. And maintainers are vetted. You trust Debian/Ubuntu to only allow trustworthy people to sign packages.
How are Docker / Python / Rust secure? I don't know any of the people who created my docker images, PyPi packages, or Rust crates.
Yes.
We're basically back to sending around EXE and DLL files in a ZIP. It's just that now we call it a container and proudly start it as root.
BTW, I agree with the author of the article: Sometimes you're best off just merging dependency source code. It used to be called "vendoring" and was a big thing in Rails / Ruby. The big advantage is that you're not affected by future malicious package takeovers. But you can still merge security patches from upstream, if you choose to do that.
but but what if people just don't want that??? it given people jobs and things to do (lol, this is serious)
also there are certain things better to use third party library than develop in house like crypto
This is true for individual libraries and also for Linux distros.
You’re also forgetting process isolation and roles which provide strong invariants to control what a given package can do.
No such guarantees exist for code you load into your process.
There are significant advantages to the distribution model, including exact provenance at the point of delivery. But I think it’s an error to treat it as uniquely trustworthy: it’s the same code either way.
- You can trust someone as in "think they're honest and won't betray me".
- You can trust someone as in "think they are competent and won't screw-up".
In this case, we trust distributions packagers in both ways. Not only do we trust that they are non-malicious, we also trust that they won't clumsily package code without even skimming through it.
"They won't screw up" is the wrong way to look at security: humans always screw up. We're essentially fallible. We would all do well to remember that the xz backdoor was not caught by distribution code review: it was caught due to a performance side effect that the next adversary will be more careful to avoid.
Software stacks are getting bigger, largely due to engineering pressures that are outside of the control of distributions. It's a basic fact that that means work for the same (or fewer) people, which in turn means less time spent on more complicated packages. Under those conditions, competency is not the deciding factor.
This is, separately, why having lots of little packages can be (but isn't necessarily) a good thing: small packages that decompose into well-understood units of work can be reviewed more quickly and comprehensively, and can be assigned capabilities that can be tracked and audited with static analysis. You can't do that as easily (or at all) when every package decides to hand-roll its own subcomponents.
Sure. So pick a system that includes at least one extra possible barrier to them screwing up than the system that has none.
(I think it’s great that distributions provide one extra set of eyes. But we’re deluding ourselves if we think that can scale to the actual realized demand for both new software and continuous change to software. It’s also not very fair to the distributions.)
The the average distro package maintainers is “person who cares that package repos stay up to date and is willing to commit their time to that work”
If you link to dependencies that you get from Canonical, then you know that a security team is caring about the packages that Canonical distributes and actually patches them. Who patches security issues in PyPi? The individual package maintainers, which may well be pushing a malware leveraging typo-squatting. Typo-squatting is not really a thing in the Ubuntu package manager, is it?
Not saying it's perfect. Just that it's better than the far west. PyPi is pretty much like downloading a .exe from a webpage without even opening the webpage (because back in the days, opening the webpage could sometimes - not always - tell you that it was clearly shady).
Canonical doesn’t do this as a good will. They make money on it, right? Who vets packages for fedora? alpine? arch?
Since none of this is particularly effective or efficient, money is made from someone’s perspective, but mostly it functions like a “cost of doing business” tax that you can’t avoid
I know who created the Docker images, because I'm the person who built them!
A lot of the time you can build your images either from scratch, or based on the official base images like Alpine or Ubuntu/Debian or some of the RPM base images, not that much different than downloading an ISO for a VM. With a base, you can use apk/apt/dnf to get whatever packages you want if you trust that more, just remember to clean up the package cache so it's not persisted in the layers (arguably wastes space). For most software, it actually isn't as difficult as it might have initially seemed.
As an alternative, you can also look for vaguely trustworthy parties that have a variety of prepackaged images available and you can either borrow their Dockerfiles or just trust their images, for example, https://bitnami.com/stacks/containers and https://hub.docker.com/u/bitnami
Most likely you have to have trust somewhere in the mix, for example, I'm probably installing the Debian/Ubuntu packaged JDK instead of compiling mine in most cases, just because that's more convenient.
Also, rootless containers are pretty cool! People do like Podman and other solutions a lot, you can even do some remapping with Docker if there are issues with how the containers expect to be run https://docs.docker.com/engine/security/userns-remap/ or if you have to use Docker and need something a bit more serious you can try this https://docs.docker.com/engine/security/rootless/
> How are Docker / Python / Rust secure? I don't know any of the people who created my docker images, PyPi packages, or Rust crates.
Me neither. But the same goes for those from Debian/Ubuntu. In fact, neither I know anyone who vets those who sign and publish packages. What I know is that I can build my own images from container files and then I’m back to installing those apparently trusted packages from Debian/Ubuntu.
> We're basically back to sending around EXE and DLL files in a ZIP. It's just that now we call it a container and proudly start it as root.
I don’t get your point. And what’s an rpm or deb? You also potentially run stuff as root… sudo apt install -y… post install scripts…
But I digress. If we're talking build system, nobody is forcing you to use Crates.io with cargo, they just make it easy to. You can use path-based dependencies just like CMake/VCPkg,Conan, or you can DIY the library.
Even with crates.io, nobody is forcing you to not use version pinning if you want to avoid churn, but they just make it easy to get latest.
It's easy to build software on existing software in Rust. If you don't like the existing software or the rate it changes don't blame Cargo. Just do it the way you like it.
No, because it's used as an excuse for a lack of large(r) standard library. It's the equivalent of "the bazaar/free market will solve it!".
You basically end up with a R5RS level of fragmentation and cruft outside your pristine small language; something that the Scheme community decided is not fun and prompted the idea of R6RS/R7RS-large. Yeah, it's hard to make a good, large and useful stdlib, but punting it to the outside world isn't a proper long-term solution either.
It's really a combination of factors.
For almost any functionality missing in the standard library, you could point to 2-3 popular crates that solve the problem in mutually exclusive ways, for different use cases.
Higher level languages like Go or Python can create “good enough” standard libraries, that are correct for 99% of users.
Rust is really no different than C or C++ in this regard. Sure, C++ has a bigger standard library. But half of it is “oh don’t use that because it has foot guns for this use case, everyone uses this other external library anyways”.
The one big exception here is probably async. The language needs a better way for library writers to code against a generic runtime implementation without forcing a specific runtime onto the consumer.
What R7RS-large is doing by standardizing stuff that was already discussed at length through SRFI seems like a good way.
From an external PoV, Clojure seem to be doing okay too.
Who said that? It won't be better, but it'll be always here, stable, conservative and won't depend on something outside std. And it won't be stored on an uncurated supply attack vector like crates.io.
If you want to reach a Common Lisp level of stability, there's no 101 ways to go about it.
Using cargo with distributed dependencies (e.g.: using git repositories) has several missing features, like resolving the latest semver-compatible version, etc. No only is it _easier_ to use cargo with crates.io, it's harder to use with anything else because of missing or incomplete features.
> You can use path-based dependencies just like CMake/VCPkg,Conan, or you can DIY the library.
Have you tried to do this? Cargo is a "many things in one" kind of tool, compiling a Rust library (e.g.: dependency) without it is a pain. If cargo had instead been multiple programs that each on one thing, it might be easier to opt out of it for regular projects.
I have never had a good experience with those. However, using
mydep.path = <path>
in cargo has never been an issue.
And I hate to say it, path-based deps are much easier in C++ / C than other "make the build system do a coherent checkout" options like those mentioned above. So we're at worst parity for this use case, IMHO and subject to my own subjectivity, of course
Something I don't see anyone talking about is, with such systems you have not one but at least two dependencies to audit per any library you want to include. One is the library itself, but the other one is the recipe for the package manager. At least with Conan, the recipes are usually written by third parties that are not affiliated with the dependency they're packaging. Now, Conan recipes are usually full-blown Python scripts that not only decide where to pull the target sources from themselves, they also tend to ship patches, which they apply to the target source code before building. That is, even if package source is specified directly and fixed on specific release, you still can't assume you're actually building the exact source of that release, because the recipe itself modifies the code of the release it pulled.
IMHO, trying to do OSS clearance on dependencies pulled via Conan makes very little sense if you don't also treat each Conan recipe as a separate library. But no one does.
> No only is it _easier_ to use cargo with crates.io, it's harder to use with anything else because of missing or incomplete features.
You're saying the same thing twice here: yes it's easier to use cargo with crates.io. It follows immediately that it's harder to use without crates.io. That doesn't make your argument stronger.
Here's an article on how to use cargo without crates: https://thomask.sdf.org/blog/2023/11/14/rust-without-crates-...
Add to that a free-for-all/no curation repository situation like pypi, npm or cargo together with a too small standard library and prepare to suffer.
In the .NET space nuget certainly makes it easy to add dependencies, but dependencies do seem to be overall fewer and the primary interesting difference I'd note is that a dependency is in fact it's own DLL file - to the extent that it's a feature that you can upgrade and replace them by dropping in a new file or changing configuration.
It strikes me that we'd perhaps see far less churn like this if more languages were back to having shared libraries and ABI compatibility as a first class priority. Because then the value of stable ABIs and more limited selections of upgrades would be much higher.
Duplicate work should only happen if you are confident that your requirements are significantly different, and that you can deliver an implementation as good as a team that has likely focused on the problem for much longer and has acquired a lot more know-how.
It is true that such an attitude might be justifiable for certain security or performance critical applications, you might need end-to-end control. But even in those cases, the argument for trusting yourself by default over focused and experienced library authors is dubious.
Either way, a good dependency manager opens the door for better auditing, analysis and tagging of dependencies, so that such critical requirements can be properly guaranteed, again probably better than you can do yourself.
You are making a ton of assumptions on the quality of the average library you can depend on: that a whole team worked on it, that it worked on it a long time and that it produced good code. That's really not a given.
You are also assuming that writing a feature instead of using a library is "duplicate work". It's not necessarily the case. Maybe the library approaches the problem in a much more generic way than you need. Your requirements are not "significantly different" then, just a subset of the library.
> so that such critical requirements can be properly guaranteed, again probably better than you can do yourself.
Are you saying that the packages available in cargo are systematically audited? I believe that the vast majority of them is not audited at all.
I'd like to see an ecosystem develop that provides a small amount of convenience on top of plain old vendoring. No central repository to create a false sense of security. No overly clever version resolution scheme to hide the consequences of libraries depending on dozens of transitive dependencies. Just a minimal version resolution algorithm and a decentralized registry system—give each developer an index of their own libraries which they maintain and make downstream developers pick and choose which developers they actually trust.
Maybe a bit like Maven if Maven Central didn't exist?
Just don't add dependencies, it's that simple. If you have enough time to control C++ dependencies, you can control them in Rust as well.
Again. Don't add dependencies. Just don't. Vendor it yourself. Or write it yourself. Absolutely nothing is forcing you to use the dependencies except your own desire to save time.
Cargo is giving you the option to A) save your own time B) minimize dependencies. You choose A) and blame Cargo.
I find many libraries in C++ that don't pull 20 dependencies. It's common for me to have a C++ project that has ~6-8 dependencies. In Rust I have never seen that. Either I have no dependency at all, or I have 100+.
I actually had this toy project that I wrote both in C++ and in Rust, for the learning experience. Both did exactly the same thing, with a very similar design. In C++ I had 8 dependencies. In Rust, 186. The thing is, 186 is enough for me to not even read all their names in the lock file. As soon as 25 dependencies magically appear in my lock file, I stop caring and start YOLOing it.
Vendoring means, you fork it, make any modification you want, such as reducing the number of dependencies, but keeping it up to date with upstream is on you.
Rewriting it for yourself is what happens in C++. C++ didn't get flatter by waving a magic wand, it just moved the slider. It's just that code reuse is hard, so you rewrite things that would be a shared dependency in Rust into a header file and call it a day.
> I find many libraries in C++ that don't pull 20 dependencies.
I can find many libraries in Rust that don't pull 20 dependencies (mostly by searching crates written by people like the_mitsuhiko). Some people care, but they are a minority. If you truly care about this, like the_mitsuhiko does, write it with no dependencies.
I, personally, try to minimize my dependencies, but there is a reason why most people don't care - it has little to no impact on them and saving time by NIH is better than not learning from history and reinventing wheel only square.
---
Problem is: There are infinitely many of properties to improve (number of dependencies, byte size, compile time, run time, alignment, crates not made by X, time to market, reproducibility, compliance with law XYZKM, safety etc.) and not enough time or possibility space to do them all.
This is not what vendoring means, this is called forking. Vendoring is just the copy+commit phase, not the modification part.
Say a pretty complex library has 30 dependencies. I fork it, I remove the dependencies, and now it doesn't build anymore. What do I do?
I feel like you're saying "You should not write the small feature you need yourself. Instead, you should heavily modify a big dependency that contains the feature you need, because that's certainly easier and better".
If the library did not rely on those 30 dependencies, it would not depend on them, would it?
> If you truly care about this, like the_mitsuhiko does, write it with no dependencies.
So you start by telling me that I should just remove the dependencies from my dependencies, and then you tell me that you don't do that and instead choose dependencies that don't have many dependencies themselves? It feels like we agree...
> not learning from history and reinventing wheel only square
This is the thing: "I learned from history so I don't reinvent the wheel" should not be treated like a religious belief. It's probably not a good idea to write your own kernel. But I often spend a day or two implementing a feature that I would have gotten from a dependency in 2h. And it's almost always worth it! And sometimes I depend on libraries (e.g. I usually don't rewrite an HTTP library).
There are two cases: 1. You have 100 dependencies. 2. You just started writing your code.
If you are doing 1) then you have to vendor/fork till it's acceptable.
If you are starting from 2) just don't add them.
> This is the thing: "I learned from history so I don't reinvent the wheel" should not be treated like a religious belief.
Nothing in programming should be treated as religious belief. But duplicating code can be just as deadly as dependencies glut, taken to it's logical extreme.
A golden middle approach is best, sadly no one can agree on it.
That's what I was saying: I think that we agree there. My initial point was that I don't think that the package management culture in Rust is a middle approach: it feels like most crates don't care about how many random dependencies they pull.
And I don't care about "cultures". They come and go, and change with time. What I care about are unchanging things - the programs.
Sane package managers allow you to make your code however you want to. OSS allows you to tweak the code in whatever way you need, arguing for CMake over npm is frankly baffling.
I was really talking about culture, which is important to me. The fact that many devs assume that everybody using Linux is on Ubuntu is culture, right? In Rust, it is often assumed that Rust was installed with rustup. It's nice to say "you are not forced to use rustup", but if it doesn't work when you don't, then you are kind of forced :-).
If you don't know what to do in this situation it means you don't know how your dependency works, which means you've abdicated responsibility for your own code's runtime behavior. Please don't take this as accusatory, I do this almost every single day at work because it's unavoidable--I'd be fired if I didn't. But it makes me feel dirty.
Imagine NIH so hard, you start rewriting the compiler.
It’s been a few years but most projects I worked on in C++ pulled in Boost which is more akin to 50 libraries all together in one.
The way I see it the issue is that it's hard to add a dependency _in such a way that no people will have issues building your project with it_. This is problematic because even if you manage to make it work on your machine it may not work on some potential user or contributor's.
> But that just destroys the OS package distribution model: when I choose a Linux distribution, I choose to trust those who build it.
Distros still build Rust packages from sources and vendor crate dependencies in their repos. It's more painful because there are usually more dependencies with more updates, but this has nothing to do with shared libraries.
From my point of view, if it's done properly I can just build/install the dependency and use pkgconfig. Whenever I have a problem, it's because it was done wrong. Because many (most?) developers can't be arsed to learn how to do it properly; it's easier to just say that dependency management in C++ sucks.
There are thousands of tools and methodologies screaming for our attention to "be arsed to learn to do them properly".
They're not owed that attention, they must deserve it. And statistically + historically speaking, most have failed to do so.
You may choose to interpret this as you belonging to a small elite group of intellectuals who "are arsed to learn to do stuff properly" -- that's your right.
I choose to interpret it as "Cargo solves a real problem that has wasted numerous hours of my time in the past". And it wasn't because, for the third time, "I wasn't arsed to learn to do it properly", it's because nobody followed basic protocol and good practices to make the legendary "proper way" work reliably.
You're likely living and working in a bubble. It's much worse than the Wild West out there, man. Any tool that reduces the drudgery and allows for quicker doing of the annoying parts is a net positive.
I disagree with the fact that it is all that. Usually, what I see from people doing CMake wrong is downright malpractice. Something like "I can't be arsed to read about it for 30min, so I'll just trial-and-error and copy-paste stuff without thinking for 4h until it works, and then I'll say it's not my fault if it sucks.
> There are thousands of tools and methodologies screaming for our attention to "be arsed to learn to do them properly".
That's bad faith. I learn to use all my tools properly. The more tools you know, the easier it gets to learn new ones. Also if you learn how they work, you can usually operate at a lower level, which means you need fewer tools (because lower level tools are more powerful). And there aren't new tools coming out every month. I mostly never have to learn new tools nowadays (and no, I'm not working in a super niche domain in C with one old framework).
> You're likely living and working in a bubble.
No. I feel the pain of most developers making a mess. The fact is that I can take a messy CMake setup, fix it, and then it's easy to use and it works.
But most developers don't actually know the two commands needed to build a CMake project. Tell them to set an option with `-D` and they will start crying.
I'll equate this to bash. I really wanted to learn it well and properly. I did, and I'd venture to say that happened at least 7 times (that I can remember of) and I forgot it each time because I don't use it often. I need it once or twice a month for a useful script and then it fades from memory. Sadface but those are the realities of the human brain (and I never wanted to resort to spaced repetition for bash in particular, though thinking of it now it was probably a much better idea than the other methods I tried for remembering all its quirks).
Ultimately I concluded that if something is quirky and needs you to put a special effort to remember how not to shoot yourself in the foot, then it's a bad tool. That's why I also abandoned `make`, learned `just` in literal 15 minutes and never looked back since.
After 23 years of a professional career and about 30 in total with computers I simply demand more quality of my tools:
- Have one and only one way to do thing X (also goes for programming languages though that's more difficult to achieve and not such a deal-breaker as it is with CLI / GUI tools);
- Make your most-commonly-used scenarios the easiest to do and the most trivial to remember;
- Make your tool and its most desired features easy to discover (via detailed help commands is one way, but there is also the "did you mean X?" etc.)
I am not taking jabs at CMake btw, I know exactly nothing about it. I am telling you that if I had to regularly do what I did with bash and make I'd ultimately conclude that it's a bad tool and will look to work around it... or I'll make a ton of shell aliases doing what I need from it and be very grumpy in the process (I mean: if I can't escape from CMake).
> That's bad faith. I learn to use all my tools properly.
OK, you got me, you are correct here. As admitted in the other comment, I am getting agitated by some of your comments because they seem to show lack of nuanced thinking. If I am wrong then I'd be happy.
> The more tools you know, the easier it gets to learn new ones.
...up to a point, after which you just get annoyed and want to scream "CAN'T YOU ALL JUST TALK TO EACH OTHER AND STANDARDIZE SOME STUFF ALREADY!".
I don't exaggerate when I say that I have learned no less than 200 CLI tools in the last 10-12 years, for many types of automating tasks (and even distributing them to clusters of machines), to wipe the arse of half-arsed niche databases some overly enthusiastic startup founder figured he needs to use as a competitive advantage (while he was on speed and LSD), to do data analysis for my own goals, to experiment with various task runners and build systems, and... yeah, maybe I should write some blog posts about it, it'll become too big.
My point is that what you say has an easily achievable peak after which you get just sick of everyone's crap. Just saying. Maybe you haven't reached that point yet.
> But most developers don't actually know the two commands needed to build a CMake project. Tell them to set an option with `-D` and they will start crying.
Now you are arguing in bad faith. :P I don't think you or I here should criticize those mystical "almost not devs" people just so we can feel smart. Using `-D` is something that every normal junior that I've coached will use and be grateful that I have showed them this "clever hack" (their words and they honestly made me laugh).
And if we try to be fair to all sides, many programmers want to, you know, only do programming, not learn a plethora of other tools that shouldn't have relevance to their work as much as they do. That's one of the reasons I gave up on C/C++ almost 20 years ago; I was doing much more effort making sure stuff compiles on several UNIX variants than actually solving any problems with code; the ratio was so bad that it was something like 70% to 30% on a good day; on some days it was 95% to 5% even.
It's all about striking a balance. Your comments come across as "git gud scrub!" to me and I can't ever let that go. It's demeaning and it hints at suffering from the main character syndrome (where you believe all others are stupid NPCs). And I am here to tell you that there are VERY GOOD REASONS why are you observing what you are observing and they are very rarely what you think they are.
Because it's easy to criticise CMake/Automake: we love to hate them. But in my experience, even though there are tons of valid points to make, many people who say "it's just because CMake sucks" are actually just making a mess and blaming CMake.
> Have one and only one way to do thing X
Agreed. To add some nuance though, most projects are perfectly fine by just using a minimal set of features in CMake: just use `find_package` and install the dependency on the system, or locally and learn how to find it with CMAKE_PREFIX_PATH and pkgconfig. I would even say that writing a custom CMake command is almost always a bad idea.
But to me it doesn't mean that you need to have one tool that does everything (and that also makes coffee). To me it's a bad idea to configure an entire CMake project to be run by gradle for Android. Mixing build systems makes matter worse, not better.
> Make your most-commonly-used scenarios the easiest to do and the most trivial to remember;
Sure, but the risk there is that many people hide the lower level because they try to "keep it simple stupid". Which IMO is wrong. Expose the lower level, but clearly do it in... a lower-level abstraction. So that advanced users can use the low-level, and others can use the higher-level. And usually it means: start with the low-level API. If it's good, maybe someone else will write an abstraction on top of it!
I am often frustrated because projects put a lot of effort into hiding lower-level APIs and then I'm stuck (and need to e.g. fork).
> That's why I also abandoned `make`
Note that I don't write projects with make. But it has merit, and I learned to understand/maintain projects based on it. I don't learn all the build systems under the sun, but make, CMake and meson (in the C++ ecosystem) seem valuable to at least understand.
The risk with introducing yet another tool like "just" is that you make "your" problem worse: it introduces a new tool. I am usually fine learning new tools, but in my projects I try not to introduce them: I use CMake or Meson because it makes sense to expect that C++ devs should understand them.
I cannot think of a single project where NIH syndrome would've been a net positive. Even dependencies that aren't "essential" are super helpful in saving time.
When you recreate parts of an "optional" dependency for every single project, how do you find the time to fix all the extra bugs, the edge cases, the support for different versions of lower level dependencies, of different platforms?
Nobody, and I mean 0, chooses to completely re-invent the wheel in every project. I understand why you would find this weird. I don't start every project with nand gates.
But the tendency is to use a library to check if a number is even or odd. It is a gradient, and you have to choose what you write and when you rely on a dependency. If you need to parse an XML file, or run HTTP requests, most likely you should find a library for it. But personally, if I can write it in 1-2 days, I always do it instead of adding a dependency. And I'm pretty sure it's worth it.
> how do you find the time to fix all the extra bugs, the edge cases, the support for different versions of lower level dependencies, of different platforms
If you take e.g. C++, maintaining your dependencies takes time. If you support multiple platforms, probably you need to (cross-)compile the dependencies. You should probably update them too (or at least check for security vulnerabilities), and updating dependencies brings its lot of issues (also in Rust: most libraries are 0.x.x and can break the API whenever they please).
Then, if you miss a feature in the library, you're pretty much screwed: learning the codebase to contribute your feature may be a lot of work, and it may not be merged. Same for a bug (debugging a codebase that is not yours is harder). If you end up forking the dependency and having to live with a codebase you did not write, most likely it's better to write it yourself.
The people advocating the "not re-inventing the wheel because NIH is really bad" philosophy seem to assume that libraries are always good. Now if you have a team of good developers, it may well be that the code they write is better than the average library out there. So if they write their features themselves, you end up with less but better code to maintain, and the people who understand that code work for you. Doesn't that sound good?
Woah, that's a lot of days.
They might save time up front, but over the lifetime of a long-lived project, my experience is dependencies end up costing more time. Dependencies are a quintessential but often overlooked form of tech debt.
If you only work on short-lived projects or build-it-and-move-on type contract work where speed matters more than quality, sure, go nuts with dependencies. But if you care about the long term maintainability of a project, actively reducing dependencies as close to zero as possible is one of the best things you can do.
Would you reimplement a regex engine yourself? I hope not. Left-pad? Obviously yes. I don't think you can have blanket advice that dependencies should be avoided.
I suspect even quite simple dependencies are still a time saver in the long run. Probably people prefer reimplementing them because writing code is much more fun than updating dependencies, even if it actually takes longer and isn't as good.
> Would you reimplement a regex engine yourself? I hope not.
I've implemented parallel-parsing-processes-style [1] parser combinators in at least three languages, though.
[1] https://www.cambridge.org/core/services/aop-cambridge-core/c...
It's different.
No point reacting to something that is almost disinformation.
No, they haven't been stable, not really. The TIOCGWINSZ ioctl has never been standardized to my knowledge, and it has many different names on different Unixes and BSDs. The tcgetwinsize() function only got in POSIX in 2024, and this whole thing has really sad history, honestly [0], and that's before we even get to the Windows side of things.
...but the libraries suck even more! If you don't want to link against ncurses then may God have mercy on your soul.
Sometimes - that's fine.
Sometimes - that's making his software worse for folks who have different use-cases, or are running on systems he doesn't understand or use himself.
The real value of a library, even with all those dependencies (and to be clear, I disagree that 3 or 4 dependencies for a library that runs across windows/linux is "all that many", esp when his platform specific implementation still uses at least 1), is that it turns out even relatively simple problems have a wealth of complexity to them. The person who's job it is to write that library is going to be more experienced in the subject domain than you (at least in the good cases) and they can deal with it. Most importantly - they can deal with your unknown, unknowns. The places you don't even have the experience to know you're missing information.
OTOH, if minimizing dependencies is important for a very specific project, then the extra work of implementing the functionality falls on that project. It will be simpler because it must only support one project. It may not receive critical compatibility updates or security updates also.
It does not fall on the community to go and remove dependencies from all their battle tested crates that are in common use. I think anyone and everyone would choose a crate with fewer over more dependencies. So, go make them?
I already mentioned this on twitter but not a lot of people work this way. I don't have to point you farther than my sha1-smol crate. It was originally published under the sha1 name and was the one that the entire ecosystem used. As rust-crypto became more popular there were demands that the name was used for rust-crypto instead.
I have given up the name, moved the crate to sha1-smol. It has decent downloads, but it only has 40 dependents vs. >600 for sha1. Data would indicate that people don't really care all that much about it.
(Or the sha1 crate is that much better than sha1-smol, but I'm not sure if people actually end up noticing the minor performance improvements in practice)
Isn’t sha1 a several hundred line function? What’s going on?
cfg-if - a tiny library making it easier to use different implementations on different platforms. Maintained by the official rust-lang libs team. Also used by the rust std library. No recursive dependencies.
cpufeatures - a tiny library for detecting CPU features maintained by the rust-crypto people, with a recursive dependency only on libc (which is maintained by rust-lang and used by the standard library).
digest - a library providing traits abstracting over different hash functions, maintained by the rust-crypto-people. In turn digest depends blockbuffer (some code to minimize boundchecks) and cryptocommon (more traits for abstracting), both maintained by the rust-crypto people. Both in turn depend on
- typenum, a third party library for compile time math, no dependencies.
- generic-array, a third library for fixed length arrays with lengths computed by typenum, typenum is its only dependency.
- version_check, a third party library for checking what features the rustc being used supports (only block-buffer depends on this one).
sha1-asm - A library with a faster assembly implementation of sha1, maintained by the rust-crypto people. Dependent only on cc (c compiler support), maintained by rust-lang, used in building the rust compiler. cc is itself in turn dependent only on
- shlex, a third party library for splitting things the same way a shell does. So this tree ends in an actual third party dependency, but one you're already indirectly dependent on by simply using the rust compiler at all.
Oh, and in the next release
sha1-asm is being removed
The crypto common dependencies are being replaced with hybrid-array, which is a replacement for generic-array maintained by the rust-crypto people, and is only dependent on typenum.
The version_check dependency is being dropped.
---
So what's going on is
1. This is a faster implementation of sha1, using all the tricks to have different implementations on different platforms. This resulted in 2 third party dependencies, down to 1 in the next release.
2. This is fitting into a trait ecosystem, and is using tricks to abstract over them without degrading performance. This resulted in 2 third party dependencies, down to 1 in the next release.
3. The count of non-third party dependencies is inflated because rust-crypto has split up its code into multiple crates, and the rust-lang team has split its code up into multiple crates.
4. It's not 40, it's 9.
Is it worth the extra code? I guess that depends on how many times you are hashing things and how much you care about the performance and environmental impact of extra code. Or if you're doing something where you want to be generic over hash functions.
Just like a bunch of others, you understood their comment backwards. Dependents, not dependencies. 40 other things rely on it.
I simply decided not to point out the source of their misunderstanding since it had already been pointed out by the time I replied, and this comment is already way too long.
(I am likely being completely naive, but I am well satisfied by Julia’s stdlib function for this, so wondering what makes Rust different since I’ve been doing more and more rust lately and am trying to understand what the difference might be).
It has zero dependencies, but only 40 crates on the eco system depend on it. Compared to sha1 which has 10 dependencies, but > 660 crates depend on it.
This is a major part of the thesis of no dependencies. General code is bad code. It’s slow, branchy, complex, filled with mutexes, nan checks, etc. Read “the old new thing”, to see the extreme
When you have a concrete goal you can apply assumptions that simplify the problem space.
A good example of this was Casey’s work on a fast terminal. At first all the naysaying was “production terminals are really hard because you have to handle fonts and internationalization, accessibility, etc”. Indeed those problems suck, but he used a general windows API to render a concrete representation of the char set on demand, and the the rest was simple.
For whom?
I think most times, as a user of software, I almost always prefer to have something that solves my problem, even if it's got some rough edges or warts. That's what general code is - stuff that solves a problem for lots of people.
Would I prefer a tool that solves exactly my problem in the best way possible? Yeah, sure. Do I want to pay what that costs in money, time or attention? Usually no. The general purpose tool is plenty good enough to solve the problem now and let me move on.
The value I get from solving the problem isn't really tied to how optimally I solve the problem. If I can buy a single hammer that drives all the nails I need today - that's a BETTER solution for me than spending 10 hours speccing out the ideal hammer for each nail and each hand that might hold it, much less paying for them all.
I'll have already finished if I just pick the general purpose hammer, getting my job done and providing value.
---
So to your terminal example - I think you're genuinely arguing for more general code here.
There's performance in making a terminal run at 6k fps. It's an art. It's clearly a skill and I can respect it. Sounds like it's an edge case that dude wants, so I'm in favor of trying to make the terminal faster (and more general).
But... I also don't give a flying fuck for anything I do. Printing 1gb of text to the terminal is useless to me from a value perspective (it's nearly 1000 full length novels of text, I can't read that much in a year if it was all I did, so going from 5 minutes to 5 seconds is almost meaningless to me).
The sum total of the value I see from that change is "maybe once or twice a year when I cat a long file by mistake, I don't have to hit ctrl-c".
I also genuinely fail to understand how this guy gets meaningful value from printing 1gb of text to a terminal that quickly either... even the fastest of speed readers are still going to be SO MANY orders of magnitude slower to process that, and anything else he might want to do with that text is already plenty fast - copying it to a new file? already fast. Searching it? fast. Deleting it? fast. Editing it? fast.
So... I won't make any comment on why this case is slow or the discussion around it (I haven't read it, it sounds like it could be faster, and they made a lot of excuses not to solve his specific edge case). All I'll say is your argument sure sounds like adding an edge case that nearly no one has, there-by making the terminal more general.
Any terminal I wrote for myself sure as fuck wouldn't be as fast as that because I don't have the rendering experience he has, and my use case doesn't need it at all.
For users and programmers. Slower to use. Harder to read and maintain. More code to do the same tasks.
> I almost always prefer to have something that solves my problem
I didn’t say anything about losing features. I don’t read Arabic. I’m glad the OS supports it. But it would be bad if Arabic complexity had to leak into every line of code. Limit the complexity scope. Casey’s terminal supports Arabic by following this principle.
> The value I get from solving the problem isn't really tied to how optimally I solve the problem.
Yes it is. There are ok products and there are great products.
> text still uses general solution
Operating systems have to solve general problems which is why they are expensive and brittle. Every program is not an operating system.
> There's performance in making a terminal run at 6k fps. It's an art.
I think you are missing context about the issue that led to this. And yes, as soon as there are no spinning wheels while doing normal tasks, then we can stop complaining about performance. But that’s not the situation.
> and my use case doesn't need it at all.
Would the world be better or worse if your operating system respected your time more?
This is a non-sequitur.
It's like saying that my Honda CRV is disrespecting my time because it has a top speed of 110, when it could be a 230 mph F1 racecar - it's a bad argument. I drive on roads with a max speed limit of 75... and most are down near 25mph.
> There are ok products and there are great products.
And users don't give a flying fuck about which yours is if they can't use it in the first place. That's why general things exist.
If this particular API has been stable, or at least reasonably defined for 50 or 25 years is a detail, because the dependency doesn't even pretend to deal with that and the function is unlikely to change or be removed in the near future.
Well, it hasn't. The tcgetwinsize() was proposed (under this name) in 2017 and was standardized only in 2024. So it's less than a 10 year old API, which is missing from lots of libc implementations, see e.g. [0]. Before its appearance, you had to mess with doing ioctl's and hoping your libc has exposed the TIOCGWINSZ constant (which glibc by default didn't).
[0] https://www.gnu.org/software/gnulib/manual/html_node/tcgetwi...
I had written nearly all of the PHP from scratch. I wrote libraries for authentication/authorization, templating, form processing etc. I used one PEAR library for sending email. The frontend was vanilla HTML and there was barely any JavaScript to speak of. We used Flash for media playback. In other words, myself and my small team built nearly all of it ourselves. This was just how you did most things in 2006.
It only took me about an hour to get the 19-year old app up and running. I had to update the old PHP mysql drivers to mysqli, and update the database schema and some queries to work in MySQL 8 (mostly wrapping now-reserved words with backticks and adjusting column defaults which are now more strict). The only thing that didn't work was the Flash.
An hour to revive an app from 2006. Contrast this with my day job, wherein we run scores of Spring Boot apps written in Java 8 that have pages of vulnerabilities from tens of dozens of dependencies, which are not easy to update because updating one library necessitates updating many other libraries, and oh my goodness, the transitive dependencies. It's a nightmare, and because of this we only do the bare minimum of work to update the most critical vulnerabilities. There's no real plan to update everything because it's just too tall of an order.
And the funny thing is, if you compare what this PHP app from 2006 did, which had truly, barely any dependencies, to what these Spring Boot apps do, there is not a lot of difference. At the end of the day, it's all CRUD, with a lot more enterprise dressing and tooling around it.
This reads like the dev equivalent of manager speak.
Take for example standard communication message formats like FHIR or HL7. You definitely don't want to implement the whole definitions for the standard, which is already complicated.
Writing Cryptographic functions by yourself is also typically a shot in your foot, has proved in all these years of found critical security issues.
We live in a time where you want to actual solve a business problem, by focusing on the problem and not on how the solution is built properly. With the advent of AI this is even more critical, since all the code feels like stitched together blindly.
Spending time on developing all by yourself might give you a good shot in the long run, but first you need to survive the competition, who maybe has already caught the market, by using fast and throw-away code at the beginning.
(I’m not affiliated with it; just curious about strategies for upgrading and maintaining apps that use big frameworks.)
At my job we have a fairly strict static analysis policy and starting in April it is going to get even more strict.
Have you looked at https://docs.openrewrite.org/ to automatically upgrade your dependencies?
I just migrated from Java 8, Spring Boot2 and Swagger to Java 17, Spring Boot 3.3 and OpenApi 3. It was pretty painless.
Now, I still have update some dependencies and transient dependencies but the biggest hurdles were taken care of by the migrations.
Even though NodeJS is largely responsible for my career, NPM has given me more trauma than my messed up childhood.
Imagine you're a new programmer, you're working on a brand new app to show to all your friends. But you want to add a new dependency, it doesn't like all the other dependencies, cool you say I'll just update them. Next thing you know absolutely nothing works, Babel is screaming at you.
No worries, you'll figure something out. Next thing you know you're staring at open git issues where basic things literally don't work. Expo for example has an open issue where a default new react native project just won't build for Android .
It's like no one cares half the time, and in that case the solution isn't even in the node ecosystem, it's somewhere in the Android ecosystem. It's duct tape all the way down. But this can also inspire confidence if a billion dollar project can ship non-functional templates, then why do I have imposter syndrome when my side projects don't work half the time!
I started to view it has a personal failure when I brought in that first dependency, which meant needing all of the packaging and organization just to use that first package.json, rather than just having plain js loaded from a script tag.
Even though it's just for my personal consumption, and I doubt more than five or six people will ever see it, I ended up having to use NodeJS and HTML because it's simply the best solution.
Imagine an alternate timeline where Google decided to offer other languages as first class citizens in the browser. Could you imagine Golang or Dart. No compiling down to JS, just running natively.
I'm doing backend work mostly right now. If I start some other project, I'll probably try using htmx and tailwindcss as my two js script tags, and the rest will be controlled from the backend.
WASM is neat though. Tends to ship with mega blobs for now
https://web.archive.org/web/20150328124454/https://news.dart... (yes, they took it down so Wayback to the rescue)
The GC barrier overhead is too damn high!
Your simple to learn programming language is what took me from a broke college dropout to six figures in about 3 years.
It's the first programming language that actually stuck with me, even though one of the senior developers at my first salaried job actively hated it. He was an old Java guy though.
Do you feel that JavaScript is always a capable tool for the job?
At a lot of companies, at least when I started programming 10 or 12 years ago, we'd have a bunch of front-end people who are nifty with JavaScript and just tell them to write NodeJS server code.
JavaScript is still my go to language when I just need to get something done. With Python a close second. C# has it's place for larger projects ( as well as stable employment).
I'm actually a bit excited for WASM. For a good while you had companies shipping plugins to extend the browser, first Adobe with flash, and then unity with their Unity Web plugin. Which lasted about 2 years before they started building native web builds.
Now everything is compiling straight to JS or JS+WASM.
JS is not the hammer for all nails, but ubiquity means it will be used to pound in screws and bolts more than it ought to be.
I fire up Node but also Python from command line; Wolfram Alpha in browser.
Wasm was the victor in old plugin vs. integrated PNaCl style VM vs. polyglot VM battles. JS VMs (ubiquitous again, and see write barrier cost of poly-GC vs. mono-GC down the page in my linked comment) rather more easily became polyglot via WebAssembly as second syntax low enough to target from other languages' compilers.
Still more to do on the Wasm roadmap. Always more to do...
"Dedicated to the memory of William Schelter."
Or does WASM make this unnecessary?
I actually really like Flutter Web, Google has done an amazing job with it. I built a small game for a friend, but it's not as customizable as an actual website coded in JavaScript.
It appears they've taken a hybrid approach where you get both a JavaScript and a Wasm build.
https://docs.flutter.dev/platform-integration/web/wasm
I feel like Unity in particular has really pushed the limits of what you can do in a browser. For example, I have a simple 3D game that performs just as well as the android build.
As far as making a small game, it's very easy to get someone to open a link. The browser sandboxes every and protects you.
It's near impossible to convince someone to download a binary, with good reason.
One last question. Do you consider yourself to be a Computer Science pioneer ? Your literally responsible for the foundations of the modern web!
MS throwing money at C# native VM (the CLR? Silverlight's DLR? lol) as second VM in browsers, with separate GCs bridged, incurring the write barrier overhead that I cited already? In all browsers, not just the ones MS can bribe to do the considerable work? No.
C# compiled to WebAssembly? One GC, all good, perhaps some warm evolution is on order, but it is an incremental cost.
As noted in this post, even a small Rust project will likely have many more than that, and it's virtually guaranteed if you're doing async stuff.
No idea how much of it is cultural versus based on the language features, e.g. in Go interfaces are implicitly satisfied, no need to import anything to say you implement it.
In Go: https://pkg.go.dev/log
Honestly, something feels off with Rust trying to advertise itself for embedded: no stdlib and encourage stack allocation, but then married to Clang (which doesn't have a good embedded target support) and have panic in the language.
Building a C++ replacement for a browser engine rewrite and building a C replacement for embedded have different and often conflicting design constraints. It seems like Rust is a C++ replacement with extra unnecessary constraints of a C replacement.
To me Rust is good when you need the performance (e.g. computer vision) and when you don't want a garbage collector (e.g. embedded). So really, a replacement for C/C++. Even though it takes time because C/C++ have a ton of libraries that may not have been ported to Rust (yet).
Anyway, I guess my point is that Rust should focus on the problem it solves. Sometimes I feel like people try to make it sound like a competitor to those other memory-safe languages and... even though I like Rust as a language, it's much easier to write Go, Swift or Kotlin than Rust (IMHO).
While I respect your opinion, I must say I find it much more difficult to get work done in golang than rust. Mainly due to stringly typed errors and nonexistent documentation culture. Trying to figure out what error states my program might get itself into is essentially a futile exercise in golang whereas in rust it's generally annotated right there in the source code by lsp.
Embedded systems maybe?
API Stability. When the standard library APIs were initially designed, "the Python standard library is where packages go to die" was very much on our minds. We specifically saw ourselves as enabled to have a small standard library because of tooling like Cargo.
There are no plans for Rust 2.0. So any change we merge into std is, effectively, something we have to live with for approximately forever. (With some pedantic exceptions over edition boundaries that don't change my overall point.)
Nuance is nearly dead on the Internet, but I'll say that I think designing robust and lasting APIs is a lot harder in Rust than it is in Go. Rust has a lot more expressiveness (which I do not cite as an unmitigated good), and the culture is more heavily focused on zero-overhead abstractions (which I similarly do not cite as an unmitigated good). That means the "right" API can be very difficult to find without evolution via breaking changes. But the standard library cannot, generally speaking, make breaking changes. Crates can.
I would suggest not reading the OP as a black-and-white position. But rather, a plea to change how we balance the pros and cons of dependencies in the Rust ecosystem.
We also came at the standard library with a cross platform mentality that included non-POSIX platforms like Windows. You want to be careful not to design APIs that are too specific to POSIX. So it falls under the same reasoning I explained above: everything that goes into std is treated as if it will be there forever. So when we add things to std, we absolutely consider whether the risk of us getting the API wrong is outweighed by the benefit of the API being in std in the first place. And we absolutely factor "the ease of using crates via Cargo" into this calculus.
Much like op said here; we have a culture of "don't write unsafe code under any circumstance", and we then pull in a dependency tree for a single function that's relatively safe to contain. It solves the problem quickly, but at a higher price.
BTW, thanks for ripgrep. I don't actually use it, but I've read through different portions of the code over recent months and it's some very clean and easy to understand code. Definitely a good influence.
- backward compatbility: a big std lib increases the risk of incompatible changes and the cost of long term support
- pushing developers to be mindful of minimal systems: a sort of unrelated example is how a lot of node library use the 'fs' module just because it is there creating a huge pain point for browser bundling. If the stdlib did not have a fs module this would happen a lot less
- a desire to let the community work it out and decide the best API/implementations before blessing a specific library as Standard.
In my opinion a dynamic set of curated library with significantly shorted backward compatibility guarantees is the best of both worlds.
- less burden on the stdlib maintainers (which are already overworked!)
- faster iteration on those libraries, since you don't need to wait a new release of the compiler to get updates for those libraries (which would take at least 12-16 weeks depending on when the PR is merged)
For example, Java developers can choose to represent time with Unix milliseconds, java.util.Date, java.util.Calendar, Joda-Time or java.time.Instant
In reality, there’s just one old API and one new API, similar to the old collection classes (HashTable, Vector, etc.) and the newer JCF ones.
This probably isn't as important as the stability concerns, but I think it still helps tilt the argument in favor of a small std at least a little.
IMO they should over time fold whatever ends up being the de-facto choice for things into the standard library. Otherwise this will forever be a barrier to entry, and a constant churn as ever new fashionable libraries to do the same basic thing pops up.
You don't need a dozen regex libraries, you just need one that's stable, widely used and likely to remain so.
> and a constant churn as ever new fashionable libraries
Isn't that the situation in Javascript? I don't work in Javascript but to me it feels like people migrate to a new cool framework every 2 months.
To be fair though, I'd argue the environment Javascript lives in also changes faster than any other.
That is the case today. Virtually everyone uses `regex`.
There are others, like `fancy-regex`. But those would still exist even if `regex` was in std. But then actually it would suck, because then `fancy-regex` can't share dependencies with `regex`, which it does today. And because of that, you get a much smoother migration experience where you know that if your regexes are valid with `regex`, they'll work the same way in `fancy-regex`.
A better example might be datetime handling, of which there are now 3 general purpose libraries one can reasonably choose. But it would have been an unmitigated disaster if we (I am on libs-api) had just added the first datetime library to std that arose in the ecosystem.
Things you can find in go’s built in to the language or in standard libraries that need a dependency in Rust:
- green threads
- channels
- regular expressions
- http client
- http server
- time
- command line flags
- a logger
- read and write animated GIFs
I don’t love the Go language, but it’s the leader for tooling and standard library, definitely the best I’ve used.
You could of course argue that that's why no-std exists in Rust, or that your compiler might optimize out the animated GIF routines, but personally, I'd argue that in this context, it is bloat, that - while it could occasionally be useful - it could just as easily be a third party library.
For a microcontroller sized runtime, there’s https://tinygo.org/
I think the lack of strong standard library actually leads to more bloat in your program in the long run. Bloat is needing to deal with an ecosystem that has 4 competing packages for time, ending up with all 4 installed because other libraries you need didn’t agree, and then you need ancillary compatibility packages for converting between the different time packages.
I don't think that's true. If the standard library is pre-compiled, and it doesn't use `-ffunction-sections` etc. then I'm pretty sure you'll just get the whole thing.
There is experimental support for building Rust's standard library from source, but by default it is pre-compiled.
It seems like a bad reason to constrain the regular standard library
E.g. for esp32 see https://docs.esp-rs.org/book/overview/using-the-standard-lib...
As a beginner this is horrible, because everybody knows serde, but I have to learn that serde is the defacto, and that is not easy because when coming from other languages, it sounds like the second best choice. And that is with most rust crates.
If the encapsulating package churns over its design, or changes it's goals it creates instability. Functionality is lost or broken apart when previously it was previously the simplest form of guarantee in software engineering in existence. It also deters niche but expert owners who aren't career OSS contributors from taking part in an ecosystem."I made a clean way to do X!" being followed up by weeks of discussion, negotiation, politics so that "X fits under Y because maybe people like Z" is inefficient and wasteful of everyones time.
If there's one thing I've learned in my life it's that the simplest things survive the longest. Miniliths, and monoliths should be celebrated way more often. Rust isn't alone in this by the way, I've seen this across languages. I've often seen OSS communities, drive hard for atomistic size packages, and I often wonder if it's mostly for flag planting and ownership transfer purposes than it is to benefit the community that actually uses these things.
I sort of stumbled upon this myself and am coming around to this viewpoint. Especially after dependency hell of a big react app.
And there's also the saas/3rd party services dependencies to consider. Many of them are common patterns and already solved problems that LLMs can clone quickly.
It also made it very easy to hack around with, and I got to the point where I understood the entire thing. I did couple it with Gevent for the server and websockets, but I was able to put together some heavy-lifting projects that way.
I still feel a strong impulse to use it for small web projects. Sadly, it didn't keep up with a lot of the more modern practices that Python has introduced over the years, so it does feel a bit dated now.
Another thing I noticed is, that one can often easily do better than existing libraries. In one project I implemented a parser for a markdown variant, that has some metadata at the top of the file. Of course I wrote a little grammar, not even a screen of code, and just like that, I had a parser. But I did not expect the badness of the frontend library parsing the same file. That one broke, when you had hyphens in metadata identifiers. At first I was confused, why it could not deal with that. Then it turned out, that it directly used the metadata identifiers as object member names ... Instead of using a simple JSON object, they had knowingly or unknowingly chosen to artificially limit the choice of names and to break things, when there are hyphens like in "something-something". In the end my parser was abandoned, people arguing, that they would have to "maintain" it. Well, it just worked and could easily be adapted for grammar changes. There was nothing difficult to understand about it either, if you had just a little knowledge about parsers. Sounds incredible, but apparently no one except me on the team had written a parser by using a parser generator library before.
And like that, there are many other examples.
About a year ago I ran a project to update 3rd party dependencies.
One of the dependencies was a rich math library, full of all kinds of mathematical functions.
I did a little bit of digging, and we were only using one single method, to find the median of a list.
I pointed the engineer to the Wikipedia page and told him to eliminate the dependency and write a single method to perform the mathematical operation.
---
But, IMO, the real issue isn't using 3rd party dependencies: It's that we need a concept of pulling in a narrow slice of a library. If I just need a small part of a giant library, why do I have to pull in the whole thing? I think I've heard someone propose "microframeworks" as a way to do this.
On one side, I think if we had a good system of trust, that's not a problem.
And part of me likes the idea of something like Shadcn - you like a component? Copy it into your library. However, if there ends up being a vulnerability, you have no idea if you are affected.
For some code, that's not a problem. For other code, we truly depend on having as many eyes as possible on it.
Median(list) {
let len = length(list)
if len % 2 == 0 {
let x = floor(len/2)
return (list[x] + list[x+1]) / 2
}
return list[len/2]
}
Managed to resist calling is_odd there!
But if I have a big enough list that I care about space usage (I don't want to make a copy that I sort and then throw away), or speed (I care about O(n) vs O(n log(n))) I'd be looking for a library before implementing my own.
Here are the relevant algorithms if you really want to implement your own fast median code though: https://cs.stackexchange.com/questions/1914/find-median-of-u...
Right, if it's already sorted just taking the midpoint is the obviously correct algorithm (and O(1) time/space). It's only in the unsorted cases where with giant lists you should start thinking about alternatives.
If I'm working with gigabytes of photon counts (each element representing the number photons detected in a time interval) I don't want to sort my gigabyte long list before getting the median - sorting would destroy the very important structure of the data so I'd just have to throw away the copy afterwards. This is referencing some code I worked on a long time ago. I'm not sure I had to calculate a median specifically, but similar enough statistics. It's a simple function, but not a one size fits all algorithm.
Unfortunately, it doesn't seem like people are super diligent about looking into the default feature set that's used by their dependencies and proactively trimming that down. It doesn't help that the syntax for pulling in extra features is less verbose than removing optional but default ones (which requires both specifying "no-default-features" and then manually adding every one of the default features that you do still want back to the list of ones you pull in), and it _really_ doesn't help that the only way for libraries to expose the ability to prune unneeded features from their own dependencies to the users who inherit them is by manually making their own feature that maps to the features of every single one of their own dependencies. For example, if you're writing a library with five dependencies, and every one of them has one required dependency and four optional ones, giving the users of your library full control over what transitive features they pull in would mean making 20 features in your own library mapping to each of those transitive features, and that's not even counting the ones that you'd want to make for your own code in order to be a good citizen and not force downstream users to include all of your own code.
More and more I'm coming to the opinion that the ergonomics around features being so much worse for trying to cut down on the bloat is actually the catalyst for a lot of the issues around compile times in Rust. It doesn't seem to be super widely discussed when things like this come up though, so maybe it's time that I try to write a blog post or something with my strong feelings on this so at least I'll have something to point to assuming the status quo continues indefinitely.
[1]: https://github.com/tokio-rs/tokio/blob/ee19b0ed7371b069112b9...
We need to know if a dependency is level 1 or level 2 or level 3 or 45. And we need to know what the deepest dependency on our project is. I might be naive, but I think we should strive to reduce the depth of the dependency graph and maybe aim for like 4 or 5 layers deep for a web app, tops.
I think it would reign in the bloat.
Although a ten year old dependency is written in Python, it's likely not going to work any more due to changes in the build system, stdlib, etc.
So they start off with an open source Free solution, and then later switch to a paid for model, and abandon the Free version. This is particularly painful when it's a part of your system that's key to security.
You're left between wondering if you should just pay the ransom or switching to a different solution entirely, or gamble and leaving it on an old unpatched version.
( Looking at you, IdentityServer )
Either way you regret ever going with them.
You know you can hire someone to support them right?
It’s really nice when someone else has created all the abstractions for you. It’s fairly easy and fun to discover features of crates thanks to the type system.
But to actually build out those low level abstractions yourself is cumbersome and not very fun. It’s easy to type system yourself in a corner.
You even lose the whole big selling point of rust when you use `unsafe`.
There was an article a while ago comparing zig and rust and it had a good quote that went something like
“Rust is for programming in the large, and zig is for programming in the small”
And I think what you’re looking for is programming in the small.
Small, focused, programs with minimal dependencies.
Rust should not be a replacement for e.g. Go, Swift or Java, which are a lot easier to write and have memory safety.
I wonder who actually needs Rust (as opposed to using e.g. Go) and finds it a lot harder to write than C/C++?
Powerful, but complex, language features.
C is drop dead simple.
* Bad abstractions which just stick around forever. There are some examples of this in UNIX which would never be invented in the way they are today but nonetheless aren't going anywhere (e.g. signal handling). This isn't good.
* Invent all of your own wheels. This isn't good either.
There's a balance that needs to be struck between all of these 3 extremes.
We can't remove the old way of course, as that would break things, but that doesn't stop improvements
That way you break a lot of things.
It's doable, but nobody wants to put in the money, time and energy into pioneering it.
It's true that dependency-free software is very rare these days. The most obvious reason is that people don't want to "reinvent the wheel" when doing something. While this is a 100% valid reason, sometimes people simply forget what they are building and for whom. Extensive usage of dependencies is just one of the forms of overengineering. Some engineering teams even do their planning and features because of the new shiny thing.
The problem of dependencies is massive these days, and most companies are focusing on producing more and more code instead of helping people manage what they already have.
You can subscribe by clicking on Watch -> Custom -> Releases.
As for smaller QOL dependencies, I might as well use them directly as I need them, instead of spending 3h building something similar but half as good and robust, just to decrement my dependencies counter by 1.
I've had very few issues with Rust where an old project can't be built anymore because of rotting dependencies, when it happens almost every time with JS or Python. This is my largest gripe with dependencies.
I do still think Rust would have been better off with a larger stdlib, but ultimately it's really not that big of a deal and isn't worth it rewriting code for the sake of reducing a meter.
But I use a lot of them; just ones that I wrote.
Almost all of my publicly-available work consists of Swift modules (SPM), with very few stars or forks. Not very popular.
Which suits me just fine. I write for an audience of 1. I publish my work, because I believe that this forces me to do a really good job on it. I don't like to worry about my dependencies. If I wrote them, and they follow my Quality bar, then they're fine.
I think, in all the work I do, I have maybe only two or three dependencies, outside of ones that I wrote, or that are provided by the development system (I write native, so there's no giant ball of mud hybrid library).
It does limit the scope of what I can do, but not really that much.
I’ve seen the exact opposite — every dependency introduces a new opportunity for vulnerabilities, and to be judicious in pulling in new ones due to the potential security risk.
One of the reasons I wrote https://crates.io/crates/self_cell/ was to avoid the dependency trees pulled in by proc macros like ouroboros.
— Joel Spolsky, In Defense of Not-Invented-Here Syndrome: <https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...>
https://www.mikeperham.com/2016/02/09/kill-your-dependencies...
Mike Perham also talks about a dynamic language in an ecosystem that was just as obsessed with left-pad-like lunacies like the Node.JS ecosystem still is.
The sentiment is not wrong but doesn't apply to Rust. Be honest: would you roll your own HTTP and WebSockets server from scratch? Also how likely do you think it is that a business will allow you this huge downtime during which no moneymaking activities will be performed?
The article shows it's a cultural problem, not a technical. The normal case should be that implementations come in all sizes of complexity on the market, and the end user can choose freely; and I consider this a healthy state of affairs. Rust and JS are a monoculture that has fallen prey to some mind virus where only big supply chain is valued, and so the market becomes distorted. Tell me whether I'm wrong in my assessment.
Is managing your software supply chain more work than writing it yourself? Probably not if there are thousands or millions of LOC involved. But if you have infinite money, go ahead and build everything from scratch..
Pulling a dependency with 1M LoC is probably more costly than writing 1000 LoC yourself.
From what I can tell the research tends to agree with me:
-https://www.researchgate.net/publication/221555430_An_empirical_study_of_build_maintenance_effort
- https://arxiv.org/abs/1710.04936
I would love to see other research if anyone has some... the quick ~5 studies I checked didn't seem to answer the question 100% but are good enough to confirm my thoughts.
It means I can keep my dependency spec broad, because I actually test my releases with minimum supported versions. Cargo has (or maybe had -- I can't find it any more) a similar feature built in.
Renovate will even broaden (rather than replace) dependency specifications, if asked, so if my package works with both the old and the new version then I can make a patch release with the wider dependency spec and not break anything.
(On which note: I can really highly recommend using Mend Renovate to keep dependencies up to date: https://www.mend.io/renovate/ )
"Every dependency is an asset. Every dependency is a liability."
It's a balance.
Well, assuming you wrote it perfectly and didn't introduce any security vulnerabilities... that you will never be alerted to, because no one else is reviewing your code.
> security vulnerabilities
Does this code parse untrusted data? Does your process have unlimited access to sensitive resources?
I don't understand the question. Yes, I trust the global community of software developers to write a parsing library with sanitization more than I trust myself + my one or two work colleagues.
Rich Hickey has been discussing this very early, but almost nobody got it when he said it, myself included. Striving for obviously simple interfaces, that avoid nominal static types like the plague, is the way to avoid this often pointless waves of changes through language ecosystem, just because someone designed a better option monad or whatever.
I don’t think that’s entirely true. Unless you’re a star programmer you’ll find bugs and edge cases in your own code. So I end up updating it a lot. Not a dependency but also not write once
Often the most popular package for X is actually low quality and/or obviously incorrect, especially for small-to-medium complexity topics. There may be better alternatives but they’re hard to find due to low usage numbers or poor SEO, so I usually need to build stuff myself out of necessity.
For example, the most popular sql`…` tagged template literal package available on NPM isn’t type safe, doesn’t understand differences in SQL dialects, can’t escape queries for debugging or manual execution, doesn’t have console.log formatter, lacks utilities like .join or binding helpers, etc. it’s an anemic package yet everyone uses it. Maybe there are better ones that are specific to a database like Postgres, but it’s a simple enough thing to build, plus if you build it yourself in the monorepo it’s super easy to improve things as they come up.
So, I stated with a little 300 line one for SQLite that’s grown over the years to have a nice Postgres dialect as well, an ecosystem of helpers and lint rules, and it seems far ahead of whatever in NPM.
Another area is ESlint rules. Often there’s an open source rule that does like 80% of the job, but the maintainers seem hostile or dislike typescript or something. So much easier to copy the rule into our repo and improve it there rather than trying to contribute back. Or, it’s the usual quality issue - original rule is doing some crazy slow shit and it’s much easier to write from scratch a simpler more correct version.
If you ship a library, you are responsible for its security. If you dynamically link to a library you don't ship, then it's not your problem anymore (someone else is shipping it). I think we tend to forget that: shared libraries have advantages in terms of security.
And whether you ship the dependency or not, you should be ready to replace it or maintain it yourself. Too many projects depend on a project they don't understand, and get screwed down the line because it becomes unmaintained. Many times it's actually better to write the code you need than to depend on a third-party you don't understand.
In what universe is that true? In this universe, you are now on the hook for the security of both libraries, and the extra one you don't actually have any influence over.
Nobody cares much about the security of their own code, and even less for the security of their dependencies. At least if you link dynamically, you give an opportunity to your distribution to have someone care about it.
> Is it a rule that anyone disagreing with you is ill-informed?
Obviously not, I am getting a bit agitated that nuance is lost during such rants, and I get even more annoyed when people are like "yeah, I agree!" with zero attention to nuance.
Again, the article is ranty and it is not even factually correct -- the terminal functionality does not have such huge amounts of churn as many other modern tech stacks have, happily, but it's not frozen in time as the OP makes it sound. And it's not "change for the sake of change" either. OP even recognizes this part (by saying that other pieces are moving around this mostly static piece) later on which makes his earlier rant even worse IMO.
Hence my annoyance with your and others "+1" comments. Again, any parallels to others, much more churn-ey, ecosystems like Node.JS, are arbitrary and unfair.
Sorry, where did I mention left-pad?
The only thing I say is: writing a project in C++, I can keep control and have a few dependencies. Doing the same thing in Rust explodes almost immediately: as soon as I add one dependency, the lock file is so big I stop giving a shit.
You can say I am wrong, but I'm just sharing a feeling.
Also I wasn't aware you should be "giving a shit" about a lock file. Nor why would that matter for anything at all.
The final program works and you didn't have to reinvent complex wheels (like terminal sizing in OP). I don't see the problem. I only see wins.
You should care about not shipping malware to your users, I don't know how to put it differently.
It's always a ROI and other analyses: stakeholders don't care much about malware in transitive dependencies, they want feature X within timeline Y.
Do I like it? No, I hate it with all my heart, but that's the reality we live in. I am not paid to just tinker with the computer; I am paid to deliver stuff, within various deadlines, and of certain quality and that's not the highest possible, otherwise I'd thoroughly use fuzzers and fiddle with property tests for months, if not years.
Perhaps I’m sheltered: is this really true in people’s experience?
Time to delivery is the main metric for almost any company that ships software.
That seems feasible to test automatically. Shouldn't an intelligent compiler automatically trim these things out?
The second developer gets all the glory.
Often, with the type of developer who insists on reinventing wheels because it's "easy", you get a pretty bad wheel, as it's the first such wheel the developer has built and their estimation of "easy" was built on a foundation of ignorance.
Which is why it doesn't belong in software - there is no software version of the wheel.
Sure, but I need a skateboard, not a train.
I don't see myself writing wgpu or tokio. And wgpu is a hefty price to pay in dependencies.
So yeah, dep's are a plague, but hey, we are writing super complex stuff nowadays.
And I don't care if my project needs 464 (yeah, you read it well) dependencies. I do my share by making sure I don't have 2 versions of the same lib and that's about it.
What I want to see is someone actually break down a real application and its dependencies and argue through what's included and what's omitted.
At least there's one example. I just think it's wrong (and I'm not the only one...https://news.ycombinator.com/item?id=42813003). The terminalsize crate has special behavior for illumos. The average developer should never have to think about illumos.
(Note in case Bryan Cantrill walks by: no slight towards illumos or you--you seem great, loved the Oracle rant. Illumos seems cool. It's just that most developers should not think about it).
If a project has 0 dependencies but uses the exact same code as a project with 100 dependencies which project is better?
They are the same. So dependencies are not inherently bad.
What if the dependencies also have another 100 lines of code that are not used in the project?
I would say that doesn't really matter. In some cases it might lead to a slower build time or even a slightly bigger binary.
The actual situation you will end up with is that you write your own code and it's nothing like the dependencies. This is great because you don't have to worry about the maintainer of the dependency wanting to add things you don't need or making mistakes. You only have to worry about you making mistakes or having made mistakes. Mistakes you will maybe never encounter and mistakes you will never find because most of your code will never be looked at again. The dependency you could have used did so you would have to deal with pesky updates that might not affect you.
You could also choose not to do any of the updates of the dependency. You just pin all dependencies to a specific version and never update. Whenever there is an issue you just pull in the dependency and fix the issue yourself. Now you've got the best of both worlds, you didn't have to do the work upfront of generating the code with AI and you still get to maintain everything yourself.
But we also know that using dependencies can be bad, experience has shown that. But why were those experiences bad. Leftpad was a problem of a package disappearing because it was not version pinned and the package manager allowed it to dissappear. This problem has been solved.
Then there is the issue of abandoned packages, version incompatibility or packages that move away from your use case. But how different is this from pinning all dependencies and maintaining them from that point.
The point that open source contributers and dependency users have made before is that once you download the code it's yours. You are always the maintainer of whatever code your project uses. This means you are responsible for maintaining the dependencies you use.
In the most basic sense you should start a fork for each dependency and start to maintain it. But that is an enormous task so not a solution. You can hope you picked correctly and the package will be maintained forever but you will surely pick some that will end up abandoned. Another solution could be to pay someone else to maintain the dependency or a fork of the dependency. If there are more projects that do this you can actually build on a stable basis where you can trust the dependencies are good. The number can be big but not bad. Otherwise you will have issues with whatever path you choose. Because fundamentally as a user of open source you are not a customer but a raccoon looking in a dumpster for code.
https://en.wikipedia.org/wiki/Sybil_attack#Economic_costs
It can also allow for maintainers of the RUSTSEC rating agency to perform some kind of basic check to see if there's any abuse.