Super Colliding Nix Stores: Nix Flakes for Millions of Developers
blog.replit.com
blog.replit.com
Not sure how if Nix/Replit has the program you’re looking for? Or what package name to use to actually get the program you want? Check out my cli search tool:
https://github.com/peterldowns/nix-search-cli
Edit: I can’t help myself from ranting, it’s insane that I had to write this tool. Speaks to the lack of user focus in the Nix community. Even the blog post here doesn’t mention how to actually find packages to install!
What good is a package manager that makes finding packages difficult? Who should be expected to tolerate such a thing?
if folks want a easy (and good way) to get up and running in the nix ecosystem w/services ala replace docker compose then https://devenv.sh is the shizz
Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc.
Many people don't use Nix for "installing" stuff. I personally use it for deploying software to servers without the (congnitive and performance) overhead of containers.
I find search.nixos.org immensely useful, but don't really need it from the CLI.
even tools for tools should be ergonomic to use. the Nix language server should have autocomplete, searches and interacting with the language should be easy, etc. and nearly everyone needs to hunt for packages, even if they're not "installing" them - they might be needed for the toolchain.
Nix is like early Git. all the raw power is there, but little of the porcelain. it's so much better than anything before it that we accept the pain of dealing with exposed plumbing, but we shouldn't get used to it.
There’s no venture backed cash here paying for the polish that folks typically see in devtools launched in 2023. That polish is expensive to do and isn’t “fun or engaging” for the majority of developers thus classic open source problem of over indexing on serving the needs/existing people who do not need the polish.
There’s some notable call-outs here of folks putting in serious effort such as https://nix.dev ala https://cachix.org out of love for nix to succeed.
Yeah nix wants to grow, and it is, largest bazaar GitHub community/repo out there right now.
starting to see venture backed companies forming who are building the polish as porcelain over the top and giving back. see https://floxdev.com/
at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community.
In the last few years, Nix has gone from a place where I was sometimes uncomfortable recommending it to people to one where I feel like it would be unstrategic not to use it where it fits. If you know Nix today, you should be pushing it at your company wherever you can see a good use case.
Super agree. Been using nix now for six(?) years and always been advocating for it but hesentiant to recommend companies adopt it (unless you can use it to attract talent as walmart did with nodejs back in the day) however with https://devenv.sh and https://nix.dev in the existence the time to amaze is now measured in < 5 minutes. nixpkgs also partially solves sbom topics, enables monkey patching any linux application/kernel/ffi dependency, has the freshest collection of packages (it's more fresh than archlinux by a major factor) whilst all being under CI/CD (shamefully rare in linux land)
Also nix does have some autocomplete, check out Nil…
also I agree that it's ridiculous nix doesn't have a search command. it baffles me.
ps: $ nix search exists via experimental flags https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3...
- you may want to search without downloading and indexing nixpkgs and the hydra results yourself
- you may want to search by the name of the binaries/programs that will be installed, which is impossible to do by looking through a local package store alone
- you may want the search to be fast
- you may want to search from the command line
`nix search` is a cruel joke which doesn't allow searching by the name/program that would be installed, only package name, and requires a flake name every time. Absolutely terrible interface.
My tool is a single-install binary that performs fast and accurate search to help you find the right package name to install a given binary. I don't understand how after years of using other package managers anyone could want a search tool that does anything other than this by default.
For more details on why this exists, check out https://github.com/peterldowns/nix-search-cli#motivation
For those who want a local search for executables, and don't mind running nix-index:
#!/bin/sh
results="$(nix-locate -1 --regex --top-level "^/bin/${1}\$")"
for item in $results; do
nix search "nixpkgs#${item%.*}"
done
We ended up building package version search into the Devbox CLI for this reason (https://www.jetpack.io/devbox/docs/guides/pinning_packages/#...). There's also a 3rd party site that lets you search for Nix package versions (https://lazamar.co.uk/nix-versions/)
The version thing is also really annoying, thanks for the helpful link, I hadn't seen that. Not sure how this will work in a Flakes world. Seems weird to me that sometimes packages are important enough to have multiple derivations each installing a different version (like the pythonFull packages) but there is no standard/good/default solution for it.
I'm 100% onboard with flakes in general, and use flakes to describe devshells + publish binaries when applicable -- like with the tool I linked in my earlier post.
I put a good amount of time getting to grips with "raw" nix with the I imagine common yo-yo-ing between "I don't get it" and "oh I see, this is great" but when I realised how the intersection of nixpkgs and package versioning actually worked.. I was done.
For a tool that from the outside seems is so heavily focused on immutability to just continually throw away old versions in nixpkgs head is a head scratcher, and as a monorepo isn't a great fit for utilising different revisions for different packages either.
I can only guess that due to single repo containing every package it wasn't seen as practical to just continually append versions to, but when the diff log is just full of 'delete version X URL and it's hash, add X+1 and new hash' from the outside at least, it felt like a real missed opportunity.
I've got 3 packages which are pinned to a specific version that way in home-manager and I'm happy. It's not an approach for the first time user of course.
For new projects though, I agree that using `latest` is generally the way to go.
But they currently have a bit of a discoverability issue, and they need adoption by legacy packages to completely fix the problem.
It is simply completely infeasible to store and build every version of every package.
While it is not user friendly to find a package’s version X, it is either just overriding the version field and the hash, or referencing another version of the nixpkgs tree and building the same package from there.
It’s not trivial with other package managers either, and on top you can easily get into inconsistent states with those — nix can handle any version of any software correctly.
But I do think there's a search issue where mapping the version field of a package to the commit hash or nixpkgs version is not trivial or user-friendly, especially for multiple packages.
The `text` package [2] has many major, minor, and fix versions. Many of these versions are still required by projects. However, the current Nix Packages only has one version [3], and maybe a second version under a different name.
Hackage.nix will provide any version of `text` published on Hackage.
One downside is that `text` is not pulled from Nix Packages. The project's specific version of `text` must be cached elsewhere, or built from source. So it is slower.
[1] https://github.com/input-output-hk/hackage.nix [2] https://hackage.haskell.org/package/text [3] https://search.nixos.org/packages?channel=22.11&show=haskell...
I don't understand this: Nixpkgs can only ever contain a tiny proportion of package versions. It's meant as a mostly-consistent set: e.g. if you want packages "foo", "bar" and "baz", that should work.
It's not intended for "foo-1.2", "bar-0.2" and "baz-99"; let alone toggling the super-duper compiler flag and applying Alice's stability patches. However, the latter are intended by the functions which define those packages; and it's exactly why so many "override" functions are provided (to do all of the above, we usually just need '.overrideAttrs').
If you want a consistent set of packages, the usual approach is to run some sort of dependency solver (e.g. many of the 'foo2nix' tools), and map over its result. For example, at work our Scala projects run 'mvn2nix' during their build; use import-from-derivation to turn the results into a local Maven repository; and use that to build the project in Maven's "offline mode".
There has been a trend in software dev towards solving problems that don’t need solved. I’ve never been confused about how to find packages for nix.
But you're right! Deeper investment, starting with actually performing your builds with Nix, yields greater benefits, and teams that never take that next step miss out on that.
We built a solution to the same problem with a similar approach[1], but that just snapshots any old files instead of doing nix derivations. Nix couples the build process to the content-addressability of the output, which works great if you want to put all the effort in to deterministic builds. We just read files like git does which works great for non-deterministic processes like npm install (tragically).
I like the idea of the Big Disk style of attaching a content addressable cache, but in our experiments we still found the network latency to the attached disk too high when reading file by file, like when booting a node app, so we’re caching a much smaller amount on a local SSD for each prod server. Maybe replit isn’t as sensitive to read perf from the cache layer, or they have fancy local per-node read through caching within the overlay setup? Regardless, cool!!
last week benchmarks from tvnix (a rust rewrite of the nix cli) was published which saw a THIRTEEN second evaluation speed up on a basic derivation evaluation.
https://twitter.com/matthewcroughan/status/16606138356553482...
How much of Nixpkgs can Tvix build these days?
Tvix can *evaluate* a serious (?) chunk of Nixpkgs, but we have strictness bugs in some areas.
Tvix has no builder yet.
You'll be interested in https://github.com/thoughtpolice/buck2-nix
First, the non-Flakes CLI wll be stabilized, in phases.
Afterwards, Flakes itself and its CLI components can be stabilized. The final design of Flakes will also require another RFC.
Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod. Even NixOS isn't running Nix code. It's running systemd units.
What people mean by "use X in production" is that X is a key dependency as part of the production process.
If you describe a NixOS configuration for the system where your services are running, you're "using" Nix, even though the Nix-specific stuff is already done.
This relates to risk. -- If it turned out that using Nix to build packages or to configure a system then caused problems when running services later, that would be a problem. Whereas, "using in prod" suggests confidence in no problems, or in being able to fix any issue in a timely manner.
For instance, this statement:
buildInputs = with pkgs;[
python34
python34Packages.docopt
];
The writer of the Nix expression does have to pay a significant cost, in my opinion. If done correctly, the users of the Nix expression can benefit enormously. Using a Nix project can be simple. These commands:
nix flake show
nix build ...
nix run ...
nix flake check
A very complex application written in multiple languages with multiple services can be set up with a Nix expression. The application developer then needs only the commands above to build, run, and test the application. Furthermore, Nix can provide the compiler and IDE support (such as an LSP server) to the developer.
Doesn't look like specs or implementations are public yet.
I’m not sure what golden standard we are comparing this to. It is not perfect, but I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
I cannot live without that anymore, that's the new golden standard.
There isn't any, of course. But one high standard of excellence that we should celebrate and look up to (or just build upon!) is the Guix full-source bootstrap!
They've achieved something pretty incredible in terms of supply chain auditability: https://guix.gnu.org/blog/2023/the-full-source-bootstrap-bui...
Maybe we could also do more with Trustix, and/or build in something like `guix challenge` to the new Nix CLI, to make verifying package provenance and contents easier for users.
> I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
With the exception (in some areas) of our little sister, Guix, absolutely. It's a real relief to have such a predictable, inspectable system as you get on NixOS. The drive for standards like software bills of materials comes from the chaos of the old world, the clumsy ways of building and distributing software, that proper functional package management like Nix obsoletes.
(Unfortunately we do still cope with and suffer from those old ways in Nixpkgs where the upstream build systems inflict it on us. The JVM and .NET come to mind.)
nix-store --query --tree "$(nix-store --query --deriver "$(which python3)")"What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies. By failing to manage dependencies (by pretending that versioning doesn't exist, and supporting only exact software sets), Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows.
So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.
Next, Cargo is not a solution: it can only do a tiny part of the problem, namely managing Rust dependencies for a Rust project.
What if you are building a tool that needs a shared lib to communicate with.. basically anything? Rust has an enthusiastic community, but it is still a tiny language — most problems require more than what can be found in any one language’s berks.
Also, what even is a version? 3.4 of package X doesn’t mean anything — if it has like 10 flags than it can be built 2^10 different ways, and then we didn’t even count its dependencies that are also not just a number. Nix does the correct thing, not the naive one.
(The same applies for the newer 'nix' CLI, but I'm less familiar with that)
> What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies.
I agree. I think Nixpkgs can/should only officially support one combination of packages in a unified collection, up to the point of including pre-built binaries for that exact combination of dependencies in the main community binary caches.
But allowing multiple versions of packages to live in Nixpkgs (or an ancillary repo), automatically marked with version ranges reflecting the versions that those recipes have previously been successfully built against, would greatly simplify searching for and using packages by their upstream versions. The most trivial and most common case for overriding packages— changing the version of the sources to be built— would be obviated, leaving overrides to more advanced use cases like adding patches, changing build flags, adding dependencies, etc.
I think there's still room to grow Nix in this direction, though! The limitation here is incidental rather than fundamental.
> Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows.
Not true. There is a problem that can occur where a lot of users or even collaborators on a project don't really know how to build it, or what all of the requirements are. This happens sometimes with projects where Docker is always the expected way to build or deploy. But sussing out the real build requirements from a Dockerfile is hard in a way that doing the same from a Nix package is not! Docker allows for all kinds of non-determinism, and dependencies in Docker can be supplied implicitly (or at least indirectly— you may have to track down how a particular base image was built to figure out if some dependency is assumed to be installed). But things like blocking network access and the prefix hacking in the Nix store ensure, if not guarantee, that the dependencies of Nix packages are explicit.
> So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.
Nix is great, and it's great when projects offer a flake.nix or a shell.nix. But (1) development communities not knowing what their own requirements are is a cultural problem and (2) I hope and expect we'll see little of this. The kinds of things Nix does to environments that are Nix-specific are generally extremely unlikely to affect program behavior, like setting the date and build time to the start of the Unix epoch.
Have you encountered any projects that depend on Nix-specific behaviors? Or are you mainly worried that the kind of natural vendorization that Nix provides will lead developers to only test their software on narrower and narrower ranges of version combinations?
I guess, but I would not characterize it as a problem of testing. As I understand it the thing you are "supposed" to do is use the upstream developer's exact dependency tree, unless you override it in which case you've completely voided the warranty as it were. Or, this applies to collaborators across the same project too. This means using whatever broken outdated environment that developer happens to prefer, possibly because of other software on their system transitively requiring ridiculous environments, or because they're just lazy or whatever.
This doesn't apply to software in the blessed set that's kept up to date, but I don't think you get points for being at parity with, say, Ubuntu.
At the end of the day, when it comes to development environments, Nix should be a way to help provide a reproducible fast track for building and/or running the software, not a preemptive dismissal of all other possible downstream users and distros! If your software has a Nix flake but no build instructions, it's missing documentation. That's probably okay for solo open-source projects that are small and indifferent to building a userbase, but I hope maintainers and companies will readily recognize that as they grow they'll have to fill that gap at some point.