Why We're Moving on from Nix | Hacker News Reader

Why We're Moving on from Nix

blog.railway.com

275 pointsmooreds136 comments

Okay I'm a Nix enthusiast but you'll have to trust me when I say that I'm not criticizing them for moving away from Nix; it isn't that strong of an emotional attachment. However, I'm not really sure I understand some of these complaints and they really could use more explanation. For example:

> The biggest problem with Nix is its commit-based package versioning. Only the latest major version of each package is available, with versions tied to specific commits in the nixpkgs repo.

While Nixpkgs is an amazing resource, Nix != Nixpkgs. Nixpkgs is highly unideal for cases where you want to be able to pull arbitrary versions of toolchains, but it is not the only way to go. For example, there is amazingly good Nix tooling for pulling an arbitrary version of Rust. Other Nix-based developer tools have shown how you can do this well.

> no way of splitting up the Nix dependencies into separate layers

That doesn't make any sense. You can literally just split them into separate layers in whatever arbitrary fashion you'd like. The built-in Nixpkgs docker tooling has some support for this even.

> We also changed the codebase from Rust to Go because of the Buildkit libraries.

This part is not related to Nix, but I find it interesting anyways. Obviously most people don't transition programming languages on a whim, it's generally something you do when you're already planning on building from scratch anyways. To me it almost sounds like different people worked on Railpacks vs Nixpacks.

(I've definitely seen what happens when people not familiar with Nix wind up having to deal with unfinished Nix solutions within an organization. It is not pretty, as most people are unwilling to try to figure out Nix. I don't generally use Nix at work out of fear of causing this situation.)

Honestly this feels more like rail... wants to make their own version, hence a new railX lol

Correct

I don't use Nix, however this seems dismissive:

> While Nixpkgs is an amazing resource, Nix != Nixpkgs.

If Nixpkgs is the default and alternatives require additional research and effort then for most users it _is_ Nix.

> That doesn't make any sense. You can literally just split them into separate layers in whatever arbitrary fashion you'd like. The built-in Nixpkgs docker tooling has some support for this even.

Is this obvious, simple, and default behaviour?

Nixpkgs isn't Nix and in production you rarely just use Nixpkgs verbatim. It's trivial to overlay whatever versions you want (including forks), and I'd say it's expected for any company in production to manage their package set.

We are talking about a company full of professionals. If they need something obvious, simple, and default to manage their build - the core business function that turns their text into deployable artifacts - maybe there is a skill culture issue.

The industry is full of ineptitude though.

> The industry is full of ineptitude though.

While I disagree with the person you're replying to, I find your reply dismissive.

I don't know the behind-the-scnene reasons for this, but I can very very easily apply a very similar situation to this from my experience.

Nix is a full blown functional programming language along with a very rich (and poorly documented, niche, only second to C++ template in error comprehensibility[1]) ecosystem in itself. It's not like "docker" or "kubernetes" where you're mostly dealing with "data" files like yaml, json or Dockerfile. You're dealing with a complex programming project.

With that in mind:

- You have a core team with 1 or 2 people with Nix passion/expertise.

- Those people do most of the heavy lifting in implementation.

- They onboarding the team on to Nix

- They evangelize Nix through the org/company

- They mod and answer all the "#nix-discussions" channel questions

Initially the system is fairly successful and everything is good. over the next 5-6 years it would accumulate a lot of feature asks. The original "Nix person" has long left. Most of the original people have moved either to other projects or not particularly that passionate about Nix. In fact, the "best" developer you have who has inherited the whole Nix thing has only really had to deal with all the shit parts of Nix and the system. They are they ones fixing issues, dealing with bugs, etc. All while maintaining 3 stacks, a Nix stack, a Go stack, and a Rust stack.

Eventually that person/team that's annoyed by maintaining the Nix project wins. They want to own that code. They don't want to use Nix any more. They know what's needed, they want to implement it as part of their main Go stack that they are actively working on. They can optimize things for their special case without having to worry about "implementing it the Nix way" or "doing it upstream".

They promise you (the management who is open to the idea, but trying to understand the ROI) feature parity + top 5 feature asks for the initial release. You trust the team enough to let them do what they think is best.

[1]: LLMs are really good at suggesting a solution given an error message. Nix errors bring them to their knees. It's always "Hmmm.... it appears that there is an error in your configuration... have you tried a `git revert`?"

I'm not being dismissive. Well, I am dismissing a lot of industry people's opinions. Because they're bad.

Just because people decide stuff for money doesn't mean I can't call them bad. Not everyone is equally skilled.

And your parable is exactly the issue. The unskilled and loud and whiny do often win, and it's a shame. I see it all the time.

(Also you're way overstating Nix as a "full blown FP language." It isn't hard to learn. I learned it just be existing on a project with it. Within 6mo, now I'm apparently a "Nix expert" and people now point at me as one of the people who "knows it" and "you can't expect everyone to know it like you do." idk maybe I'm some genius but I think it's more that I just don't have a bad personality.)

So you are being dismissive. That's what you're doing. You're dismissing more than just "stuff for money". You're dismissing anything that doesn't fall under the "skill" or "technical" category. All software projects contain a human element. I was showing an example from my experience on how something like that could happen.

> A perfectly capably (but perhaps a bit esoteric) technology is picked by a smart passionate person for a project.

> The novel technology is in 1 isolated module that's mostly feature complete for the first 1-3 years.

> People in the team/company deal with that "thing" as a blackbox more and more

> 5-10 years later, mostly new team maintaining the project. They hate the weird choice of tech. "Why is only this one component different???"

> People understand the contract with the "black box" very well, but HATE the "black box". People think "We can implement the black box contract very easily"

Yes I am dismissing. People don't have a right to not be dismissed if I judge them poorly. People are allowed to have bad professional opinions of others.

And I am dismissing the types you describe specifically. I dismiss them (privately amongst the likeminded) at work all the time too. I just put them on a list in my head when they start spouting these sorts of bad values.

Two things can be true in your hypothetical:

1. Someone was forced to maintain Nix and want to switch to easier to maintain tooling 2. That someone can lack in technical understanding of the problems they are facing

The former doesn't negate the later.

The way I would put it is sometimes you choose a worse option because the people you have available are better at that option. That doesn't mean you made a mistake but it does mean your lack of expertise sent you down a different path.

And of course to finalize I will re-emphasize my "didn't make a mistake" comment. Ivory tower isn't a good idea either.

But someone responding "it is too bad the company that built packages couldn't properly use the package tooling they depended on" can still be true in a situation where a company made the correct decision of dropping that package tooling.

> maintaining 3 stacks, a Nix stack, a Go stack, and a Rust stack

Nix is a package management system with a little bit of programming tucked onto the side. "Nix stack" is not the same type of thing as "Go/Rust stack". If you try to move things over to Go/Rust, you'll spend a little bit of time rewriting the code and a whole lot of time reinventing the wheel on everything else involved. You're not moving between implementations, you're building your own implementation. That's almost always a bad idea, and it's a much higher cost than learning the syntax.

Moving from a Nix stack to a Go stack only makes slightly more sense than moving from a docker stack to a Go stack. Which is to say, very little sense.

But they also have to manage the Git stack on top of all that! /s

(Though really, what's up with so many people in the industry being absolutely bad at git well into their careers?!)

Wow, this is literally the story of my team. Luckily there’s enough autonomy for us the escape the gravitational pull of the few remaining evangelists, but this is essentially what led us to this point.

I think that's kind of sad. There are a lot of good technologies that require some investment to understand them and a lot of worse is better approaches that end in a mayhem that is ultimately a terrible work environment.

For a while it seemed like investment would win but now I think we will have massive tombs of poorly written stuff that can only be worked around with even more LLM generated solutions. It is like the software bloat problem we've had recently but on steroids.

> If Nixpkgs is the default and alternatives require additional research and effort then for most users it _is_ Nix.

This feels rather dismissive. They wrote a bespoke solution, not a weekend toy. Surely you'd agree that they have more than just surface-level knowledge of Nix, to be able to distinguish between Nix and Nixpkgs? They're already doing non-trivial things by merging multiple commits of Nixpkgs in order to get different versions of different tools!

> Is this obvious, simple, and default behaviour?

Well, Nix doesn't do much of anything "by default", it's a pretty generic tool. But insofar as it matters, Yes, pretty much. `dockerTools.buildLayeredImage` will in fact automatically build a layered image, and it is the most "obvious" way (IMO) to build a docker image. There is also `dockerTools.buildImage` but there's no particular reason to use it unless you specifically want a flattened image. (The documentation available is clear enough about this. In fact, in practice, much of the time you'd probably actually want `dockerTools.streamLayeredImage` instead, which is also documented well enough, but that's beyond the point here.)

But that's not my point. As far as I know, Nixpacks don't even use this functionality, I'm pretty sure they wrote their own OCI image building tools. And in that sense, it is not obvious why they can't split the Nix store and the article doesn't explain it.

My point wasn't to be dismissive about the difficulties of Nix, it's that the blog post doesn't really do a good job of explaining things. It makes it sound like these are normal problems in Nix, but they are not; even the official Nixpkgs documentation often points to third party solutions for when you're working outside of Nixpkgs, since most of the Nixpkgs tools is geared for Nixpkgs and NixOS usage. As an example, take a look at this section of the Rust documentation in Nixpkgs:

https://github.com/NixOS/nixpkgs/blob/master/doc/languages-f...

So even if you're relatively new to Nix, as long as you are reading the documentation you will indeed definitely be aware of the fact that there is more to the Nix ecosystem than just Nixpkgs. It may not be surface-level Nix knowledge, but it's certainly close.

This isn’t “most users”, this is a large company building a product on top of Nix. I’m pretty sure most orgs using Nix at a minimum have a custom overlay.

If you identify these things as an issue, any competent engineer should find a variety of solutions with search and/or LLM assistance within an hour, since they’re not super obscure requirements.

I’m not saying Railway didn’t do this and realize that these common solutions weren’t viable for them, but it’s odd to not mention anything they tried to get around it.

To emphasize this point: dleslie's comment is valid on a blog post "we tried Nix for a while to manage our dependencies, we are just building an app and want builds to work, and we decided to move on". For an end user, it is absolutely understandable to assume "nix = nixpkgs".

But as kfajdsl points out: that's not what TFA is. This is a company building a product on top of Nix. Package management is their expertise. Anyone using Nix in that capacity understands the distinction between nix and nixpkgs. Which they certainly do--GP only remarked it was odd they didn't explain it, not that they didn't know.

> Is this obvious, simple, and default behaviour?

Yes. There are two options IIRC, minimum layers and maximum layers (one per dep by default unless that makes too many, which is handled automatically) depending on what you want, and it’s a Boolean flag. If you need more control it’s more complicated but this one really is a strange criticism unless they’re using non-standard wrappers for the usual nix way to do this.

> Is this obvious, simple, and default behaviour?

In the documentation[0] they're right next to each other, and `buildImage`[1] (builds a single layer) specifically calls out that you probably want to use `buildLayeredImage` or `streamLayeredImage`[2] (both produce a separate layer per dependency) instead.

Neither should cause the final image to include build dependencies, that sounds like they were doing something silly like running `nix-build` from inside a Dockerfile and just taking that as their final image. Which.. yes, would include build cruft. Oh,[3] I guess that was exactly what they were doing after all. And mixing in Debian packages... for reasons, I guess.

[0]: https://nixos.org/manual/nixpkgs/stable/#sec-pkgs-dockerTool...

[1]: https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-dockerToo...

[2]: https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-dockerToo...

[3]: https://github.com/railwayapp/nixpacks/blob/205b33b515282cdf...

> If Nixpkgs is the default and alternatives require additional research and effort then for most users it _is_ Nix.

This is not what parent commenter is getting at. Nix itself is a deterministic build tool. There is also a package manager built on top which uses a large collection of nix files to describe each package - this is nixpkgs.

They use the same primitives, but the same way you don't just yolo your node/rust etc build versions to whatever your OS comes with and use a lock file, you also want to have more control over the exact versions and thus may use something other than what nixpkgs packages. Especially that it makes it easy to override any property of your dependencies, unlike any other tool out there.

I think this is pretty well stated. I'll add that while nixpkgs isn't nix, nixpkgs is kind of the good part. I use NixOS and for the first time in my life, I'm using the latest version of the Linux kernel on release day. That's pretty excellent. While I've come to tolerate Debian Stable in my old age, it is always like stepping a few years into the past ;)

The Nix language is something I could criticize for hours without getting bored, but it is what it is. It's old and they did the best they could and it's probably not worth changing. The Nix build system feels awfully primitive to me, often rebuilding stuff that doesn't need to be rebuilt for no good reason. (For example, my NixOS installer ISO has a ton of the build depend on the cmdline I pass to the kernel [just console=ttyS2,1500000n8], and so changing the speed of my serial port requires about 3 minutes of build time. It's goofy and makes me laugh, I'm not going to stop using Nix because of it... but it's also something that I wouldn't let happen in MY build.)

Nix for Docker images is, in my opinion, what it's the worst at. A long time ago, I was writing some software in Go and needed to add the pg_dump binary from Postgres to my container image. The infrastructure team suggested using Nix, which I did, but our images blew up from 50MB of our compressed go binary to 1.5GB of God Knows What. pg_dump is 464K. I ended up doing things my way, with Bazel and rules_debian to install apt packages, and the result (on top of distroless) was much cleaner and more compact. My opinion with some actual Nix experience is that a Nix system always ends up being 1.4GB. My installer ISO is 1.4GB. My freshly installed machine is 1.4GB. That's just how it is, for whatever reason.

Finally, the whole "I would like to build a large C++ project" situation is a well worn path. s/C++/Rust doesn't change anything material. There are build systems that exist to make the library situation more tolerable. They are all as complicated as Nix, but some work much better for this use case. Nix is trying to be a build system for building other people's software, supporting nixpkgs, and lands on the very generic side of things. Build systems that are designed for building your software tend to do better at that job. Personally, I'm happy with Bazel and probably wouldn't use anything else (except "go build" for go-only projects), but there are many, many, many other options. 99% of the time, you should use that instead of Nix (and write a flake so people can install the latest version of Your Thing with home-manager; or maybe I'm just the only person that uses their own software day to day and you don't actually need to do that...)

> a Nix system always ends up being 1.4GB

That's strange, I never had problems building really tiny docker (release) images with nix, in fact it felt easier than doing it with alpine. You just get exactly what you specify, no more.

(OTOH, when developing in nix, I always end up with a huge /nix/store and have no idea how to clean it without garbage collecting everything and having to wait all over)

> I always end up with a huge /nix/store and have no idea how to clean it without garbage collecting everything and having to wait all over

FYI you can avoid things getting garbage-collected by doing `nix-store --add-root`; that makes an "(indirect) garbage collector root"[0]. Especially useful if you're using import-from-derivation, since that imported derivation won't appear in the dependencies of your final build output (which, to be clear, is a good thing; since it lets us calculate a derivation, e.g. by solving dependency constraints or whatever, without affecting the eventual hash if that calculation happens to match a previous one!)

[0] https://nix.dev/manual/nix/2.18/package-management/garbage-c...

I think the part that's easy to miss is that their users are devs who will want to specify their own dependencies and versions for arbitrary packages.

The way nix works with the way nixpkgs is structured, pinning a version of any package means pinning a commit of the entire nixpkgs tree. Since package builds of node/python/ruby packages depend on stuff outside of the package dir in the tree, you need that mapping between versions and commits. It is also a leaky abstraction, so they will need to expose that to their users, who now may run into situations where they need to align various states of the nixpkgs repo when they just wanted to "yarn add new-fancy-nodejs-package-with-linked–native-deps".

Using nix without nixpkgs may be fine for more scoped use but seems hard to justify for a platform like Railway.

> Since package builds of node/python/ruby packages depend on stuff outside of the package dir in the tree, you need that mapping between versions and commits.

> Using nix without nixpkgs may be fine for more scoped use but seems hard to justify for a platform like Railway.

Nixpkgs isn't all-or-nothing. You're right that Nixpkgs itself rarely packages more than one version of something, but the standard approach for "language package managers" is that you use a tool like crate2nix[0] which automatically generates pinned derivations for all of your dependencies.[1] For system dependencies which aren't covered by your language package manager.. you're basically in the same position as for something like Debian: you can either pull it from Nixpkgs (and give up control beyond "which Nixpkgs am I on?") or you can write/fork your own package. Or you can pull specific packages from specific Nixpkgs checkouts and splice them into your "main" Nixpkgs version as an overlay (though this is definitely getting into the Weird Territory(tm)).

[0]: https://github.com/nix-community/crate2nix

[1]: For example: https://github.com/stackabletech/secret-operator/blob/30f0eb...

> The way nix works with the way nixpkgs is structured, pinning a version of any package means pinning a commit of the entire nixpkgs tree.

You can do that for a single package. For example, the latest version of linux-firmware is currently broken for my wifi cards so I pinned linux-firmware to commit dda3dcd3f of nixpkgs, whereas the rest of my software comes from the latest version of nixpkgs. There are multiple ways to accomplish this, but the way I did it was:

1. Add an input of a specific version of nixpkgs (which I called `nixpkgs-dda3dcd3f`): https://code.fizz.buzz/talexander/machine_setup/src/commit/e...

2. Add an "overlay" which replaces linux-firmware in the regular nixpkgs with the copy from nixpkgs-dda3dcd3f: https://code.fizz.buzz/talexander/machine_setup/src/commit/e...

> Nix != Nixpkgs

I've been told this when trying FreeBSD in regards to freebsd ports. pkg generally works fine for me, but one day I tried to go off the beaten path and compile vim with some custom USE flags (I forget what they are called in freebsd) in the ports section. It pulled down 20+ dependencies and each `make menuconfig` kept asking me "would you like any of these options" I selected a few that seemed reasonable, and lo-and-behold, package 16 out of 23 fails because "this-requires-that and that-needs-Fubar3.32.1 and Fubar3 is deprecated for Fubar4" and I just gave up. I get that the Core OS devs can't support all 10k+ packages, but they should also be very clear that if you actually try to use them (i.e. enable custom features, not just compile stock code) there's a high chance they won't work. Another option would be to yank them from the ports list if they don't compile and require at least some standard, independently-produced build to succeed before they appear in portsnap fetch.

> While Nixpkgs is an amazing resource, Nix != Nixpkgs. Nixpkgs is highly unideal for cases where you want to be able to pull arbitrary versions of toolchains, but it is not the only way to go. For example, there is amazingly good Nix tooling for pulling an arbitrary version of Rust. Other Nix-based developer tools have shown how you can do this well.

It’s so exhausting that every single time a basic use issue comes up with nix, the response is “but there’s a way to work around it (that you need tribal knowledge for and will require writing dozens to hundreds of lines of code to fix in a language that doesn’t work like any of the mainstreams with bad error messages and poorly documented standard libraries)”.

People’s problems with nix are not that it isn’t turing-complete, it’s that it often creates more problems than it solves by refusing to provide a simple first-class API that interoperates with idiomatic projects in that ecosystem that just works.

If every project you try to use nix for devolves into centering around trying to fix issues with nix that you have to write your own modules for, why even bother using nix instead of mainstream tools with good documentation? Exactly what happened in this case. In most cases people are probably just deciding to use docker instead.

Nix’s refusal to address practical developer experience issues for a developer-facing product on a non-geologic timescale in favor of ideological pure flakes is rather frustrating.

Yes, people are contributing their own time, but it’s so damn frustrating to see so much technical effort going into something that’s rendered practically unusable because of bad UX.

You can use nixpkgs as a single versioned input, and you can also give certain things their own distinct input to be tracked separately in flake.lock, but I wouldn't describe either as a "workaround" for the other.

It is a bit of a headache to have to pick which mode you want for each dependency, but I'm not sure that's a headache that can be dispensed with via UX improvements.

> why even bother using nix instead of mainstream tools with good documentation?

nix lets you have a single source of truth for all of it. It's not nix vs apt or nix vs pip, but rather nix vs (pip & (apt|brew)).

So far as I know, the only other tool that scratches that itch is bazel, otherwise you're kind of stuck with multiple overlapping packaging strategies that don't communicate, which is a recipe for "works on my machine".

> It’s so exhausting that every single time a basic use issue comes up with nix, the response is “but there’s a way to work around it (that you need tribal knowledge for and will require writing dozens to hundreds of lines of code to fix in a language that doesn’t work like any of the mainstreams with bad error messages and poorly documented standard libraries)”.

> People’s problems with nix are not that it isn’t turing-complete, it’s that it often creates more problems than it solves by refusing to provide a simple first-class API that interoperates with idiomatic projects in that ecosystem that just works.

Well, first of all, this article isn't even about the UX of using Nix directly, it is about Nixpacks, a tool built on top of Nix. If Nix already solved the problems Nixpacks were trying to solve, they would've had no reason to write it, so I don't really see how this could be relevant.

> If every project you try to use nix for devolves into centering around trying to fix issues with nix that you have to write your own modules for, why even bother using nix instead of mainstream tools with good documentation? Exactly what happened in this case. In most cases people are probably just deciding to use docker instead.

Would it matter if there was good documentation if you were not willing to read it anyway? Both of the issues I talked about actually are covered in the official Nixpkgs documentation and have been for years. For example, here is some of the documentation for layered docker images:

https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-dockerToo...

Nixpkgs and NixOS are maybe not perfectly documented, but on the other hand, they're absolutely enormous projects. The documentation that does exist is already staggering. And of course it is! Seriously, try finding a singular project that has as big of a scope as Nixpkgs...

Of course, what people seem to read from what I said is that Nix is actually perfect and there are no problems, but what I'm really saying is they didn't go into very much detail on how they tried to solve the issues they ran into. The two possibilities are that they didn't try very much, or that they did and they omitted it. The latter is certainly as plausible, but it leads to this confusing problem where their complaints don't really make much sense on their own. What I am talking about is not obscure bespoke tribal knowledge. It's pretty close to the first stuff you would learn if you read docs or tutorials.

> Nix’s refusal to address practical developer experience issues for a developer-facing product on a non-geologic timescale in favor of ideological pure flakes is rather frustrating.

> Yes, people are contributing their own time, but it’s so damn frustrating to see so much technical effort going into something that’s rendered practically unusable because of bad UX.

This really seems like it's veering far away from the discussion about Nixpacks and into personal grievances about Nix, but honestly there's thousands of build systems and package managers that don't work the way Nix does, I do not really see why Nix should compromise its ideals. for the sake of UX. But please don't get me wrong: I agree on the point that Nix has a worse UX than it could; I think the language is full of unneeded papercuts and confusing things and meanwhile the time it has taken to stabilize flakes and the new Nix command has really been a drag and introduced a lot of confusion. On the other hand, though, I'm not even sure it's worth wasting too many tears over this: as hard as it is to get started in Nix and as many things as there are that could be improved with Nix and Nixpkgs itself, really the chief pain I feel when dealing with anything involving Nix is not Nix itself but the world it exists in.

Nix has existed for over 20 years and in those years the world of OS design and package managers only really started to move towards immutability and purity relatively recently. Most software still likes to have tons of mutable, impure shared state and trying to encapsulate this into Nix can be very painful. Nix has grown many limbs and improved in many ways to try to deal with this, but it is still far and away one of the biggest sources of confusion that I have with Nix.

It is not a given that things have to work this way, but it is a natural consequence of the fact that Nix is trying to do something that is very much at odds with the way software has worked for a very long time. The impedance mismatch is massive and unavoidable, but I don't think the answer is that Nix should bend to deal with this. If someone finds what they feel is a better sweet spot between what Nix is today and the world outside of Nix, they should feel free to pursue that, but I'm involved with Nix because I think it has the right ideals just way too early.

Having dealt with systems that try to be hermetic like Bazel, you can certainly get some meaningful benefits from sitting at a midway point, but what we're chasing are the benefits you get much closer to the end-game of hermetic systems, when hermeticity is enforced strongly across the entire system. This is about more than just being a convenient tool for developers and much more into the future of how systems are designed. And yeah, sure, when you try to build on top of this in today's world, it can be awkward, nobody is denying that, but a lot of this awkwardness is, unfortunately, a feature, and while clever Nix solutions may eventually exist for some of those problems (a lot of clever work is being done, with concepts like dynamic derivations) I really feel strongly that you shouldn't hold your breath.

And sure, by all means, switch to OCI tooling if you feel like it works better. It may very well actually be better for some use cases! But you literally can not replace most of what Nix is used for and can do with OCI tooling, it's ultimately a very small subset of what Nix is capable of overall.

I’m very sorry. But for a person that tries to get into Nix this reads like pure gaslighting. I haven’t even found an official description of what Nix vs NixOS vs nixpkgs tries to achieve vs what they do not want to achieve.

There is a near infinite collection of blog posts what people use Nix for. All full of Nix-internal jargon derisively critiquing alternative (that do in fact actually work) without ever coming out and laying the groundwork for what the underlying problem is _and what the solution to that is_.

I think that I would end up liking Nix in principle. I really did enjoy the Spack package management system and I do think Nix is doing things reasonably similarly (everything is Nix, all inputs are securely provided, reusing things as much as possible).y current problem is taking a homelab server into “production” and all I really want is a way to write down a state I want the services on it to be in, have that in a git repo and be able to regenerate this server after loosing it (in a first step without any data backup). I’m still, after 2 months (wall clock, call it maybe 3 days actually getting into this project), not sure if that’s a thing Nix tries to cover! (I think it does, but..)

A lot of work about packaging specifics reads like inside baseball because it is. Most people manage to avoid getting into the specifics of packaging because it’s stuff that mostly gets in the way of the problem they’re trying to solve. But for blessed few packaging is either critical to the problem or is the problem itself. If you never had to learn the inside baseball terminology, it is likely that packaging is not critical to problems that you solve, and I say that without judgment. If you need to get into Nix, you will. There are lots of reasonable ways to manage personal infrastructure that don’t involve Nix. For the problem you describe, I’m not even sure Nix is a preferable solution unless you already use it.

That said, I agree with the original comment. I’m willing to believe that they had these problems and that moving away from Nix was the right decision, but there is little detail in the explanation. They’ve been pretty closed users of Nix to my knowledge, building proprietary tools on top of it without contributing back significantly, and it feels like orgs such as repl.it are contributing back more actively, but this may be a marketing difference as well.

I spend significant parts of my PhD maintaining the packaging and deployment infrastructure of our group. But yes, I have close to no computer science and reasonably little intrinsic interest in the problem space (though I have a deep appreciation). Still I think I have a better than average understanding of the packaging space..

I think you may be overthinking things. In brief;

Nix is two things: a functional programming language called Nix, and a package manager written in C++ that compiles and installs packages written with Nix expressions. Nix the language itself, and it's implementation.

Nixpkgs is a repository of Nix packages. It also contains NixOS, which is also just a bunch of Nix.

NixOS is a Linux-based operating system built on Nix. It is configured using NixOS modules, a configuration system of sorts built on top of the Nix programming language. A NixOS system is literally a single big Nix package built from NixOS modules that depends on a bunch of other packages.

If this still seems too complicated it would probably be better to just dive in and mess with it so you can physically see what's going on.

> current problem is taking a homelab server into “production” and all I really want is a way to write down a state I want the services on it to be in, have that in a git repo and be able to regenerate this server after loosing it (in a first step without any data backup). I’m still, after 2 months (wall clock, call it maybe 3 days actually getting into this project), not sure if that’s a thing Nix tries to cover! (I think it does, but..)

Yes that does sound like something NixOS can cover. Obviously, you will still need backups for your mutable data (databases, files, etc.), but if you are able to encapsulate the entire setup of a server into a NixOS config you can indeed throw a machine out the window and have a new one set up and booting the exact same services in minutes.

This is pretty much the original thing that drew me into NixOS, especially because at the time I was highly interested in qemu GPU passthrough setups and I really liked how with Nix all of the complicated setup bits could be expressed in a single file, with about 20 lines of Nix, all accounted for and version controlled. A lot better than just hoping you remember to keep things documented.

I won't lie, learning NixOS and using it for a real machine setup was very painful for me at first. I struggled with many things over time. Still, I really think because the scope of things these projects do is huge it's really easy to overcomplicate things. All of the explosive scope of Nix comes from relatively simple primitives applied in large systems.

I really don't want to encourage people to suffer with Nix if they feel like it's just too convoluted and hard to work with. I am not trying to convey that Nix is simple or easy on the whole. I was only ever trying to explain why I felt the issues raised by this post were confusing to me, not trying to tell people that Nix is generally easy. I think a lot of replies genuinely continue to misunderstand this, but the point of saying that it's easy to do that in Nixpkgs is not to dunk on people who struggle with Nix, it's to say that it's not clear why Nixpacks would struggle with these particular issues. You can literally just split the /nix/store paths into separate OCI layers. There is no obvious reason why this strategy can't be used, and the blog post doesn't go into the background behind it. I bet there actually is background, it's just left unsaid.

I think I got the first part, including and up to

> If this still seems too complicated it would probably be better to just dive in and mess with it so you can physically see what's going on.

And it really seems like NixOS being _the_ correct :tm: solution for what I want from an home server. The problem is that Nix doesn't quite cover (or at least didn't when I first deeply looked) my needs (it lacked a useful Clickhouse package). And I don't even want to say "Nix should be simpler". I don't even think it _can_, it's fundamentally a complicated and complex problem to deal with!

My problem is solely with the Nix community. The amount of jargon-loving and argumentative "you shouldn't attempt what you are attempting in the first place" or "you are holding it wrong" gaslighting is really something :D

I think the point where you fail to grasp the problem is: Nobody is claiming that Nix can't do a thing. My complaint is: There is no way to find _the_ way to do the thing. And there seem to be multiple _incompatible_ ways to achieve any given state. Without any way to distinguish those (let's call them) camps for a new person. In this way it's everything Unix but dialled to eleven! You don't need to argue (for me) that anything _can_ be done. I'm getting that from "Turing complete". My problem is only the practice: How do I get there?

edit: I think my immediate problem is: How do I get the series of "commands" that a nix derivation will invoke? Kind of like "what's the receipt, that is being executed?" That's missing for a sensible repl-loop for me. I'm sure I'll be getting there. The capability that Nix offers is extremely appealing. But at the same time the documentation and community are unhelpful to the extreme

> My problem is solely with the Nix community. The amount of jargon-loving and argumentative "you shouldn't attempt what you are attempting in the first place" or "you are holding it wrong" gaslighting is really something :D

> I think the point where you fail to grasp the problem is: Nobody is claiming that Nix can't do a thing. My complaint is: There is no way to find _the_ way to do the thing. And there seem to be multiple _incompatible_ ways to achieve any given state. Without any way to distinguish those (let's call them) camps for a new person. In this way it's everything Unix but dialled to eleven! You don't need to argue (for me) that anything _can_ be done. I'm getting that from "Turing complete". My problem is only the practice: How do I get there?

I honestly have relatively little experience with the majority of the Nix community, so I can't speak too much to it. I did not personally experience too many problems.

The frustrating thing to me about this thread is that it definitely wasn't in this vein. I know I'm harping on this a lot, but it matters.

What the blog post says is "we can't split the image into multiple layers with Nix". And again, they're doing their own OCI building, so this actually has little to do with Nixpkgs and is much more conceptual in nature. If you build a Nix closure, what you wind up with is a number of store paths. For example, for bash, you might have the following store paths:

    /nix/store/za53jjhjl1xajv3y1zpjvr9mh4w0c1ay-xgcc-14.2.1.20250322-libgcc
    /nix/store/yypqcvqhnv8y4zpicgxdigp3giq81gzb-libunistring-1.3
    /nix/store/6nkqdqzpa75514lhglgnjs5k4dklw4sb-libidn2-2.3.8
    /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66
    /nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37

So how do you split these into layers? Well, however you want. You can place them into individual layers, or you can group them based on dependencies to make more "logical" groupings and get less layers, or basically any number of things. There's plenty of metadata on what depends on what, so it's really up to you to decide how to do that.

I was not talking to end users when I said this, I was talking to developers building on Nix that almost definitely knew this. So it's unclear based on what they said why they couldn't.

> edit: I think my immediate problem is: How do I get the series of "commands" that a nix derivation will invoke? Kind of like "what's the receipt, that is being executed?" That's missing for a sensible repl-loop for me. I'm sure I'll be getting there. The capability that Nix offers is extremely appealing. But at the same time the documentation and community are unhelpful to the extreme

I see.

Well, for starters, you are trying to start at the very low-level bottom of the stack. I understand this because it is the approach I prefer to take when learning new technology, but I'm telling you right now this is a hard road to go down with Nix, I've been down it. The documentation is going to continue to feel unhelpful because you will find plenty of documentation that is happy to take you down the dark paths of how things actually work, but that doesn't make them any easier to navigate. It's much easier to start by asking "how do I accomplish this end-goal" than to ask "what actually is a derivation".

With that out of the way... to understand what's going on with builds, you really genuinely do need to know what a derivation is, because it matters. Almost nobody calls the `derivation` function directly in Nix, instead using Nixpkgs mkDerivation tool at the bare minimum, but at the very bottom, there is only one function that matters:

https://nix.dev/manual/nix/2.22/language/derivations

At the bottom, in order to actually build something, a derivation has to be evaluated and realized. Internally, instantiating a derivation will cause it to be built. It is built by taking the parameters passed to derivation and serializing them to an internal format, then "instantiating" it, causing it to be built inside of the sandbox. The internal derivation format looks like this:

    Derive([("debug","/nix/store/xsxanvyy93g3k5dnh0j2wisrwvq012b2-bash-5.2p37-debug","",""),("dev","/nix/store/lnzar8zawj43wkqlsdqhywda9hjymrgf-bash-5.2p37-dev","",""),...

Basically a lot of arrays and tuples and whatnot. Not very easy to read. `nix derivation show` can give us better human-readable output:

    {
      "/nix/store/rfkzz952hz2d58d90mscxvk87v5wa5bz-bash-5.2p37.drv": {
        "args": [
          "-e",
          "/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh",
          "/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh"
        ],
        "builder": "/nix/store/razasrvdg7ckplfmvdxv4ia3wbayr94s-bootstrap-tools/bin/bash",
        "env": {
          "NIX_CFLAGS_COMPILE": ...

So what does `nix build nixpkgs#bash` do? Well, aside from the sandbox setup, on my particular machine, it would run... /nix/store/razasrvdg7ckplfmvdxv4ia3wbayr94s-bootstrap-tools/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh. So, basically, a bash script. What does that bash script look like? Here's the entire contents of default-builder:

    genericBuild

This is in the heart of the Nixpkgs "stdenv" machinery, and now we're outside of what Nix itself does and into what Nixpkgs does. Literally all Nix does is handle the book-keeping: sandboxing, execing the builder, handling the Nix store; for the actual build, it is just execing the builder, and in most cases, it will wind up being handled by a bash script. You could have whatever builder you want here, of course, and it needn't be bash or bash-based. (Note that we're calling bash to build bash - funny choice for me to pick, but you can see how it gets the copy of bash to build bash with from a special derivation called bootstrap-tools. Gotta break the cycle somehow, and Nix doesn't have as impressive of a bootstrap seed as Guix.)

So, for this build, it just calls genericBuild. What does that do? Well, you can keep chasing down the scripting here, but ultimately it calls all of the build phases, e.g. unpackPhase, patchPhase, configurePhase, buildPhase, checkPhase, etc. which are all just bash scripts inside of environment variables. At the end of the day, it does roughly what you would do to build bash, plus some patches. The nixpkgs stdenv automation helps handle many special Nix things, but it's not actually horribly complicated.

By the way, when you see this sort of thing in a Nixpkgs derivation:

    patchPhase = '''
        ...
    ''';

It is in fact just setting up environment variables. You can see them in `nix derivation show`, even. They get called just by substituting them directly into a command, not any different than:

    test="echo Hello world" bash -c "$test"

Basically, all of the attributes in the mkDerivation call become environment variables, including src, and all of the magic that gives them meaning is just in the stdenv default builder. (This is roughly true, anyway. I think there is some move to make it "stricter" but the default way it works is still like this roughly.)

So how can you debug what actually happens when building a derivation? Unfortunately it's not terribly easy. You can view the logs with `nix log`, but it will only show one if it actually built it, not if it substituted it from Hydra cache for example. Also, the default behavior of the builder doesn't output that much diagnostics, though you can add NIX_DEBUG = 7 in a derivation and it will output tons of information about what is being executed.

The way derivations depend on other derivations is something like this:

- During build-time, when you try to coerce the result of the derivation function to a string, what gets returned is the realized store path. So if I try to do `${pkg.bash}/bin/bash` inside of Nix, it might return `/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin/bash`. In this way, it is natural to literally just depend on something implicitly by realizing it as needed.

- As far as I know, for runtime dependencies, it is detected by literally scanning the $out directory after a derivation is built and looking for the Nix store paths of other derivations. Merely realizing a dependency during build-time won't make it a runtime dependency.

It's more complicated than this, but that's the broad strokes. So when you build bash, it will implicitly depend on all kinds of build tools like GCC and make, but the resulting derivation will only depend on things like libgcc that are actually referenced in the final derivation.

Honestly, it is a bit of a black box. It is not easy to navigate. Trying to learn Nix from the ground up first will be very hard. I wouldn't ever tell someone not to try, but I will definitely say it won't be easy.

I don't understand the versioning argument. New to nix, but I definitely have packages from a specific commit.

The version selection part sounds weird. The versions in nixpkgs make sense when you're running/building the system. If you're providing runtimes/compilers as a platform, you really want something like what devenv does - provide the versions yourself. You really don't want to end up building an old system to provide an old nodejs - you're leaving security patches in dependencies behind. Devenv does it for example through https://github.com/cachix/nixpkgs-python "All Python versions, kept up-to-date on hourly basis using Nix."

> Railway injects a deployment ID environment variable into all builds.

They could've done it in the next layer after installation. Also, you can split packages into different layers. There's even automation for it if you need batches to keep the number of layers down.

> we don’t have any problem with Nix itself. But there is a problem with how we were using it.

A good example of 'use the right tool for the right job'. Nix is great for some use cases and awful for others. The problem is the Nix learning curve is so high that by the time you've grasped it enough to make a decision you feel you've invested too much time to back out now and pivot to something else so you try to shoehorn it to solve the original need.

I feel it like that as well, but in some ways Nix is a more normal programming paradigm than other OS’s. We’re just not used to thinking about an OS that way. Nix expressions have inputs (a package repo, lots of key-value pairs) and outputs (a Linux system). Idk perhaps in a couple of years it will be much more normal.

Ie it is very easy for an AI to create a to-spec shell.nix (some Python packages, some Linux packages, some env vars, some path entries etc), or configuration.nix because of this paradigm.

I do this a lot to include envs with repos that fully support the package. It would probably be more reproducible with flakes (a flake.nix is like a shell.nix but with version pinning… or something, I’m still climbing that learning hill).

I often find it takes about 10 minutes of my time to package your average python package into nix, or 1 minute if its pyproject.toml based (uv/poetry projects!.

There is a long tail though (cough weasyprint cough).

That's mostly just a tooling issue/a paradigm mismatch. Python's package management is notoriously bad, so it sorta makes sense that it won't fit as nicely with Nix.

> Python's package management is notoriously bad, so it sorta makes sense that it won't fit as nicely with Nix

This sort of thing is immaterial to users of Nix/Python who just want it to do the god damn thing it's supposed to do.

"just a tooling issue/a paradigm mismatch" is how many people pronounce "bad UX".

Looks like they are trying to force versions into where there are none. Just like trying to force a square cube into a round hole.

"Default versions" breaking things that depend on them? What is that? It is like using docker's ":latest" tag and being surprised each time that a new server falls on its face because the "default" image is actually a different version from the previous "default" image.

I don't understand any of the explanations in this blog post. Seems like people who have zero clue about what a "version" of a software is.

"no way of splitting up the Nix dependencies into separate layers" - Why? Of course you can split /nix/store into as many layers as you need. Do they even know how to use containers and how to use Nix in the first place?

With the clear incompetence of these people, no wonder that their proposed solution smells like a decomposed fish.

Classic NIH syndrome. There is going to be no surprise to see them meet the exact same problems they didn't solve with Nix to infest their new "solution".

What works better to raise VC (which for many is the goal)

A nix wrapper or a deployment platform

Traction.

I don't think any VC worth the time is going to sit around nitpicking how much Nix matters to their offering if they're making increasing amounts of money.

Depends who’s pretending to be a VC, or accelerator

It works best to post your "solution" on hacker news and get roasted by ALL the commenters who have a clue while you have none.

> What works better to raise VC (which for many is the goal)[?] A nix wrapper or a deployment platform

Well, if I wanted VC money for this, even if I was buliding a simple nix wrapper, I'd still tell the VCs that it was a deployment platform. (to be clear I have no opinion on TFA or any idea whether this is what they're doing)

I’m all for not using nix, especially where it doesn’t make sense. But rebuilding a working system from scratch, for reasons that aren’t actually a problem if they were to have spent even a couple hours looking at how people already solve those issues, seems fundamentally insane.

Like others have said here, nix2container and flakes seem like they would address every problem they have.

With regard to versioning, I have flakes written 3 years ago that still build with exactly the same versions and the same output as when it was first written.

Sure does sound like that want to go to market and raise off a platform :D

Edit: Literally just checked nixpacks’ github and it immediately jumped out to me that they are using rustPlatform in nixpkgs, not oxalica’s rust-overlay[0] which would have come up in any cursory search for the rust issues. And is one of the most useful and powerful overlays I’ve used.

[0] https://github.com/oxalica/rust-overlay

Yea, i kinda hate Nix - but it's fundamentally changed how i view OSs and the minimum features i want from them.

Though I really want another language than Nixlang, but i've been stuck on Nix for years despite not liking it because the concrete builds are just so good. If Nix actually had a language i liked i'd probably go full immutable user config too and fully embrace nix. It's just so, so good.

I'd switch in a heartbeat if someone iterated on Nix for some of my complaints, though. But i'm not switching away from the concrete system builds.. it's just amazing.

If you like Lisp there is GUIX which is the Nix packaging idea but using a language that is slightly less trash. The ecosystem is much worse so it would not be great for a desktop end user AFAIK.

I'm hoping Nickel some day can be used in place of Nix. I'm not a fan of lazy piles of config-func-soup regardless, but if i just had types and sane errors my UX in Nix would go up tremendously. Nickel promises some of that, at least.

> With no way of splitting up the Nix dependencies into separate layers

nix2container [1] is actually able to do that: you can explicitly build layers containing a subset of the dependencies required by your image. An example is provided in this section: https://github.com/nlewo/nix2container?tab=readme-ov-file#is...

For instance, if your images use bash, you can explicitly create a layer containing the bash closure. This layer can then be used across all your images and is only rebuild and repushed if this bash closure is modified.

> > pull in dependencies often results in massive image sizes with a single /nix/store layer

This is the case for the basic nixpkgs.dockerTools.buildImage function but this is not true with nix2container, nor with nixpkgs.dockerTools.streamLayeredImage. Instead of writing the layers in the Nix store, these tools build a script to actually push the image by using existing store paths (which are Nix runtime dependencies of this script). Regarding the nix2container implementation, it builds a JSON file describing the Nix store paths for all layers and uses Skopeo to push the image (to a Docker deamon, a registry, podman, ...), by consuming this JSON file.

(disclaimer: i'm the nix2container author)

[1] https://github.com/nlewo/nix2container

Just wanted to say thanks for nix2container. I’ve been using it to do some deploys to AWS (ECR) and my iteration time between builds is down to single digit seconds.

We’ve had issues with docker image sizes and have been meaning to take some time to experiment with nix2container. Thanks for your work!

The main problem here is wanting to hang on to the "bespoke version soup" attitude that language package managers encourage (and is totally unsustainable). The alternative Mise doesn't appear to have any ability to understand version constraints between packages and certainly doesn't run tests for each installed package to ensure it works correctly with the surrounding versions. So you're not getting remotely the same thing.

"the "bespoke version soup" attitude that language package managers encourage"

Care to elaborate what that means and what the alternative is?

Not sure on the exact take of the OP, but:

Package maintainers often think in terms of constraints like I need a 1.0.0 <= pkg1 < 2.0.0 and a 2.5.0 <= pkg2 < 3.0.0. This tends to make total sense in the micro context of a single package but always falls apart IMO in the macro context. The problem is:

- constraints are not always right (say pkg1==1.9.0 actually breaks things)

- constraints of each dependency combined ends up giving very little degrees of freedom in constraint solving, so that you can’t in fact just take any pkg1 and use it

- even if you can use a given version, your package may have a hidden dependency on one if pkg1’s dependencies, that is only apparent once you start changing pkg1’s version

Constraint solving is really difficult and while it’s a cool idea, I think Nixpkgs takes the right approach in mostly avoiding it. If you want a given version of a package, you are forced to take the whole package set with you. So while you can’t say take a version of pkg1 from 2015 and use it with a version of pkg2 from 2025, you can just take the whole 2015 Nixpkgs and get pkg1 & pkg2 from 2015.

> Constraint solving is really difficult and while it’s a cool idea, I think Nixpkgs takes the right approach in mostly avoiding it. If you want a given version of a package, you are forced to take the whole package set with you.

Thank you, I was looking for an explanation of exactly why I hate Nix so much. It takes a complicated use case, and tries to "solve" it by making your use-case invalid.

It's like the Soylent of software. "It's hard to cook, and I don't want to take time to eat. I'll just slurp down a bland milkshake. Now I don't have to deal with the complexities of food. I've solved the problem!"

I mean you can do it in Nix using overlays and overrides. But it won’t be cached for you and there’s a lot of extra fiddling required. I think it’s pretty much the same as how Bazel and Buck work. This is the future like it or now.

It’s not an invalid use case in nixpkgs. It’s kind of the point of package overlays.

It removes the “magic” constraint solving that seemingly never works and pushes it to the user to make it work

> I was looking for an explanation of exactly why I hate Nix so much

Note that the parent said "I think Nixpkgs takes the right approach in mostly avoiding it". As others have already said, Nix != Nixpkgs.

If you want to go down the "solving dependency version ranges" route, then Nix won't stop you. The usual approach is to use your normal language/ecosystem tooling (cabal, npm, cargo, maven, etc.) to create a "lock file"; then convert that into something Nix can import (if it's JSON that might just be a Nixlang function; if it's more complicated then there's probably a tool to convert it, like cabal2nix, npm2nix, cargo2nix, etc.). I personally prefer to run the latter within a Nix derivation, and use it via "import from derivation"; but others don't like importing from derivations, since it breaks the separation between evaluation and building. Either way, this is a very common way to use Nix.

(If you want to be even more hardcore, you could have Nix run the language tooling too; but that tends to require a bunch of workarounds, since language tooling tends to be wildly unreproducible! e.g. see http://www.chriswarbo.net/projects/nixos/nix_dependencies.ht... )

There’s no clear definition (in most languages, of major/minor/patch versioning). Amazon did this reasonably well when I was there, though the patch version was implicitly assigned and the major and minor required humans to follow the convention:

You could not depend on a patch version directly in source. You could force a patch version other ways, but each package would depend on a specific major/minor and the patch version was decided at build time. It was expected that differences in the patch version were binary compatible.

Minor version changes were typically were source compatible, but not necessarily binary compatible. You couldn’t just arbitrarily choose a new minor version for deployment (well, you could, but without expecting it to go well).

Major versions were reserved for source or logic breaking changes. Together the major and minor versions were considered the interface version.

There was none of this pinning to arbitrary versions or hashes (though, you could absolutely lock that in at build time).

Any concept of package (version) set was managed by metadata at a higher level. For something like your last example, we would “import” pkg2 from 2025, bringing in its dependency graph. The 2025 graph is known to work, so only packages that declare dependencies on any of those versions would be rebuilt. At the end of the operation you’d have a hybrid graph of 2015, 2025, and whatever new unique versions were created during the merge, and no individual package dependencies were ever touched.

The rules were also clear. There were no arbitrary expressions describing version ranges.

For the record, Amazon's Builder Tools org (or ASBX or whatever) built a replacement system years ago, because this absolutely doesn't work for a lot of projects and is unsustainable. They have been struggling for years to figure out how to move people off it.

Speaking at an even higher level, their system has been a blocker to innovation, and introduces unique challenges to solving software supply chain issues

Not saying there aren't good things about the system (I like cascading builds, reproducibility, buffering from 3p volatility) but I wouldn't hype this up too much.

Nix generally speaking has a global "nixpkgs" version (I'm greatly over-simplifying here ofc) in which there is a single version of each package.

This is likely the source of their commit based versioning complaint/issue, i.e the commits in question are probably https://github.com/NixOS/nixpkgs versions if they aren't maintaining their own overlay of derivations.

This is in contrast to systems that allow all of the versions to move independently of each other.

i.e in the Nix world you don't just update one package, you move atomically to a new set of package versions. You can have full control over this by using your own derivations to customise the exact set of versions, in practice most folk using Nix aren't deep enough in it for that though.

It's the idea that every application can near-arbitrarily choose a bespoke-but-exact mix of versions of every underlying package and assume they all work together. This is same attitude that leads to seemingly every application on planet earth needing to individually duplicate the work of reacting to every single dependabot update for their thousands of underlying packages and deal with the fallout of conflicts when they arise.

Packages in nixpkgs follow the "managed distribution" model, where almost all package combinations can be expected to work together, remain reasonably stable (on the stable branch) for 6 months receiving security backports, then you do all your major upgrades when you jump to the next stable branch when it is released.

Put out fewer versions of things. It is entirely possible to write a piece of software and only change the interface of it at rare intervals. The best solution I can think of though would be to allow one version of a package to provide multiple versions of its interface. Suppose you want to increment the minor version number of your code and this involves changing the signatures of a number of functions, you could design a programming language packaging system such that both versions are defined in the same package, sharing code when needs be.

That's why everything in Nixpkgs is defined as a function, which takes dependencies as arguments.

It also puts a function in the result, called `override`, which can be called to swap out any of those arguments.

Which leads to the exact problems defined in this article. Many programs using many library versions. It would be much better from both a security and size perspective if these disparate packages could be served by a single shared object using versioned symbols.

Hmm, not sure I agree. Most of those arguments get populated by taking the fixed-point, along with any given overlays; so it's easy to swap-out a library everywhere, just by sticking in an overlay. The exceptions mostly seem to be things that just don't work with the chosen version of some dependency; and even that's quite rare (e.g. it's common for Nixpkgs maintainers to patch a package in order to make the chosen dependency work; though that can causes other problems!)

You can actually have both if you do it right. It's trivial to build a rust package with Nix from a Cargo.lock file, for example. Nixpkgs is contrary to bespoke version soup, but Nix itself can be fine with it.

Bespoke version soup is unsustainable, but part of why people keep doing it is that it tends to work fine. It tends to work fine in part because OS-level libraries come from a different, much more conservative world, in which breaking backwards compatibility is something you try to avoid as much as possible.

So they can take a stable, well-managed OS as a base, use tools like mise and asdf to build a bespoke version soup of tools and language runtimes on top, then run an app on top of that. It will almost never break. When it does break, they fiddle with versions and small fixes until it works again, then move on. The fact that it broke is annoying, but unimportant. Anything that introduces friction, requires more learning, or requires more work is a waste of time.

Others would instead look for a solution to stop it from breaking ever again. This solution is allowed to introduce friction, require more learning, or require more work, because they consider the problem important. These people want Nix.

Most people are in the first group, so a company like Railway that wants to grow ends up with a solution that fits that group.

In my experience as a DevOps/SRE, I feel like every time someone tries to have a system to manage dependencies etc, its goes one of two ways (I'll use Python as an example):

OPTION 1

"We'll have one big shared mono repo"

Pros:

- it's all in one place

- it's "batteries included"

- everyone uses the same one (so things like vulnerability issues are easy to fix)

Cons:

- someone always wants a special version

- hard to do tiered rollouts so changes tend to be big bang

- "But what about if we want to build a small docker version?"

OPTION 2

"Everyone gets their own conda/venv!"

Pros:

- Everyone gets exactly what they want

- Don't use packages they don't need

- Easy to upgrade in phases/tiers etc

Cons:

- "Wait, we have HOW MANY different conda environments??"

- Libraries from different groups may not be tested with the same Python libraries

- Vulnerability management is a nightmare b/c you don't even know were all of the different conda envs are.

The above is why I'm always skeptical of "This new way will fix it all!".

In short, "there are no solutions, only tradeoffs" gets more and more true the later I go in my career.

As someone with only a little experience with Nix, the points here don’t really seem right?

> This approach isn’t clear or maintainable, especially for contributors unfamiliar with Nix’s version management.

> For languages like Node and Python, we ended up only supporting their latest major version.

What is not maintainable about this? That they need to make a list of available versions? So, can this not be automated?

Furthermore, why is Railway defining how a user uses Nix?

Surely one of the points of Nix is that you can take a bare machine and have it configured with exactly what versions of packages you want? Why would Railway need to get in the way of the user and limit their versions anyway?

Or did I misunderstand and they don’t even expose Nix to the user? If so, the original question still stands: can’t they automate that list of package versions?

> Or did I misunderstand and they don’t even expose Nix to the user?

That's at least my understanding, yes.

you can not have a dockerfile in your project at all, push your code to them, and they’d build an image for it with nixpacks. you’d see nix stuff in your build logs, but it’s behind the scenes for the most part.

The version limits come from the fact that the Nix cache doesn't maintain older versions. So, if you use an older version, you will have to compile from sources. It sounds like they didn't want to take it upon themselves and provide a cache with older versions, even though it doesn't sound like much effort.

Honestly, the reasons given don't feel very solid. Maybe the person who introduced Nix left and the ones remaining didn't like it very much (the language itself is not very nice, the docs weren't great either in the past).

Still, I'm not familiar enough with the stack they chose, but does it provide a level of determinism close to Nix? If not, it might come to bite them or make their life harder later on.

Nix cache (cache.nixos.org) does in fact maintain older versions[0]. In fact, they maintain so much older stuff (binaries and associated source), that they are used in both research[1][2] and are having issues with gigantic cache size[3][4].

And yes, their reasoning implies NIH and just unfamiliarity combined with unwillingness to really understand Nix.

[0]: https://discourse.nixos.org/t/how-long-is-binary-cache-kept-...

[1]: https://hal.science/hal-04913007

[2]: https://luj.fr/blog/is-nixos-truly-reproducible.html

[3]: https://discourse.nixos.org/t/nixos-foundations-financial-su...

[4]: https://discourse.nixos.org/t/the-nixos-foundations-call-to-...

Even if this weren’t true, setting up their own binary cache on S3 would have been trivial. It took me half a day to get it set up for our CI pipeline.

Nix cache does provide old versions. What they seem to want is old versions built with new versions of dependencies. That's also possible, but you will have to build things.

Patching old software with newer components? Does another tool really offer that automatically? This article is a promotion for their tool that does?

That’s what dynamic linking does in most linux distributions.

From the article:

> The way Nixpacks uses Nix to pull in dependencies often results in massive image sizes with a single /nix/store layer ... all Nix and related packages and libraries needed for both the build and runtime are here.

This statement is kinda like “I’m giving up on automobiles because I can’t make them go forward”. This is one of the things Nix can do most reliably. It automates the detection of which runtime dependencies are actually referenced in the resulting binary, using string matching on /nix/store hashes. If they couldn’t make it do that, they’re doing something pretty weird or gravely wrong. I wouldn’t even know where to start to try to stop Nix from solving this automatically!

I wouldn’t read too much into their experience with it. The stuff about versioning is a very normal problem everyone has, would have been more interesting if they attempted to solve it.

To be fair to the authors, this IS a problem, albeit one they phrased poorly, especially with building docker images via nix. The store winds up containing way more than you need (eg all of postgres, not just psql), and it can be quite difficult to patch individual packages. Derivations are also not well-pruned in my experience, leading to very bloated docker images relative to using a staged Dockerfile.

Image size isn’t something we’ve focused a lot on, so I haven’t spent a ton of time on it, but searching for “nix docker image size” shows it to be a pretty commonly encountered thing.

Nix gives you a commit guarantee rather than arbitrary versions. You're going to put yourself in a bad time when you have edge cases: glibc changes or conflicting shared libraries.

It sounds like it's a little bit too late, but I'm happy to provide some consulting on how you can get it to work idiomatically with Nix.

Product looks cool!

nix solves the shared library incompatibility problem by being extremely conservative. every time anything changes, consequential or not - a comment got modified, documentation changes, a testcase got added, etc. - it will rebuild all dependents. and not just that, but all dependents of dependents, and dependents of dependents of dependents, on and on. this often results in massive massive rebuilds.

sure you are not going to get shared library conflicts, but i think this solution is extremely wasteful, and can make development painful too - look at nixpkgs' staging process.

The reason someone changes a dependency at all is because they expect a difference in behavior. No one would feel the motivation to go update a dependency if they aren't getting something out of it, that's a waste of effort and an unnecessary risk.

Each person doesn't have to perform the build on their own. A build server will evaluate it and others will pull it from the cache.

The greater waste that nix eliminates is the waste of human time spent troubleshooting something that broke in production because of what should have been an innocent change, and the lost business value from the decreased production. When you trust your dependencies are what you asked for, it frees the mind of doubt and lets you focus on troubleshooting more efficiently towards a problem.

Aside, I spent over a decade on Debian derived distros. I never once had one of these distros complete an upgrade successfully between major versions, despite about 10 attempts spread over those years, though thankfully always on the first sacrificial server attempted. They always failed with interesting issues, sometimes before they really got started, sometimes borking the system and needing a fresh install. With NixOS, the upgrades are so reliable they can be done casually during the workday in production without bothering to check that they were successful. I think that wouldn't be possible if we wanted the false efficiency of substituting similar but different packages to save the build server from building the exact specification. Anything short of this doesn't get us away from the "works on my machine" problem.

Yep, anyone not getting how absolutely huge the Nix model is should just install the whole KDE desktop, the Gnome desktop, and uninstall both. Only nix can make it basically a no-op.

> With NixOS, the upgrades are so reliable

Yeah they may be reliable _for you_. And do note this reliability doesn't come automatically with Nix's model, it is only possible because many people put a lot of effort into making it working correctly.

If you use the unstable channels, you would know. My NixOS upgrades break _all_ the time. On average, probably once a month.

Nix has support for bit reproduction and will not rebuild on comments if you specify it.

Of course lots of software isn't ready for but reproduction which is why Nix has taken such a pragmatic approach. (I have written a lot about this).

It's all a series of tradeoffs. If your goal is reproducibility (as close as you can get), you will have a larger graph likely ..since you are accounting for more!

Sometimes we like to believe we can have our cake and eat it too rather than understand life's a series of tradeoffs.

When we think we are getting a silver bullet, we've likely just pushed that complexity somewhere else.

IIUC you are talking about CA-derivations? Yeah they may help but it's hard to know how much since it's not in production yet, despite being part of Eelco's original paper describing nix. So my hope isn't high.

> When we think we are getting a silver bullet, we've likely just pushed that complexity somewhere else.

True but we kind of just stopped looking. and I feel much of the solution space hasn't been explored.

I mean, there is no other way that guarantees correctness across arbitrary tools/functions/builds. Like, what if I have a step that replaces certain comments with code?

Also, the primary way to develop with Nix is to create your exact, reproducible environment in the form of a shell, and then develop there using the usual, language-idiomatic iterative way.

But now you can actually have a very specific compiler-flag for only a single dependency mixed with a full different libc working in a given shell 100%, for you and everyone else, instead of iterating through nodejs and npm version combination to start working on this new project, taking a couple of days..

> You're going to put yourself in a bad time when you have edge cases: glibc changes or conflicting shared libraries.

I totally understand the value proposition of Nix. However I think saying "bad time" is a bit hyperbolic. At most it's "You'll be losing a pretty significant guarantee compared to Nix". Still probably "packed to be more likely to work correctly" than 95% of software out there.

I dont see why they couldnt made their own derivations instead of relying on nixpkgs hashes.

Fascinated by so many replies of "actually Nix does this just fine, you just have to be an expert like me"

It's "the usual" when mentioning nix anywhere.

When a company is writing all of their technology and business using, lets say JavaScript. And then they come here and post about them switching to some NIH home-brewed language instead because they couldn't understand how functions or arrays work. That is not a problem with the people commenting on their stupidity.

Yeah but if everyone was saying "I don't understand how functions or arrays work in Javascript", that's a pretty solid indicator that functions and arrays are badly designed in Javascript and are unnecessarily hard to understand.

I think in some cases things are just fundamentally difficult and them being hard-to-understand is intrinsic. For example formal verification in Lean is hard to understand but I don't think Lean is badly designed.

But it's hard to see why package management is one of those things. There are soooo many ways Nix could be easier to use and understand. The language itself is unnecessarily esoteric in my experience - compared to something like Starlark for example.

Welcome to the world of Nix.

Just the latest in the line of "my totalizing world view will solve all your software problems" to which the answer of "this doesn't do what I want" is always "you're holding it wrong."

From the headline I thought they were making a single incremental change (Nix) but the article sounds like they’re doing an entire rewrite under a new project name:

> Since we transitioned away from Nix, we also transitioned away from the name Nixpacks in favor of Railpack. We also changed the codebase from Rust to Go because of the Buildkit libraries.

Suddenly the move away from Nix seems less like an incremental change and more like one part of a complete overhaul of the entire project. Did they have a team changeover or something? Or did they just want to start over from scratch and rewrite the whole project?

It also seems strange to switch to an entirely different programming language for a single library. I haven’t found library FFI to be a huge obstacle in Rust.

That also jumped out to me as pretty weird. Buildkit as a library is nice, but there's no way it's good enough to justify throwing out the entire project. It feels like there was some big change internally, and the new team either wanted to rewrite and didn't know Rust, or ideologically prefer Go to Rust.

Edit: Although looking at it, maybe not?

Both the new project railpack[0] and the older one nixpacks[1] are both started by and mostly written by the same person[2], who is also the author of the article in question. So it doesn't look like a team change.

It still feels... odd? Less that they made the change, projects go from Rust to Go all the time. But usually it's because of issues with Rust (hard to hire for, learning curve, etc.), describing it like this feels unusual?

[0] https://github.com/railwayapp/railpack

[1] https://github.com/railwayapp/nixpacks

[2] https://github.com/coffee-cup

Nix strikes me as an incredibly well thought out solution to a set of problems that should not exist.

The OS should be immutable. Apps and services and drivers/extensions should be self contained. Things should not be installed “on” the OS. This entire concept is a trillion dollar mistake.

Well, what you're describing is NixOS.

Everything you said "should" is exactly what Nix and NixOS is.

The problem is that "should" is papered over a system not designed that way from the start, and in a heterogeneous world which operates in its own way, not Nix's way.

So it's just an eternal battle between the real world and the fantasy Nix world, built on an army of thousands of volunteers writing up .nix files and contributing packages, and trying to tame the beast of Unix complexity.

Except most of us are just trying to get things done and now I work in a shop where I can't even run a development setup with my favoured IDE, or just "cargo run" my package, because Nix has been placed in the middle of everything.

Clearly in a sincere effort to tame complexity, but in the process creating a whole new world of it, with a lot of normative lifestyle assumptions.

To me... in my experience... Nix promises a lot more than it can realistically deliver, comes with a totalizing world view, and in practice ends up making life more complicated unless you completely and totally buy into its way of doing everything, and are willing to leave many of your old tools and practices behind.

The right way to tame complexity is not that much of a hassle. This is why virtualization and containers have been so successful. Make the OS immutable by converting into a nested Russian doll of OSes with specific instances being mutable but also disposable.

It’s ugly in its own way but it has the virtue of not exchanging one ugly complexity tax (OS rot and manual admin) with another (a complicated “manager” that wants to get in the way).

I kind of dig the idea of a unikernel world where there's just a hypervisor and applications running directly on top of it shipping only the bits of the OS they need.

>The biggest problem with Nix is its commit-based package versioning.

I am naive about Nix, but...

...isn't that like...the whole selling point of Nix? That it's specific about what you're getting, instead of allowing ticking time bombs like python:latest or npm-style glibc:^4.4.4

Odd to attach yourself to Nix then blog against its USP.

Eh. I've been using nixOS for years now and still find that I often desperately, desperately wish I could upgrade just one program that I need a new version of without risking that, you know, any individial single one of my installed packages has a change between the last update and now that messes up my workflow. Or that I could pin a version of a software that I'm happier with that version of without essentially rolling my own package repo. It is, in fact, the only package manager I'm aware of that makes it such a pain to do that. It's only because people in this thread are insisting it's doable that I say 'such a pain' instead of just 'impossible'.

A few weeks ago I needed to update firefox for a bug fix that was causing a crash, but of course that meant updating all of nixpkgs. When I finished the switch, the new version of pipewire was broken in some subtle way and I had to roll it back and have been dealing with firefox crashing once a week instead. I can't imagine pitching this to my team for development when I'm having this kind of avoidable issue just with regular packages that aren't even language dependencies.

To those who say 'if you want to lock your dependencies for a project, you can just build a nix flake from a locked file using the <rust | python | npm> tools' I say, why the hell would I want to do that? Being able to manage multiple ecosystems from the same configuration tool was half the draw of nix in the first place!

Creating overlays on nixpkgs is fairly trivial. There was a bug a couple of weeks ago with yt-dlp, which was fixed in the nightly version. But even today the new version is not yet available in nixpkgs-unstable. Did I wait all this time with a broken yt-dlp? No! I created a derivation that overrides the git commit of the original yt-dlp, and added it as an overlay to nixpkgs. Once nixpkgs has the newer version, I'll remove the overlay and customized derivation. This is a 5-lines-of-code change. You don't have to use flakes if you don't want to, it works without flakes as well.

Now compare the above with how you would customize a version in other systems, like Debian with apt-pkgs ...

Bingo. Nix doesn't give you a generalizable-across-languages-and-ecosystems way of specifying specific versions without blowing up your package size, unless you hand Nix to your users (which we didn't want to do)

Maybe we were holding it wrong, but, we ultimately made the call to move away for that reason (and more)

Um? That's trivial with flakes (and I think it was doable without flakes, but I don't really remember/care). For one-offs (I'd probably do this for your firefox example but YMMV), just tell it the version to run:

  $ nix run nixpkgs#firefox -- --version
  Mozilla Firefox 138.0.1
  $ nix run github:nixos/nixpkgs/nixos-unstable#firefox -- --version
  Mozilla Firefox 139.0.1
  $ nix run github:nixos/nixpkgs/b98a4e1746acceb92c509bc496ef3d0e5ad8d4aa#firefox -- --version
  Mozilla Firefox 122.0.1

Or, if you want to actually incorporate it into your system, tell the system flake to pull whatever versions you want:

  {
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
        nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
        nixpkgs-b98a.url = "github:nixos/nixpkgs/b98a4e1746acceb92c509bc496ef3d0e5ad8d4aa";
    };
    outputs = { self, nixpkgs, nixpkgs-unstable, nixpkgs-b98a }: {
        nixosConfigurations.yourmachinename = nixpkgs.lib.nixosSystem {
            system = "x86_64-linux";
            specialArgs = {
                nixpkgs-unstable = import nixpkgs-unstable {
                    system = "x86_64-linux";
                };
                nixpkgs-b98a = import nixpkgs-b98a {
                    system = "x86_64-linux";
                };

            };
    ---snip---

and then when you pull packages say which one you want:

    packages = with pkgs; [
      dillo  # from stable nixpkgs
      nixpkgs-unstable.firefox  # from unstable
      nixpkgs-b98a.whatever  # from some exact commit of nixpkgs
    ]

I assume you could do the same thing for project-level flakes, but TBH I don't usually do that so I don't have the code to hand. (In contrast with grabbing system packages from whatever version of nixpkgs I want, which I know works because I pulled the example code from the config on the machine I'm typing this comment on.)

> > The biggest problem with Nix is its commit-based package versioning. > ...isn't that like...the whole selling point of Nix?

Not quite.

That sentence is definitely the most ... discussion-worthy comment in the blog.

To my understanding, OP wants to write a tool to make it easy for use cases like "use ruby 3.1 and gcc 12 and ...".

The main Nix repository is nixpkgs. Nix packages are source-based, so the build steps are declared for each version. To save maintenance effort, nixpkgs typically only maintains one version of each program.

I read OP's "commit-based package version" phrase to mean "if you want ruby 3.1, you need to find the latest commit in nixpkgs which used ruby 3.1, and use that nixpkgs revision". -- Although, worth noting, this isn't the only way to do it with Nix.

Though, regarding 'commit-based versioning' as Nix's USP? I'd say that's also a reasonable description, yes. (By pinning a particular revision of Nix, the versions you use will be consistent).

The new design is very similar to Dagger, interesting.

Hmmm. Interesting. I looked at the deployment file and... I'm doing a very similar thing, but with docker-compose and caddy: https://github.com/piku/kata/tree/compose

Completely different approach to dependencies, though. For now.

Couldn't quickly find the blog introducing nixpacks, but weren't these issues clearly visible from the start?

> Smaller Builds > Better caching

what were the benefits that overcame this, and what about those now?

Nix seems to be being undermined internally. I wonder why that would be happening.

Do you have any more observations of this happening?

https://x.com/jonringer117/status/1927872172514840677

Not to be dismissive either, but guix might have been more apt for what they wanted versus nix. Not trying to suggest guix is superior, just possibly a better solution considering their gripes with packages.

this is from the company that didn't have support for Docker images until 2023? come on...

tldr we are moving on from nix because we are selling an alternative to nix?

This is not appropriate to post here but the site has nothing to do with railways.

> We also changed the codebase from Rust to Go because of the Buildkit libraries.

Go is the best choice at the moment for such tools. These tools start a process, do lots of IO and exit.

Very pragmatic choice.

Go also statically links all dependencies and reinvents all the wheels usually provided by the system land. Cross compilation is trivial. It is unrivaled when it comes to deployment simplicity.

Rust does the same.

Yea and at least Go provides a giant Google engineering tier quality standard library so reinventing the wheel here doesn't hurt so much productivity.

Meanwhile Rust requires a pile of variable quality community driven crates to do basic things.

Both languages have enormous cargo-culting issues when you try to do anything that isn't fizzbuzz. The bigger difference that I'd expect people to identify is that Rust generates freestanding binaries where Go software requires a carefully-set runtime. There are pros and cons to each approach.

None of them generate true freestanding executables by default. Both Go and Rust require glibc.

And Go's runtime is built-in by default. Unlike Java so there's nothing to "carefully set".

By that definition, I cannot name a single high-level programming language that generates freestanding binaries.

Many of them can, including Rust and Go. Just not with default arguments. You need to pass linking arguments.

This is not done by default to reduce binary sizes.

By default no for Go, but it's much easier to get their then Rust.

In Go you run this to build:

  CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' .

I do not actually know how you get this done in Rust (you do something with musl and a bunch of other stuff).

Rust can cross-compile, yes, but is not as seamless. For example, Rust can not cross-compile Windows binaries from Linux without external support like MinGW.

Go can cross-compile from Linux to Windows, Darwin and FreeBSD without requiring any external tooling.

We've been working on Pixi, which uses Conda packages. You can control versions precisely and also easily build your own software into a package to ship it. Would be curious to chat if it could be useful as an alternative to `mise`.

Reply on news.ycombinator.com