Hacker News Reader

Comment by hamandcheese | Hacker News Reader

> Nix, on the other hand, does have a concept of state. If you make a one-line change to a 200-line Nix configuration, it doesn’t have to re-do all the work from the other 199 lines. It can evaluate the state of the system against the configuration file and recognize that it just has to apply the one-line change. And that change usually happens in a few seconds.

The author seems to have some misguided ideas about Nix. Nix is not fast because it is stateful. It is fast because it is functional and reproducible, which allows for caching without compromising correctness. I don't want to split hairs, but referentially transparent caching like this is not quite what I'd call state.

Yes, there is some statefulness in system activation, but this is not what makes Nix Nix -- quite the opposite.

View on HN

Explaining what Nix is seems like one of its biggest barriers to adoption.

Part of the difficulty is it means different things to different people. My colleague spent a whole lot of time trying to answer this question and ended up with this:

    My main take away after spending some time learning about Nix is that it embraces the functional programming concept of a pure function. 
    If I give a function a certain set of inputs, it will return the same result every time, no matter what. Nix is about building software the same way, whether it’s your own software, someone else’s software, or your entire OS: You declare all your inputs explicitly and it will be built the same way every time.

    What really mattered to them, regardless of how they were using Nix, was its ability to bring purely functional programming concepts to computing areas that were previously off-limits.

    From that single idea you get a whole ecosystem of tools. We mainly covered the Nix language, the Nix Package Manager, and NixOS, but there’s also a continuous build system called Hydra, nix shell, and a deployment and provisioning tool called NixOps. Probably, there’s even more.

https://earthly.dev/blog/what-is-nix/

Another way someone else described Nix to me was "Gentoo for Haskell devs".

The biggest barrier to adoption unfortunately is not people's inability to explain what the tool is in my opinion. It's that the tool is incredibly complicated, extremely hard to walk someone through compared to alternative projects, and honestly... In my opinion the problem it attempts to solve doesn't really exist.

I'll take an "impure" os or package manager over a pure one any day if complexity is a thousand fold less and the learning curve doesn't require half a decade. Got stuff to do!

> The biggest barrier to adoption unfortunately is not people's inability to explain what the tool is in my opinion.

That's simple: nix is a package manager and the language used by the package manager, NixOS is a Linux distro.

> It's that the tool is incredibly complicated, extremely hard to walk someone through compared to alternative projects, and honestly... In my opinion the problem it attempts to solve doesn't really exist.

From someone who is working as a DevOps Engineer for some years and managing Linux servers for a few years longer that thought is incredible naive. The problem of undefined and undocumented system state is a fundamental problem I encounter everywhere especially bad with legacy systems. I often do things on them blind and just pray for the best outcome, realising months later that some system was broken by one change I did and no one realised that for months.

> I'll take an "impure" os or package manager over a pure one any day if complexity is a thousand fold less and the learning curve doesn't require half a decade. Got stuff to do!

I thought the same first but unwedging Debian once a week on a different system is also not fun and a waste of time and having servers in some undefined state and no one who how the config is supposed to be and why or when it got changed, too.

The result in the end is that every system is different and unique and your Ansible playbook to run a common and good thought out task succeeds on 15 VMs and sometimes completely blows up the 16th because no one could have thought that the state of configuration there is so widely different.

Nix vs not Nix seems like a parallel to Infrastructure As Code (terraform for example) vs Cowboying the AWS Console. Is that a fair comparison?

Yes that is a fair comparison. In the latter you have to write down or remember what you did to reproduce it, and even if you make a script it could screw up and leave your system in a bad state.

That script can also stop working or misbehave once the underlaying system state changes, e.g. as the repository packages get updated.

Not really, unfortunately. In my opinion, Nix is more like using Haskell instead of whatever language your team is using to write software.

> Nix is more like using Haskell instead of whatever language your team is using to write software.

In my professional experience, this has worked well :)

I got pretty far the first year without really writing much nix code at all.

As someone who seems to be going along a similar path (started as a dev, no one was around to do sysadmin so I did it, and now in trying to modernize a bunch of really old/unpatched servers running a legacy system I'm learning how to devops) I feel better knowing I'm not alone in this struggle.

Yes. Think if you coordinated a software project with email and a Sharepoint folder for the code. Then you use git.

Current server state management is the former. Nobody knows what is running where and if some performance differences over time or between servers exist, how can it be bisected etc...

You basically have two choices: you can take the complexity upfront, and in a predictable fashion by learning Nix, or you can deal with the complexity after the fact when you’re dealing with dependency hell and a deadline is looming and your boss and/or client is mad.

The Nix language is basically JSON plus syntax sugar plus pure functions. A Nix derivation can be thought of as a super-powered lockfile that includes not just the versions of the dependencies, but also the build instructions and the environment in which to build them.

The argument for Nix is basically the same argument as the one for writing pure functions as much as possible, or not doing so. Any amount of experience doing the former will demonstrate that it is superior.

Now, Nix may be complex, and some of that may be reducible, but the fundamental idea of treating a build like a pure function is NOT reducible, and is well worth the effort of learning, because it will apply to ANY future pure build and dependency management tool

> In my opinion the problem it attempts to solve doesn't really exist.

It does, but some people are good at numbing themselves to it.

So they block losing a day or half day from lack of reproducibility out of their memory or recall it as "no big deal".

Nobody loses time on Nix issues?

They do, but the people who spend time on Nix issues don't typically notice or count the time saved because it's harder to notice.

When reproducibility issues take 16 hours every 3-4 months/12 weeks and Nix maintenance like updating pins takes 8 hours per month most will feel like the first option is less work.

Imagine if you had the data showing with Nix your build is:

- 99% likely to work

- without Nix your build is 90% likely to work, but 16 hours to fix it when it breaks.

- The non-Nix build also has a 10% chance of it breaking randomly at any time.

- The nix build initially takes 8 hours per month to maintain for 6 months, 4 hours for the next 6 months, then 1 hour per month thereafter

Which do you feel would be better? What I describe above has been what the situation seems to be in my experience.

I can say from personal experience I've seen many days devoted exclusively to Nix upkeep and maintenance. That was from junior people to people who had spent half a decade or so deeply in the community and using Nix for their daily driver.

I've never had to do much for Nix itself, but packaging something to build from source can often require quite some effort. Applications that use a pretty unconstrained build/install process upstream may expect to do a lot of things that are not allowed in the Nix build sandbox, like unconstrained network access and or overwriting files in existing packages on the system. To deal with that you really have to dive in, learn how the sausage gets made in the upstream package, make some choices about if/where to compromise, and then spend some time tweaking and debugging. That can be a pain and can definitely take a day or two.

I've only had 'maintenance' issues with Nix itself on macOS, where OS upgrades routinely nuke Nix's hooks into the OS or add restrictions that break things. (But they do that to other package managers as well.)

You can mitigate this to some extent by approaching it as described here:

https://www.haskellforall.com/2022/08/incrementally-package-...

I think Graham Christensen had a blog post along these lines... I'll see if I can find it.

Edit: I couldn't find it... but I thought someone made a blog post about gradual adoption of Nix into a codebase.

Idk about a blog post, but he did a talk along those lines at the most recent NixCon: https://youtube.com/watch?v=asc1D5yPZhQ&

I'm taking that approach with the package I've been working on, which has a somewhat pathological (by Nix standards) Gradle build which does things like

  - manually download a copy of Elastic search outside of the normal Java dependencies scheme
  - run NPM to fetch remote libraries to build web assets at build time
  - *also* run Yarn, for some reason
  - use Git at build time

The ways it does all of these things are actually fairly thoughtful (for example, it does checksum the artifacts it manually grabs at build time to verify their contents), but they don't play nice with running builds in offline mode or under a user that has no $HOME. But it's one of those freeform 'my build tool configuration is a weird DSL in an imperative, general purpose, Turing-complete language' situations, and I'm not very familiar with either the language (Groovy) or the DSL. So it's a lot of quirks to cope with.

I've made quite a bit of progress in building it from source by making a few small patches and eventually disabling the sandbox for now, but it's still dying on a weird test failure for reasons I don't yet understand. At this point I'm just back to munging the binaries provided by upstream because I was mostly building from source to learn about the project and how it's distributed/deployed anyway.

I messed a bit with gradle2nix for a better-behaved, old school FOD-based build with Gradle in offline mode, but that was pretty brittle as gradle2nix is unmaintained, and due to some design limitations it couldn't actually capture all dependencies. I'm kinda interested in working out something better but on the other hand, this is a third-party package and I don't myself use Gradle or Groovy for any kind of development, so mastering Gradle's quirks and wrangling it into the Nix sandbox for this package is more of a yak shave than a practical skills investment for me.

> In my opinion the problem it attempts to solve doesn't really exist.

Almost all of Docker use-cases are for solving that same problem, but badly and with partial completeness. The lack of adoption is really not caused by lack of value.

This matches my experience with it so far. Extremely complicated and hard to understand, projects that use it have builds fail anyway except now with very hard to debug errors.

When a nix build fails it'll fail with errors from the build system the package uses. The upside is that your failure is now reproducible.

Good point. I'm much more likely to help others because I know I can get to the exact state they are in and reproduce their issue with a simple `nix build`.

> Extremely complicated and hard to understand

That is absolutely not true. If you start to get the hang of it and follow the way things are supposed to be done then things get easier over time. You need to invest upfront more time into your configuration but on the long run it pays off and saves you from an entire error class.

> projects that use it have builds fail anyway

The point of Nix/NixOS is not to have no failing builds but that those are reproducible and deterministic as much as possible and that those failures are noticed early and before the point of no return. A system build is supposed to fail early and not mid way through a major update and prompting you to merge some config under /etc by hand.

You say it's "absolutely not true" that it's "hard to understand", then go on to explain how you have to "invest upfront more time", and also that you have to learn "the way things are supposed to be done".

You are very clearly describing that people have to work more to understand it. The person you're replying to even tried! Denying the experience of other people does not make that go away. It just means that the problem you're pretending doesn't exist will never get fixed.

I've never seen it actually pay off in industry. I've seen it be used as good job security while other devs just wrote docker files and got things done.

As the only guy maintaining the flake.nix in my team's repo, I don't think it's really contributing to my job security. I'm just happy that they don't mind the extra files and commits here and there because I value the ability to contribute from different devices without worrying about which versions of what are installed.

Maybe it'll be job security if people start agreeing that downloading binary tools in in CI without a hash check is unacceptable attack surface, but until then it's just this weird thing I'm doing on the side.

I do catch a lot of bugs where people are relying on dependencies that they happen to have installed but have not declared. It's the kind of thing that prevents newcomers from being successful out of the gate, or makes taking a local process and putting it in CI difficult, but fixing those is not exactly high visibility.

Then you're not paying enough attention. There are plenty of companies using nix to distribute a reproducible environment (if you don't believe me, why not go search GitHub for "flake.nix" and see how many "industry" repos you find).

I think it would be more productive for you to sit down and give it a fair chance than posting little rebukes all over this thread.

I gave it a fair chance and it was a deciding factor in why I left a company believe it or not. Only one person could maintain and fix deployments. Not from lack of trying from seasoned experts and no new comers. It was the worst user experience I have probably ever encountered. Meanwhile I was able to pick up terraform and docker in a matter of days...

In my experience those "others" that "just wrote docker files" are exatly the ones that don't know how to build the system in a reproducible manner if their ci environment gets reset for some reason as they find out that stuff that was "supposed to be there, pinned and configured" wasn't.

In my experience the months required to get a handle on Nix is not worth the benefit(which is shakey in my opinion) compared to competing technologies. We don't have to agree, but that's my take...

You might want to look at things like:

https://devenv.sh/

https://www.jetpack.io/devbox/

If you have something easy to deploy like a go binary you can just write a dockerfile but for big python projects that start to compile dependencies that is quickly no longer true. The dockerfile likely is also not matching the software you run and test on your local machine, so sometimes debugging is not as easy. Ofcourse you can debug inside the container but then you are missing all your tooling and need to bring that with you. And rebuilding a dockerfile is often not reproducible, so if you want the container back from 1 year ago and you no longer have the artifact you are probably out of luck.

With nix you can easily open a shell with the packages used in the docker image or go back in time and reproduce that image from a year ago with the flake.lock from a year ago.

Also applying patches to dependencies used in dockerfiles is not dead easy as with nix.

Most people would opt to not apply patches to their dependencies in my experience. Seems kinda sketchy if that's something you have to do on a regular basis. I'd chalk that up as a possibly serious business concern depending on the magnitude of the fixes, the importance of the dependency, and the frequency.

I have written software that would have been 100% not package-able any other way.

Also, let’s not lie to ourselves, there are plenty of ridiculous contraptions out there, like docker-images used for ML that take up some insane space, and are updated each day. Packaging is a hard problem, and there is finally a tool that can actually solve it.

You can continue spending your time messing around with your system then. I learned nix in a short amount of time and it has supercharged my development workflows and reduced the overall complexity. I have too much stuff to get done to not use it.

supercharged!!!

how much faster are your development workflows now, versus before? like 100x?

how complex were your previous development workflows? what did that complexity manifest as? how has nix made it less complex?

i'm excited to learn more

Based upon how I manage my system, Nix appears to be something that I could use productively. The problem isn't so much of them being able to explain what the tool is, but one of them being instilling confidence that it lives up to their claims. We exist, after all, in an industry of hyperbole. It also doesn't help that their solution is layered on top of an operating system that has traditionally been managed in a very different way.

A friend of mine said that he is currently using it instead of packer at his current gig. He can use the same code to build any type of output, AMI, docker image, VM etc. I dig that. But I'm still not gonna learn nix because I don't do enough of that stuff to warrant the pain of learning nix.

You don't have to manage your system with NixOS to reap the benefits of Nix. It solves very real problems that very much exist, it might not exist if you're a one-man show deploying WordPress to GoDaddy though.

Barrier to entry:

1. Run the nix installer

2. Enable flakes

3. cd project

4. nix run

This ensures you run the package with every dependency except the kernel pinned to a hashed version. If dependency hell is not a problem for you, be happy!

> Run the nix installer

Ok and this requires root access, sets up some global directories under root, and a new user. Me as the administrator: why the hell do I need a new user and what is the nix store and what are the conditions that mutate it? (I know the answers to this question, but it's a barrier for people who give a shit).

> Enable flakes

What the fuck is a flake? Reads a bit... what the fuck is a derivation? (again: I know the answers to these questions already, but the invention of jargon by nix devs is a massive barrier to entry that shouldn't be overlooked, it's extremely confusing)

> cd project

Ok now I'm comfortable doing things I know

> nix run

Fine, but what about auto envs and nix shell? I don't use these with make or cmake. I need to attach a debugger, where does it go? How do I set up my IDE that has no idea nix exists?

My point is, nix has a lot bigger of a barrier than these four lines, and it's really naive to think that's it.

Okay, so you're required to read some documentation ahead of time, that's where your problem lies.

What's a Terraform module? What is Terraform? What is a provider? Why don't I just build all my infrastructure with the AWS Console? Why is it it's own weird language? What is this state thingy that just ended up in my folder? Do i give it to the devs?

I think it's pretty much consensus that Terraform is great for provisioning anything with an API. Nix does the same for your packages, partitions, OS, containers, shells and many more things in the same functional manner.

In a company not everyone has to be a Nix wizard either, if a small team knows Nix they can build the Nix infra, then developers can reap the benefits of not having to mess with it at all.

Just because people are unable to comprehend the benefits doesn't mean they do not exist. And if you wanna reap great benefits you might need to spend an hour or two reading things.

Yes it's a novel way of doing things, but it's also one of the most actively developed projects with one of the highest amounts of contributors in the world.

https://discourse.nixos.org/uploads/default/original/2X/9/9b...

> Fine, but what about auto envs and nix shell? I don't use these with make or cmake. I need to attach a debugger, where does it go? How do I set up my IDE that has no idea nix exists?

The people that know Nix well enough will assist the ones that doesn't know, if you enter a nix shell and start vscode from there it'll be aware of $PATH which Nix sets, meaning it'll find all your dependencies.

> > Run the nix installer

> Ok and this requires root access

On a tangent: I wonder why this is still the default.

The nixStatic binary has, since quite a while, support to as a non-root user create a "${XDG_DATA_HOME:-${HOME}/.local/share}/nix/root/nix" -> "/nix" unshare chroot before running the rest of the command if "/nix" is missing.

It's only a real issue if you really need to run something as root, or something else that needs unshare chroot itself, but in that case, I guess you could just have a /nix store folder anyways.

Almost everything here is valid, but iirc the last time I ran the nix installer I thought it offered a home directory based install now that doesn't require root.

You can use in theory something other than /nix, but then you have to recompile everything yourself. So not many people do if, not sure about it’s state.

I think this is only true if your distro doesn't have user namespaces enabled, and many popular ones do not for security reasons.

I just dipped my toe into the Nix pool a couple of weeks ago, and there are instructions to use root to create a /nix and grant ownership rw to your user account. No further admin required, but everything else seems to work as if using the Nix root user method.

The installer used to have an option for this, but nowadays it's discouraged. How come you wanted a single-user install so much? And j/w, are you on macOS?

Linux, but if there is ever an option to have one fewer services and/or root privileged things, I will take it.

Ok cool! The reason I asked is that the multi-user setup has more benefits on Linux than on macOS. That's because on Linux, the full build sandbox is actually available, and the store's immutability is enforced by default.

The tradeoffs are obviously yours to consider. But the normal Nix build sandboxing helps protect you from nasty things like crypto miners in setup.py or whatever, as well as improving reproducibility.

That's less relevant on macOS where the sandboxing story is not so great.

Personally, using Nix with a daemon seems like a better setup to me but adding another highly privileged process unnecessarily is obviously a real security concern. There is some ongoing work, btw, to reduce the level of privileges that the Nix daemon needs.

You're omitting the entire thing about learning how to write nix. Which is nightmare fuel even for FP fans.

Unless you are attempting advanced things you don't need to know a lot about the language and how the more advanced things work.

Imo the harder part is learning bespoke build processes that you may not own in order to get software that assumes it can perform arbitrary network access or other naughtiness at build time to build successfully in a restricted sandbox.

The language is maybe a little strange at first but there's really not much to it.

Just find a project written in the same language that is already packaged and copy it, modify the name, version, source url, hash and that’s it.

It is one thing that is easier done from the top, instead from the bottom.

You don't have to know how to write Nix to write 95% of all Nix, it's just "JSON with functions" after all. There are definitely advanced things that the Nix and NixOS developers maintain for me.

Just like I don't know how to implement any crypto, or how to implement efficient 3D pathfinding I don't know how to implement NixOS. But I can write a derivation using the helper functions for the language I want to package, which aren't many these days since nixpkgs is huge already.

Step two is not even necessary if you use zero-to-nix's installer. https://zero-to-nix.com/start/install

Yup, I just didn't wanna confuse the already pessimistic person by saying "use the unofficial installer" :)

> Part of the difficulty is it means different things to different people.

That's definitely how it seems to me. The pro-Nix stuff I see is generally about the theory much more than the practice. Which was also my experience with functional languages when their hype cycle was last on the rise.

On the one hand, that's fine. I like ideas, and I think taking an idea and running with it can be really interesting. You can clearly see that in history's various art movements, for example. On the other, for people who are just trying to get things done, it's often alienating and tedious, because the people in the grip of their Big Idea often seem heedless of other perspectives, and frequently can be quite evangelical about it.

Personally, my strategy with Nix, as with the various functional languages, is to keep a distance from it, waiting and seeing. Perhaps it will influence more mainstream projects, bringing the benefits to me without a lot of upheaval. Perhaps I'll have a project that really needs its particular benefits, and so I'll take on the cost of a paradigm switch. But in the meantime, I have stuff to do.

This is probably a pretty good read on the situation. People who have really thrived with Nix are often at the intersection of 'FP people' and 'extremely stubborn Linux people', and that's because sometimes it takes getting your hands dirty and fighting a build system that belongs to a package you've never used before to get the Big Ideas to pay out. When the footwork is already relatively familiar to you, it makes it easier to push through whatever obstacles there are to playing with those attractive Big Ideas in practice.

The Nix community's roots are definitely with FP people, partly due to the language and its inspiration and design, and partially also due to early success using Nix to solve particularly painful Haskell dependency hell problems years ago. All of the original 'marketing material' for Nix focused on principles and properties that would be attractive only to people who already knew and valued those things, which was mostly FP folks.

> Perhaps [Nix] will influence more mainstream projects, bringing the benefits to me without a lot of upheaval.

This is definitely already happening. Off the top of my head, Nix has served as inspiration for Guix, Habitat, and Spack, which are all respectable package managers that try to make things a little smoother than they are with Nix in terms of UX. The latter two are also more conventional, with a relaxed notion of 'purity', and so it may be easier to get packages to build in them when those packages are built in problematic ways. (Guix, if anything, is even stricter about packaging conventions than Nix, but it has a really nice CLI and the language might resonate more with some people, so if Nix has given you pain it's definitely still worth trying.

> Perhaps I'll have a project that really needs its particular benefits, and so I'll take on the cost of a paradigm switch. But in the meantime, I have stuff to do.

I love Nix and its fundamental design, and I want to see it flourish and grow, both in general and in my own professional life. But at work, I try to maintain the same attitude as you describe here. I use Nix for myself everywhere I can (with some escape hatches in place!), but for projects that others work with, I only use Nix where I feel that some specific aspect of the project calls for it.

All of that is just to say that even to some folks who really are drawn to Nix in substantial part due to the Big Ideas that power it, your pragmatic stance is quite understandable and entirely welcome.

Feel free to just play around with Nix in a low-stakes way and advance your usage as curiosity or new problems drive you to do so. You don't have to jump all the way in to benefit from Nix or get a taste of it.

Thanks! Really helpful comment. Especially this bit:

> People who have really thrived with Nix are often at the intersection of 'FP people' and 'extremely stubborn Linux people'

As an extremely stubborn Linux person, that makes total sense to me. But I also almost never recommend Linux to average folks, because I'm keenly aware how far along certain bell curves I am. I would love it if more advocates reflected on whether the personal characteristic that makes a technology great for them is one that makes it bad for others.

Thanks also for the pointers to projects inspired by Nix. I'll check them out.

I often find difficult to explain ideas the most interesting ones. It’s a new paradigm, naturally we need to develop “the neurons” to deal with it, to reason with it, to hold the paradigm in our minds.

I find the best way to get the paradigm is to dive in.

I see from people that did that they are always enthusiastic, it must be worth it.

It doesn’t help that the term is hugely overloaded. Nix might mean the package manager, the shell, the operating system. There are so many valid permutations of letters; why they chose to reuse Nix for everything is a mystery to me.

The other issue is complexity. If you can manage to figure out the jargon, you’re greeted with the requirement that you completely port an entire project to get any benefit, and that’s non trivial. It requires learning a whole new language, and when the (highly opinionated) language conflicts with other tools’ ideas, for example pip, the documentation generally bashes the other tool, boasts how much better it is, and then proceeds to have devs write dozens of lines in a new language they invented when one line of Python used to be enough.

you so not need tonport whole project. you can have shell, remote dev envs, remote build machines all use nix as shell providing build and run env.

nix provides one liner. what one liner in python nix cannot do?

Just to elaborate a bit for those not familiar to Nix (slightly simplified to exclude recent support for content addressing). Nix work with derivations, a derivation is basically a data structure that specifies how a package is built. Derivations are normally not created by hand but using a function (eg. stdenv.mkDerivation).

When you ask Nix to build a package, it hashes a normalized form of derivation data structure. This hash is useful in various ways, but one way it is used [1] is to look up whether the derivation is already in the Nix Store. Because if it is, there is no need to build it. So Nix looks up whether

    /nix/<the_derivation_hash>

exists. If it exists, the build is done. If it doesn't exist and you have a binary cache configured (which by default is the binary cache provided by the NixOS project), Nix will look up the derivation hash in the binary cache. If it exists in the binary cache, Nix will download the path to the local Nix store. After that

    /nix/<the_derivation_hash>

exists in the store and the build is done (without building anything). Only if that fails, Nix will actually build the derivation.

Now, one of the cool things about Nix is that it is derivations all the way down. So, it's not that just what we traditionally think of as packages is a derivation, but people wrap up all kinds of things as derivations, including configuration, etc. Since derivations are usually generated by functions, there are all kinds of useful functions that make derivations for eg.: single configuration files, scripts, etc.

In the end, building a NixOS system generation is just building a derivation. nixos-rebuild switches to a different generation by just setting a bunch of symlinks to an output path in the store containing that system generation (/nix/<system_config_derivation_hash).

At any rate, when you make a one-line change to a 200-line Nix configuration, Nix does have state to keep track of what it needs to rebuild or not. Nix will just try to build the derivation (and its dependencies), but it hashes the derivations, finds that their output paths are already in the store.

Some might argue that then the store is state. But it's not, at build time you are evaluating a pure function with memoization (the Nix Store).

[1] There is also a package name and version in the store path, but lets keep it simple.

I agree with everything but the last statement. This all comes down to: do you consider memoization to be state.

I predict people's answers to this question will come from experience with memoization. Here's mine: I kept trying to get nix to build tensorflow locally, so that I would get the avx512 benefits of the big, but gpu-less machine I had. I hadn't realized some other derivation had already downloaded tensorflow from online cache, so didn't have avx512 enabled. I kept making shells, trying tensorflow, seeing it doesn't have support. The solution was to tell nix to disregard the nix store, in order to force the local build. This experience has left me with the concrete feeling that the nix store is full-on state, and I the user must be aware of it.

> The solution was to tell nix to disregard the nix store, in order to force the local build.

If this actually led to avx512 being enabled in the package, then that's a bug. Nix builds should not be dependant on the machine doing the compilation, all such autodetection should be disabled via configure flag or patched out.

Then, the right way to enable avx512 would be to pass some 'enable avx512 please' flag to the package's configure flags. Which would then trigger recompilation, without any 'disregard the nix store, in order to force the local build' options.

It is, but the possibility of such bugs is a downside to the approach (not saying it's a showstopper but it is a negative). Some of the nastiest software problems to track down are the ones that cause some fundamental assumption everything rests on (often a cache keying assumption!) to be broken, making everything behave wrong, including the tools you're supposed to use to track down problems.

If you're going to build an entire system on an assumption of referential transparency you want to be able to guarantee that everything really is referentially transparent, and one valid criticism of nix is that it can't really enforce that in all cases.

There is work being done to address outputs by their content hash instead of xor-ing their input hashes, which in theory should eliminate this problem.

Yes, you have unfortunately discovered that sometimes the hardware itself is an input that isn't always captured explicitly, but also isn't controlled for with sandboxing. Ideally enabling avx512 would be an explicit input to the tensorflow package, but based on your experience it sounds like this feature is detected during the build automatically.

I hope that issues like this get better over time thanks to projects like Trustix, which would make non-reproducibility like this more apparent.

> Ideally enabling avx512 would be an explicit input to the tensorflow package, but based on your experience it sounds like this feature is detected during the build automatically

Looks like it sets e.g avx2 on the flags, forcing the package to be most compatible, thus removing the hardware state (the builder may or may not have avx512, ideally Nix packages should remove hardware autodetection to make it pure and consistent in face of cross-compiling).

It should indeed have an input in some way, e.g to add more flags. Then AIUI (still learning Nix) one would be able to call the package function with that input from the dependent package function, thus defining another package than the default one, which would be reified as its own specific derivation for that package to depend on.

https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...

I think part of the problem ist that derivation hashs sometimes don't fully cover the intermediate states of a derivation during the build process. This might lead to two hashs pointing to effectively two different configurations. I've had that experience in particular with non-reproducible derivations.

Even one of those in the store will make the store, or at least a subset of it, a state.

Hopefully this sort of issue will be addressed more satisfactorily when content-addressed stores find general usage.

Huh, is this what Nix users mean by “impure”?

If so, then it makes so much more sense to me now.

Yes, but 'pure' might mean different things at different levels.

In a programming language, 'pure' would just refer to "function where you get the same output from the same input; no side effects".

In parent's case, there was an 'impurity' such that the package is meaningfully different depending on the machine it was built on.

Author here.

Thanks for the clarification! I'm still new to Nix, so I'm trying to share useful things I'm learning without overstepping my expertise and saying wrong things.

My mental model of Nix was that if I'm in system state A, which is the result of performing task X + Y, and I want to get to system state B, which is the result of task X + Y + Z, then Nix would recognize that it's already in state A, so it only has to perform task Z to get to state B.

It sounds like what you're saying is that Nix's actual behavior is that if I'm in state A and tell Nix to bring me to state B, then Nix still performs tasks X and Y again, but the results are cached, so tasks X and Y feel as fast as a no-op, and the only task I perceive as taking time is task Z.

Is that right?

Yep!

Nix builds don't know anything about state, and they work just like you describe. Nix builds power basically everything you do with Nix.

But for 'installing' packages, more happens than just creating builds and leaving them somewhere in /nix/store. In those cases, you also have a profile manager (nix-env, `nix profile`, nixos-rebuild, darwin-rebuild, home-managwr) which builds a symlink forest (pointing into the Nix store) that represents your complete configuration. That forest is called a profile generation, and represents how, e.g., your user's Nix profile was configured that particular time. (When you perform a rollback, your profile manager is just setting the current profile to a previous generation.)

Each generation of your profile has an activation script (and some profile managers may also have some bit of profile activation logic of their own, idk). That activation script may have to perform some state management, e.g., restarting a service or telling the service manager (systemd on Linux or launchd with Nix-Darwin) to reread its configuration files to recognize the availability of a new service.

This little bit of state management is hopefully as minimal as it can be. In fact, it's generally expected/communally enforced that Nix packages have to work normally when run directly from their respective residences in /nix/store, without having been installed into any kind of profile at all!

So during normal daily Nix usage, I've never really had to think about it. If you want to implement or contribute to a profile manager, you'll have to!

That's largely correct! To use more precise language, let's discuss two distinct phases: building and activation.

Nix will always "build" x, y, and z. But x and y might turn into a no-op by being cached in the nix store.

System activation is when all those things are symlinked into place. If x, y, and z are, say, systemd services, then there might be some logic that checks if x and y have changed and if the services are already running and decide what to do -- this is probably the most similar to how Ansible works. But this is also a small part of the big picture. Activation is pretty fast even starting from nothing.

Ah, okay. That makes sense. Thanks!

I've updated the post based on your feedback.

When it comes to building derivations, yes. Which describes almost everything you can do.

For live system rebuilds in NixOS, the final step involves examining the system to decide which systemd services need restarting and restarting only those; that’s part of the process called activation. But that’s the only exception, and doesn’t happen if you reboot instead.

Memoization is state. As a functional programming maximalist myself, I know it hurts a little bit, but still, Caching/Memoization is statefulness.

That said, you are right that as a user of the system, that statefulness is abstracted from you and you don't have to worry about it (until some subtle caching bugs forces you to dive deep in the rabbit hole)

Yeah I really bumped on this. A cache is absolutely state. It is sufficient to say "a referentially transparent cache is a technique to store state while avoiding most of the drawbacks typical to storing state". This is already a big accomplishment! No need to also redefine the word "state" to make the technique seem even more magical than it is.

It's state at a different level than most configuration management software deals with. When Puppet reasons about the state of an existing system, it asks things like 'what's on the PATH right now?'. NixOS doesn't— it reconstructs the directory tree under /run/current-system/sw/bin without asking that. Same with 'modification' of files that live under /etc. What you and the GP write here is nuance worth admitting, but I wanted to highlight what people are getting at when they say Nix's approach is 'stateless'. What they mean is that Nix's approach to state is really qualitatively different, and hands state off to be managed by a lower level part of the system as much as possible.

Reminds me of this more abstract perspective: you can draw an analogy between working the filesystem, and programming languages working with memory.

https://www.tweag.io/blog/2022-07-14-taming-unix-with-nix/

Quibbling about what is and is not state isn't particularly productive. There is intrinsic and unavoidable subjectivity in the definition of state. The common pattern is to ignore out-of-memory errors, completely ignore the fact that in the real world time-of-execution is irreducibly an observable side effect, and that whether or not a given computation completes may even depend on whether or not an entirely separate OS process has or has not consumed the memory on the system. There's already non-zero pragmatism in the definition of "statefulness", and adding "whether or not the state of the disk has changed as a result of this 'pure computation'" isn't exactly an impossible stretch. It isn't a binary characteristic; when you really get into the nitty-gritty it is unavoidably a continuum. All real systems have state of some kind.

I'm not quibbling about what state is - the author is very much confusing Nix with something like a control-loop based approach that converges on the desired state by examining the current state. Regardless of your definition of state, that's not what Nix is.

Nix is only fast if your time has no value :D

It is not incredible fast and while it is running you can do something else like reading hackernews but there is definitely room for improvement and smart people are already thinking about that right now.

Functional?

Just in case: https://en.wikipedia.org/wiki/Functional_programming

I mean how Nix uses FP?

Did you read the entire comment? The core idea of functional programming is pure functions (given the same outputs, the output is always the same). Nix applies that to building systems.

Pure functions are just a concept that is used in FP world

Yet, pure functions can "stand alone". You do not need FP to use pure functions. They are independent from FP. You can be OOP maniac and still use them.

So unless you want to sound fancy and trendy then why call Nix functional instead of side-effects free?

Because it is basically a dynamically typed Haskell? It also does currying.

Like, there is no one definition of FP, but Nix is definitely an FP language as well as the whole idea behind it.

> You do not need FP to use pure functions. They are independent from FP. You can be OOP maniac and still use them.

This is not true. OOP is fundamentally built around impure operations.

Objects are persistent references that you send messages to or that you call methods on (depending on your OOP language of choice).

A persistent reference that is stable across different invocations (as opposed to a new reference being created on each invocation) requires that those invocations be impure operations, because the same invocation performed multiple times must have different effects, otherwise there is no point to having a stable reference.

That's not necessarily a bad thing, but OOP is fundamentally impure.

(If you do create a new reference on every invocation, then you no longer have OOP. You merely have a namespacing system where the first argument to a function can alternatively be written with a dot).

> You do not need FP to use pure functions

I think this is a bit more subtle, and a bit less interesting to say, than you're thinking it is and responding to. You need to be impure in OOP world but not in every function you write. I write many pure functions in my OOP work; they make the whole thing easier to reason about.

Sure, but it's precisely those places where you're using pure functions that aren't object-oriented. That is it is possible in some nominally OO languages to write code that is not object-oriented and that's where pure functions come in. In a language that has only classic OO-style objects pure functions (and their counterpart in data space, immutable data structures) are generally not used or are not very idiomatic, e.g. as in Smalltalk (you will have things that are kind of pure, but the purity can almost always be broken in some way).

You definitely can write method CalculateSomething which is both: pure and oop viable

Sure in the same way you can create an object in FP by sticking functions in a data structure or create functions in a logic programming language by constructing certain relations.

Heck you can even build "objects" in C.

And all of these methods are indeed used in those languages.

But none of those capture the spirit of object-orientation, in the same way that CalculateSomething is not object-oriented, unless it in turn generates a mutable object. That's not to say it's bad code or that it is uncommon, even in say Java. Simply that it's not object-oriented (and these days Java is getting less and less object-oriented anyways, with records and pattern matching explicitly separating data from functions).

Indeed in Java materials and blog posts from > 5 years ago, using pure methods of the form `CalculateSomething` was generally something used begrudgingly, where too many of those methods was specifically called out as an anti-pattern (e.g. the dreaded "utility class").

How is "side-effects free" less fancy and trendy than functional?

Reply on news.ycombinator.com