tomberek

Nix (and by extension Guix) has had remote builds and signed build provenance for nearly 2 decades. It has also had a story to deal with both content-addrrssed and input-addressed (this is our term for the "stable identifier" for a workflow mentioned above).

I can understand not using Nix, but please take a look at how we've solved some of these problems, and our mistakes. There is no need to reinvent the wheel blindly. Feel free to reach out if interested.

Look at how much room you have for data! I wonder what we can fit in there.

More seriously, a metacircular example to draw from would be: https://github.com/kragen/stoneknifeforth

This was my first thought, but this connection was not made clear in the article - or if it was a naming coincidence.

Especially with the remote SSH features one needs a way to ensure everything needed for your process is on the target machine; Nix makes this easy.

Nix + GDAL + GNUParallel + autoscaling groups === massive geospatial data processing pipeline

Generally what we see people do is:

- build in CI

- push to a cache (eg: S3-compatible/HTTPS/SSH)

- use the cache elsewhere

In that paradigm, the artifacts live in the binary cache (sometimes called substituter), much like container images live in a registry. This raises the question of if or how to track precise artifacts in use. There is no standard for this yet. I've seen people use tags, branches, separate repos with manifests, DNS TXT records, etc. [ I work for https://floxdev.com/ where we are creating a standard for artifact lifecycle management and distribution called the Catalog into which you publish your artifacts, contact us if interested ].

The key selling point is that the resulting artifacts often have high de-duplication compared to container layers and have good correctness/completeness properties. We also can keep track of how things are built, so SBOM and supply chain information is very good.

I never understood the concept of boarding from the front. I would never want to be the first on a plane; regardless of status. And this means that every single person passing them will bump, cough, shuffle, past those that boarded first.

I guess it's just that people would feel bad going *last*? Psychology is weird - but powerful.

30 seconds:

Can you build/test/run your organization's software from 5 days/months/years ago - across multiple platforms, systems, and ecosystems - reliably?

Can you develop, build, manage, and maintain your software without fear?

Do you want a system do the bookkeeping for proper software dependency management? Or do you want people to do it by hand?

Do you want a declarative system, or try to remember and keep track of all the changes you've ever made by hand?

You don't get these things for free, but if you value them, then Nix is the tool for you. Or at least learn about it so that you can learn from the decades of accumulated knowledge.

next 30 seconds: I can confidently compile and run multiple incompatible versions of software simultaneously. I can build projects from years ago. I can package large projects from different ecosystems (python 2/3, c/c++, go, Javascript, Octave, Rust) and be confident they will not interfere with each other. I can try bleeding edge software with no risk to it interfering with my system. It is faster and less hassle than juggling various Docker containers and VMs. It protects me from dependency hell. Using Nix to manage an operating system (ie: NixOS) then gives me those benefits for my entire machine. Then I get even more benefits when using Nix on any other (not even necessarily NixOS) machine by pushing around package closures and managing a simple binary cache.

But overall: it makes me more productive. It is my secret weapon to manage the complexity of software development. The benefit is not on day 1, but on day 100 and day 1000.

There are some efforts underway to address the people problems in the Nix community; empowering the various teams, spreading out knowledge among more people, making it easier to participate. In fact, this is a current topic of discussion; trying to better understand the blockers and address them: https://discourse.nixos.org/t/where-did-you-get-stuck-in-the...

I would donate a few beers and pizzas to such an effort.

Stabilization of flakes and certain other experimental features is a top priority for the Nix team at the moment (disclaimer: myself included). Please take a look at RFC136 for an outline of our plan for the way ahead…… and please comment.

At flox, we are specifically focusing on those people who are the advocates and bringing Nix into their organizations, teams, and companies. We call them the “Ambassadors”, and want to call out how important that role is in making Nix mainstream. Ideally, it should become the obvious choice - the default choice - where specific reasons need to be provided when not using it.

It’s funny; I forgot to mention sandboxed builds because at this point I’ve forgotten that they are not ubiquitous. Oops. Thanks!

Is any of this available or would you be willing to share or discuss? I’ve done some HTMX/_hyperscript for prototypes and hobby projects and loved it; contributed a few of the cookbook examples. Overall it is a pleasant ecosystem. I’m curious how this worked out in your use-case because I’m considering it myself. What are the hurdles? Any lessons learned? Recommendations?

Other than the very explicit tracking of the packages, their dependencies, the full build instructions, the public history, cryptographic signatures for the pre-built binaries, and the trivial ability for anyone to re-build from source and audit the entire chain all the way from bootstrap if they wished?

I’m not sure what golden standard we are comparing this to. It is not perfect, but I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.

Translating from Nix to another packaging ecosystem usually entails removing specificity. This is why going in the opposite direction is harder. So I suspect we can provide escape hatches from Nix into Debian/RPM/Arch/Gentoo so that one would only have to maintain one fully specific package, but get easy translators for the other ecosystems.

Would you be interested in a session for me to better understand (and hopefully eventually fix) why Nix was not comfortable? Not looking to evangelize, but to learn about the experience from your perspective.

Excellent. Any custom modifications can go here: https://github.com/NixOS/nixos-hardware/tree/master/framewor... to make it easier to collaborate and distribute.

I’ve loved my Framework so far and the excellent customer service. Is there still enough attention put into ensuring Linux compat? I’m especially interested in providing NixOS support as the Framework has been popular in that community.

My biased suggestion and would benefit from your C++ understanding is the Nix community (https://github.com/NixOS/nix).

More frankly, “best” is very subjective and you should find a passion or topic or task you are already interested and use that to guide your decision.

I’m looking to put one together in Bay Area for next week or so. DM me if interested or if you know more people in the area.

There are two sides to this problem, the first is to improve the UX, but the second is to clearly describe a compelling reason for people to adopt. It is very tempting to only blame the first, but I think we need to also need to tell a better story and highlight the values in a better way. This would then give people a reason to get past the UX issues in the hopes of achieving those desired values.

For example; people seem to have accepted that the benefits of using terraform in spite of various difficulties - the learning of its language or needing to hire specialists. The mantra of "infrastructure as code" is enough to drive adoption. What is our mantra? We need to accept that "reproducibility" isn't quite working and that we need either a clearer message, or to explain the message.

Be able to start with a broad understanding and be able to dig into specifics for your area of expertise. Be able to explain the connection between the larger scope and the details. Take a look at the values (https://www.usds.gov/mission); this org actually uses the values in day-to-day and are looking for people who can exemplify them.

(disclaimer: i was on the Defense Digital Service team)

This usage is very similar to how people sometimes use (or abuse) Makefiles to create run-scripts. Nix's special gift is that you can talk about external resources - such as packages - and safely bring them into existence.

I'm reading these.

It seems this is where a better description of what NixOS (and it's module system) is. My suspicion is that by starting with NixOS it is harder to be introduced to the simpler concepts. It is best to think of the NixOS modules as a domain-specific-language on top of Nixpkgs which is itself a large set of conventions on top of the fairly simple Nix language. I think people should be introduced from the bottom up.

> having systemd manage a StateDirectory and creating the unit file/Nomad conf in configuration.nix for the machine

Seems like you did get what you needed working; my best advice to understand that the key abstraction being created is the activation script. Perhaps we never clearly describe all of its moving parts and how it works?

> ... but they are NOT functional programming enthusiasts

I believe the key is for the Nix community messaging/marketing to focus less on the theoretical elegance and more on the practical benefits.

1) what would you consider to be "wow" or magic that would go beyond a+b. What is steps c+d that would be incredible value?

2) We're working on stabilization... just asking for a bit of patience.

I am working on the Nix project; resolving this "in-between stage" is our highest priority at the moment. I'd like to better understand two independent things:

1. What were the drivers and motivators for you to try Nix? What did you hope to achieve?

2. Was the current experimental status of the new nix CLI and flakes the primary cause of the confusion?

Best seems to be when you can transition between various representations smoothly and intuitively; between iterative development, isolated builds, containers, deployment and so on. The design difficulty is how to craft a good user experience along the way.

We're always looking for better ways to tell this story; what would be a good way to demonstrate this value? what is a quick demo that would convey the message?

It would be helpful to teach people without so much being done by mkDerivation or other builders. While they provide value, they sometimes block learning. I like start with super simple example, if possible.

As we make flakes stable, this will improve as the docs can prioritize this information, and tooling can focus on making this more clear. On a much longer horizon, future Nix language (eg: Nickel) improvements provide a gradual typing system. On a short-term, I'm trying to provide some opinionation and a framework to describe this (https://floxdev.com/docs/custom-packages/) . The challenge is to provide enough simplicity, but not so much that one loses the very benefits we are trying to achieve.

We've seen the opposite. New package managers are built with a much better understanding of the problem space because of Nix and sometimes explicitly taking ideas and concepts. This results in those new systems being far more compatible and easily integrated with Nix itself. Consider the iteration towards better and better locking in python + Node. Consider Rust/Cargo.

There has also been sharing of ideas and collaboration between things us and bazel/buck/spack/etc. I think it is clear that we are all moving towards a similar end state, that often looks very much like Nix, or a re-invention of it.

For things like OCI, Nix predates it, but we should start to re-use the standard. The Tvix group is exploring this and I'd suspect our sandboxing will be OCI at some point. For outputs; we make it easy to convert a Nix package into an equivalent container. Things like flatpack,AppImage,Snap should also be easy for people to output (i'm trying to collect these "transformers"/"bundlers" here: https://github.com/NixOS/bundlers).