Tvix: We Are Rewriting Nix
tvl.fyi
tvl.fyi
That being said, in perusing related issues on GitHub such as [1], I have found no serious attempts to even profile the damn thing. The typical attempt seems to be throwing a random idea at the evaluator and crossing fingers. The example of a "performance experiment" [2] from the article is self-described as follows:
> Add an alternative impl of the now-abstract Bindings base class that is backed by a std::vector, somewhat similar but stylistically a little superior to the array-backed implementation in upstream nix.
I fail to see how that is a performance experiment when it does not mention performance at all.
The post also mentions issues with the general Nix design and a desire to modernize it, which I am unqualified to comment on, but seems a worthwhile endeavor in and of itself. I do hope that they won't end up writing a new evaluator for the Nix language without a proper understanding of what makes the current one slow and a clear plan not to repeat its mistakes, or the new one risks being just as slow as the old one.
Once we get further along with the evaluator we will be incorporating some of that material into more specific blog posts, but it seemed a little too deep for the initial announcement.
Regarding that CL linked from the post: It was part of laying the groundwork for trying to have the same Nix language type backed by different implementations; the reason for this is that Nix's attribute sets (similar to dictionaries or k/v maps in other languages) have a wide variety of use-cases with wildly conflicting performance characteristics.
For example, one very common use-case is the data structure representing the top-level of nixpkgs: An enormous attribute set with many thousands of keys that is frequently modified & copied. An even more common one is related to the builtin function `listToAttrs` which deals with two-key attribute sets (name, value pairs). In current Nix, these are some of the biggest evaluator hotspots (apart from higher-level issues like cache-unfriendliness due to the tree-walking evaluation style) and optimising for one case vastly diminishes performance of the other.
The problem we ran into with all experiments of this form is that there is no clear interface separation in `libexpr` (the part of the Nix codebase that implements the evaluator): Everything is modifying and reading internal state of everything else, including in other parts of Nix that depend on libexpr. It made it basically impossible to really try anything without refactoring huge amounts of (memory-unsafe) code.
Now it's important to mention that the reason for this is understandable: Nix is an old project (18 years now I believe?) and I don't think anyone expected it to get to where it is now, so obviously different decisions were made in its implementation than would be made if it were started today knowing everything we know.
It's good to hear that you did do some profiling, even if the results are not public! Of course I would not expect the details in the initial announcement, but I would have expected a mention of this work. At least I have a much higher confidence in the success of this project now than just after reading the announcement.
Switching between the proper implementation of high-level concepts is a staple of untyped (and sometimes typed) runtimes, so in this context the changeset makes more sense than simply "replacing an array-backed implementation with an std::vector-backed implementation" as described, thanks for the explanation!
I'm also not putting any blame on Nix --- the fact that the performance of the evaluator is such an issue is in a way a testament to its success. Unfortunately large, long-lived systems do require active care to enforce proper boundaries between subsystems, and it's a shame if it has got to a point where it's no longer possible to reinstate that separation.
As guix has shown, nix principles can be applied as a DSL/library within your favorite language of choice. I would be curious to see what a "rustix" would look like!
As a contributor I actually think that the success of projects like TypeScript have shown that there should be a way to build a type checker for something like Nix, with incremental migration. But it probably isn't going to be the ideal solution we could all imagine and it will be very very difficult nonetheless. But it could work with the code we have now, and that's a very important bar to clear (I don't take some of the other suggestions like "auto translate 30,000 packages to some other better language that doesn't exist & isn't the one people are using now" all too seriously, I'm afraid.)
I do think having a store implementation (even a "virtual one") decoupled from the evaluator and builder tooling is a step in the right direction for experimentation and hope that it works out.
However, it's good to see an ecosystem forming around the derivation data type, with the possibility of reuse across projects that use derivations.
Good stuff!
I'd be interested to hear a perspective from the Nix and Guix authors.
Guix used to use the same derivation format as Nix, but they've since diverged
OTOH, Guix operates on in-memory representations of profiles, packages and derivations, and delays using the store until an actual build needs to be performed; whereas I believe Nix keeps making RPC's to the store during evaluation (please correct me if this is wrong).
I don't think adding Guix support to Tvix is as easy as plugging in a Guile evaluator, as the Nix tooling would not be able to understand the higher-level abstractions in Guix. You really need Guix and all its supporting modules to evaluate Guix packages.
Then, you could generate a new version of the entire 30,000-strong package collection automatically, with the new language swapped in in place of the Nix code, which could then be abandoned.
It is highly unlikely that this is feasible. I don't know nix in detail but I infer from your post that it is an untyped language. Untyped languages can, in general, not be translated to (statically decidable) typed languages.
> which could then be abandoned.
It could not, unless you convince a majority of the contributors to the package repo to switch to your new language.
Some more complex nix features would not be easy to transpile automatically, but many (i don't have stats sorry) packages are very simple declarations which could be ported automatically.
For example, all language-specific libraries imported from rust/python/etc could very easily be automatically ported, and in my experience, writing declarations for those many dependencies is what takes most time in making a small package (although admittedly i only tried once).
You don't really know this - there are many code paths in nixpkgs that are never touched during Hydra evaluation (various overrides, user-facing flags, packages marked broken and so on). We think it might be possible to slowly move towards more static analysis of the existing Nix language, but this is a little bit further down the road.
(Note by the way that we have a "runtime type checker" for Nix, but it is mostly an experiment: https://code.tvl.fyi/about/nix/yants)
A less ambitious approach would be creating/extending a static analyser/linter for Nix (there are already a few out there). This could nudge packagers towards certain styles, which language implementations could optimise for.
For example, an interpreter could have multiple implementations of the attrset interface (key/value mappings) with different performance characteristics, using some default for literals like '{ foo = "bar"; }' and a specialised version for known use-cases like nixpkgs.lib.nameValuePair ( https://github.com/NixOS/nixpkgs/blob/e9e53499b26ad98ac97f97... ) (mentioned in a sibling)
Linters could then spot when such functions are appropriate, e.g.
Found literal name/value pair on line 123:
{ name = "x"; value = "y"; }
This can be slow, consider using:
nixpkgs.lib.nameValuePair "x" "y"'Quick question. Can you expand on IFD and integrations? That is one thing i would love to know more about your plans.
Also i hope it works, the nox code base is imho the biggest impediment to the nix model being more used.
Small advise: please consider making it really easy to contribute to through wll the tooling. This is one of the reason Nix codebase rotted
Yes, that's one area where Guix shines in comparison to nix. They use an established language with its standard library (Guile lisp) making it very easy for people to contribute without learning arcane incantations.
I would love to see nix/Guix implemented in a statically-typed, high-level language like Rust.
I'd go even further: Nix is just a build tool, like Make.
In fact, it assumes far less about the system than Make; e.g. Make does all sorts of shell-related things, whilst Nix simply executes a binary (with a list of arguments and a set of env vars).
(Nixpkgs definitions almost always use a bash binary as their executable, so it feels more like Make; but Nix doesn't care)
https://nixos.org/manual/nix/stable/installation/supported-p...
- Nix itself has problems of compatibility with Nix! Let me explain...recently NixOS 21.11 released. It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility issues. When the Nix project itself is not able to deliver backwards compatibility I feel uncertain that Tvix might be able to deliver their promise of full compatibility with all the deep architectural changes it plans. Nix does not have a formal specification: Nix is what its C/C++ implementation says. Nix also has deep architectural changes planned going ahead. So Tvix will need to re-architect Nix keeping in mind (and keeping up) with the future changes that are coming. It's going to be tough to build this "alternative" implementation. Users are going to run into weird compatibility issues, not have the patience to resolve them and then just go back to the original Nix implementation.
- A rewrite to a popular project will find it difficult to succeed. There is simply too much momentum in the parent Nix implementation (I'm not talking about the package definitions here) that very few people are likely to adopt this rewrite. This is because most people might not really need the speedup. However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here.
- The rewrite needs to offer more reason to shift. Right now this rewrite seems to be mainly about performance. There is also a proposal to be able to integrate with GNU Guile. To me this is just avoiding one untyped language (Nix) to replace with another slightly better untyped language (Guile Scheme) (GNU Guix uses Guile also AFAIK). What about introducing typing? https://github.com/tweag/nickel is an interesting project and it would be great to have a system like Nix with optional typing to make writing the derivations more robust. What about rewriting in a safe programming language (e.g. Rust or a GC-ed language)? These things would be likely to excite would-be-contributors and would make your rewrite viable. In other words, the rewrite needs to offer more than just the promise of performance.
Perhaps I have not understood the project goals fully. Maybe the project is all about just being able to leverage the Nix package definitions but have a totally different implementation to instantiate them? But the issue here is that nix-the-language is very intimately tied up with the nix-the-platform and the rewritten implementation will still need to really be extremely compatible to capture all the implicit assumptions in nixpkgs.
Once again, improving nix through a rewrite is a noble goal. It's a tough problem and it would be awesome if the project succeeds!
I have obviously not thought about this as much as you guys and would love to learn how you wish to tackle some of these problems. Maybe some of these issues are not as big as them seem??
Also as someone who likes simple FP but is not some sophisticated typelevel programmer, Dhall looks really complicated and verbose. To me, the Nickel approach, where authors of libraries in the language (equivalent to things like nixpkgs.lib) can pepper their functions with type annotations, but users can use what looks and feels like a simple configuration language, seems like a better approach for the domain.
I can't imagine getting all of the PHP developers I support comfortable editing the equivalent of shell.nix in Dhall themselves. Nix files I can ask them to edit without taking up too much of their time or focus.
I would also like that, but for the exact opposing reason. I love the concepts of nix/guix but i can't stand the dynamic languages with cryptic error messages.
You can set up Nix 2.4 on NixOS 21.11 by adding
nix.package = pkgs.nix_2_4;
I like the new CLI, too, and it's worth trying if you want to see what all is new in Nix. Here's how to enable flakes on NixOS, if you have Nix 2.4:
nix.extraOptions = ''
experimental-features = nix-command flakes
'';The reading I'd recommend as someone sympathetic to both 'sides' isn't really a technical comparison, but this: https://grahamc.com/blog/flakes-are-an-obviously-good-thing
As far as technical pros and cons, it's worth looking at colemickens' PoC demonstrating that in principle, you can use Nix to generate literally identical output with and without flakes: https://github.com/colemickens/nixos-flake-example
> Both approaches
it's also worth noting that prior to flakes, there was no single alternative approach to the main problems flakes aims to solve. There were (and are) a whole bunch of competing and bespoke practices for pinning nixpkgs and other Nix sources. You can take a look at niv and nix-thunk to get a sense of them if you want
Yep, and this causes a whole lot of issues. In tandem with evaluator development we are writing an initial language specification which will represent the language as it is used in nixpkgs. Our hope is that we can, in collaboration with the Nix developers, eventually find a fixed point between the two implementations here.
> It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility
Yep, we're very aware - in fact we had some debate on IRC yesterday about whether or not to touch on this in the blog post, but felt that for the majority of people it's a bit too deep into "current Nix politics" (for lack of a better term).
This is why we have explicitly committed to nixpkgs support - many of the experimental features in Nix (some of which are causes for the new incompatibilities) are not actually used in nixpkgs, and they might have a long road to go before getting there. We believe that this gives us enough time to get to a synchronisation point.
> However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here.
We'll see, on the corporate side of TVL (https://tvl.su) we've had some discussions with a variety of Nix-using companies and there are strong signals that some of our plans would be a big reason for them to switch (or possibly even just partially switch). We want to not worry about this too much this early, to be honest.
> The rewrite needs to offer more reason to shift
Yes, there are more things that will become relevant but felt like they were outside of the scope of this initial announcement. For example:
* Implementation will be in memory-safe languages (evaluator in Rust; we will see about things like the sandbox & builder - Go might fit that reasonably well). This is a big point for companies that are worried about running current Nix (which is fairly easy to segfault) on production machines as root.
* We do intend to have more language-related tooling (including various kinds of static analysis, like dead-code analysis or maybe even going as far as an Erlang-style success typing approach)
* Streaming evaluation: We want to yield derivations while evaluation is still ongoing, essentially turning a whole build process into a sort of graph reduction. This has a huge impact on our ability to implement better Nix-native CI, which is very relevant for TVL's ambitions around better monorepo tooling.
It is also worth noting that a bunch of TVL members are nixpkgs committers, and as we proceed we would like to clean up bits of nixpkgs that use features which only exist "by accident". (An example of such a feature, which fortunately is not used in nixpkgs now, is __findFile: https://cl.tvl.fyi/c/depot/+/1325).
I appreciate that it's kinda difficult to serialise all of our thinking here in a handful of comments and a blog post, so we might not get the full point across. That's okay for now. If you're interested in following along more closely and discussing some bits of this, you're welcome to join our IRC channel :)
Oh wow, wait till they find out what language the OS on their production machine is written in!
[0]: https://github.com/ivmai/bdwgc/blob/master/doc/README.enviro...
It seems that North Americans struggle with the word, but most Europeans (including Brits) don't. Don't know enough about linguistics to guess why!
Try saying "Twix" the way you'd imagine a German or Scandinavian accent to sound.
Yes, definitely!!
I think it wasn't clear whether you were supposed to say "tee-vix" or /tvɪks/.
For non-Russians, try and figure out which of the above two is right. No one help them, because you'll be taking away a learning opportunity. https://www.youtube.com/watch?v=9hNphbAm_Hs
Sorry, but Nix is a total waste of time.
That said, you underestimate the power of being able to hack on any dependency -- for real purpose of Nix is to defeat Conway's law, allowing any person to work on any codebase. It would be quite easy to fork the linker/loader to try out some experiment, or even just do some per-package symbol mangling, for example. I wouldn't want to try that experiment with any other distro, because it would be too hard to make my modification and then put everything together again!
Back when I was a Gentoo user, I too bought the story that being able to re-emerge my whole system was some kind of performance or configuration boon. Eventually I realized that I just wanted to use my computer to get through life, that literally any operating system would work, and that any experimental hacks could simply be done when they were needed rather than trying to manage an over-complicated, poorly-supported hobby OS.
I use Ubuntu now. It sucks. But I spend my time on doing real world things now, rather than constantly tweaking my computer to eventually do a real world thing.
For work I use containers with any distribution at all. They all have the same result in a container.
> I too bought the story that being able to re-emerge my whole system was some kind of performance or configuration boon. Eventually I realized that I just wanted to use my computer to get through life...
I don't have fun constantly mucking with my settings either. That novelty long wore off long ago, like with most people who have used Linux 10+ years.
I do however get a benefit being able to set up my same config on multiple machines. For example, recently, I switched from XMonad to sway on my work computer, which necessitated a bunch of config changes. I then synced those changes over to my personal machine with https://github.com/nix-community/home-manager .
C.f. my phone just died, and I got a new android, and the android backup/sync situation gets worse and worse, and it will be a huge pain in the ass fixing my phone so i can hopefully move a few more account settings, 2FAs, etc. over.
The moral of the story is that when I do need to configure something, I want to do it once. Sysadmining repetitively is infuriating.
-----
The linker thing is quite different. I still do like programming (as opposed to sysadmining) for fun, and I want the tools to suck less. I also find infrastructure fun and interesting, be it computer or real world. I am a fairly prolific contributor to the Nix ecosystem, not because what I necessary want to do by myself on my own machine, but simply because I enjoy making the stuff better (and this hobby also sometimes becomes current task on the job, too).
Nix starts from a strong baseline of "sound" builds --- anything that is not sand-boxed, and therefore unlikely to be reproducible / cached correctly now pisses me off a shitty software that doesn't respect by time. The goal is to build on that foundation and also make Nix builds more incremental, so we have both a reliable and tight debug loop. Ultimately I want to see my work and the Nix community at large swamp the rest of the computing world with the productivity that is unleashed and when we stop crippling ourselves with shitty tools.
The flip side is that since goal is social, I have 0 interest in solo dog-fooding. I only use my PRs which I merged upstream; whatever benefits they would bring are certainly outweighed by the costs of living on forks all by oneself.
-----
If not being totally relatable (I rarely find people interested infra changes that don't immediately benefit them) hopefully it is at least clearly distinct from "building out my person nerd terrarium" customizing the computer as individualistic self-expression.