Tvix – A New Implementation of Nix | Hacker News Reader

Tvix – A New Implementation of Nix

tvix.dev

88 pointsvngzs105 comments

Maybe worth mentioning rust and performance more prominently as the motivations.

Is Nix eating the world? We have seen lots of unbelievable projects using it on HN recently. It feels like anyone serious about complex systems and building software are embracing it.

I use it extensively also via nixpkgs, though somehow I have yet to make the leap to full NixOS.

One consideration for anyone who hasn’t jumped in, is that I have nixpkgs on Linux, MacOS, and Win11 hosts. I’ve found that incredibly useful.

Can you provide some links to some projects? You’ve piqued my interest

Tow-Boot comes to mind for me. https://github.com/Tow-Boot/Tow-Boot/

Also plenty of projects here: https://github.com/nix-community/awesome-nix

https://github.com/majbacka-labs/nixos.fi

https://news.ycombinator.com/item?id=40010991

https://news.ycombinator.com/item?id=39720007

https://news.ycombinator.com/item?id=39692801

https://news.ycombinator.com/item?id=39658684

https://news.ycombinator.com/item?id=39004785

I didn’t find these via HN search, I found them in my bookmarking account. I have a slightly strange way of bookmarking - I send emails to a dedicated email account, so these I found there looking for Nix in a search.

do you use special email client and host provider for that?

does it work for 10k bmarks?

Nope it’s just a regular email provider with IMAP support. I just send myself an email with the link and optionally some cut and paste content or notes. I have offline backups of all my IMAP accounts also. I find it convenient because I just add that account to whatever mobile or desktop environment and then can use search functionality of email client. It’s a bit silly but it’s worked out ok for me for many years now and yes definitely works for 10k emails of bookmarks, no problem.

I would not call these projects unbelievable, but they are neat.

Opt-in state:

https://github.com/nix-community/impermanence

https://grahamc.com/blog/erase-your-darlings

Ease of setting up a real-time audio on Linux:

https://github.com/musnix/musnix

Generating virtual machines/installators from a configuration:

https://github.com/astro/microvm.nix

https://github.com/nix-community/nixos-generators

It's interesting to scroll through nix-darwin options, I wasn't even aware of some useful macOS options before:

https://daiderd.com/nix-darwin/manual/index.html

A neat collection of music information retrieval packages in one place (and the ease of creating your own package registry):

https://github.com/carlthome/mirpkgs

Not super recent, but nix-snapshotter is one that I'd call awesome(but I'm also a k8s fanboi): https://github.com/pdtpartners/nix-snapshotter

https://news.ycombinator.com/item?id=37407758

There's also nix-darwin and home-manager. I barely use brew anymore, not directly anyway, for the rare packages that aren't in nixpkgs.

One useful webpage this project provides is this godbolt for nix:

https://bolt.tvix.dev/

really useful.

Well, we have the ambition to make it a sort of Godbolt for Nix, but none of us are frontenders and it's not quite as visually impressive and easy to follow as the actual Godbolt. It does work, though!

We've also been thinking about ways to make nixpkgs available there. Adding `lib` at least in some fashion would be quite useful, I think.

Does not care about flakes and does not care about adding types, the most important omission in Nix. A minor performance bump seems unlikely to make me want to boil the ocean.

nix modules have types. in flakes they can be quasi statically verified

For somebody who hasn't used Nix yet, but heard good things about it, why use Tvix instead of Nix? The page didn't seem to mention the reason(s) behind the project.

It looks like they’ve factored out the main features of Nix into modules to allow for more flexibility regarding the implementation details of each underlying service.

Not ready, so not a good reason to use it yet.

I would recommend against using Tvix at this time, but if you are wondering why they wrote it, they wrote an article: https://tvl.fyi/blog/rewriting-nix

The goals they list are:

> Creating an alternative implementation of Nix that is fully compatible with nixpkgs.

No explanation needed.

> More efficient Nix language evaluation, leading to greatly increased performance.

The Nix language is evaluated whenever doing the Nix equivalent of installing a package, or rebuilding your configuration. Note that most of the time it's "fast enough" but the authors of Tvix are doing more with the Nix language (which leads into)

> No more strict separation of evaluation and build phases: Generating Nix data structures from build artefacts ("IFD") should be supported first-class and not incur significant performance cost.

and

> Well-defined interaction protocols for how the three different components (evaluator, builder, store) interact.

As it currently works, the Nix language is evaluated to something called a derivation, which is a specific format for defining how to build something (including a list of dependencies). The derivation is then used to actually build the artifact. This is about when you want the metaphorical snake to eat its tail and have the results of a derivation be more nix language to evaluate.

> A builder implementation using OCI instead of custom sandboxing code.

Hermetic builds require some sandboxing; Using OCI containers seems like an obvious choice, but didn't exist when Nix was first created.

> us evakyated

huh?

presumably meant to be 'is evaluated'.

"evaluated", but with a small key shift on qwerty

Oh, I see. I was having a hard time with that one. Thanks!

typo, fixed.

> A builder implementation using OCI instead of custom sandboxing code.

A complete non-starter, never in a thousand years am I gonna use something that requires a Docker installation for building derivations.

Podman is also an option.

No, I don't want "dollar store Docker" either.

Maybe once you learn what OCI is, you'll rethink that ;)

I know full well what OCI is. (Hint: it's a transparent attempt to be a NOT-docker and avoid infringing on whatever Docker Inc.'s latest monetization scheme is.)

Using this for what Nix is doing is absolute madness. Docker is madness in general, but dragging it into Nix's core is madness cubed.

"why yes, the open-source Apache-licensed implementation controlled by the linux foundation is just as commercial as the commercial thing! i'm so contrary and interesting!"

if you don't want docker-contributed code, here are some alternative implementations of the spec: https://github.com/opencontainers/runtime-spec/blob/main/imp... I believe nix could use one of these with some volunteer effort? Perhaps you could even be the one to purge the Docker heresy from the project!

(...in hindsight, i'm sorry for posting this, it's needlessly adversarial and picky. i was mad at something unrelated this morning and i shouldn't have taken that out on you.)

OCI is not an attempt for a useful, general-purpose, open containerization standard.

It is an attempt at an open-source "just good enough" Docker clone.

You understand the difference?

Would this be the equivalent of Mock (Fedora) or sbuild (Debian) pristine chroot builders? Containers are just fancy chroots.

The implementation of Nix's build sandboxing is outside of the core focus of Nix. The macOS sandboxing code is famously buggy. Seems like a perfectly reasonable thing to leave to a battle-hardened, standardized, outside implementation to me.

> > A builder implementation using OCI instead of custom sandboxing code.

> Hermetic builds require some sandboxing; Using OCI containers seems like an obvious choice, but didn't exist when Nix was first created.

Presumably this would make this implementation Linux-only, since OCI isn't natively supported outside of Linux?

Does tvix support targeting windows? And I do not mean WSL. iirc, the nix attempted a windows port but there were some hardcoded assumptions made about unix in the implementation that made it intractable, for example the location of the store. Do these issues also hold for tvix?

(Nix itself is slowly chugging along with Windows via MinGW - https://discourse.nixos.org/t/nix-on-windows/1113/108 and https://github.com/NixOS/nix/issues/1320 , for example.)

[not related to this thread]

working on a project that I'd love to run by you in the men's health space.

came across a comment you made on a previous post, so figured I'd reach out.

amitthakrarcoaching@gmail.com

The form of this message pattern-matches perfectly to spam, by the way, especially from an account whose only activity is to make essentially this same post three times; if you are in fact human, you might want to do something about that (such as being precise about which comment you mean, and why it's relevant).

I recently learnt about this: https://nlnet.nl/project/libnix/

A project funded by the EU to bring Nix to Windows.

(edit: typo and clarity)

Tvix developer here; we do have Windows in mind. Rust makes a bunch of things easier regarding this, but _not everything_. It's not a priority.

The location of the store is not really the main blocker in those sorts of situations, IMHO.

Looks like the dev team mosly speaks Russian? (Russian is native for me, but I've lived most of my life stateside.)

In Russian, "tvix" is pronounced, roughly, "tweaks" (but with a German - hard "v" - W sound), which to a Russian probably sounds super clever, given what Tvix is (Nix, tweaked).

In English though, how is it pronounced? I'd like to propose "t-f-icks", because it at least sounds like a candy we think the Lebowski nihilists might well have been ordering at a diner.

I pronounce it "tiv icks". Remember DIVX?

That works!

I think the more likely etymology is because it's from The Virus Lounge. Tvix = The Virus Lounge Nix.

My first thought was Tuvix from Star Trek Voyager.

Expect more like: T-Vix

https://memory-alpha.fandom.com/wiki/Tuvix

That is the actual etymology, but we're happy to retcon it to something more clever))

I would pronounce it tee-vix, because of the stock ticker symbol TVIX[0] which was a leveraged volatility ETN and very popular in its day. a little too popular - credit suisse delisted it a couple months after the initial market crash brought on by the pandemic caused it to skyrocket

naming is hard :)

0. https://www.thestreet.com/etffocus/market-intelligence/rip-t...

"betwixt" => twix => tvix

(kindof like tuh-veeex run together)

...as the candy bar etymology in their logo implies.

Привет) Almost everyone on the team speaks a language where this pronunciation is obvious (most prominently Russian and German, but it also works in e.g. Swedish).

I've noticed when talking to English native speakers that Brits seem to get it right away and say something close to "твикс", whereas Americans almost always say "tee-vix" ("тивикс") and need serious convincing work to switch over to how we say it.

So, how ready is this for general use? Is there anything still missing in the implementation?

builtins.fetchTarball is not there, among others, and from watching Matrix I think there are still some things in nixpkgs that eval slightly differently in tvix.

It feels like things are heating up, and "soon" is on the horizon.

Tvix developer here; correctness is still not guaranteed, there's nothing to use here except if you already understand well Nix concepts to pick parts and build stuff on the top of it and accept the inherent instability :).

By the way, is there some way to specify dependencies in C? Like requirements.txt in Python, where you put libraries names and versions and the user has to run a single command to install them (and without requiring root privileges)? Is this nix/tvix/guix an equivalent of pip?

Check out devenv.sh from the Cachix people, it allows you to list stuff like pip for any language, including services / postgresql extensions:

   { pkgs, ... }: {
   services.postgres = {
    enable = true;
    package = pkgs.postgresql_15;
    initialDatabases = [{ name = "mydb"; }];
    extensions = extensions: [
      extensions.postgis
      extensions.timescaledb
    ];
    settings.shared_preload_libraries = "timescaledb";
    initialScript = "CREATE EXTENSION IF NOT EXISTS timescaledb;";
  };
   }

One way to wrap your head around it is apt/systemd in a pip-like config file.

Interesting, but the word "containers" is scaring me. I wanted just a simple tool to download and install several libraries and not a full Docker/Kubernetes level system...

But it looks like a good choice for running tests in CI.

I think containers are completely optional and it can install and run everything directly.

nix eco has project(s) which instead building full images just plug tiny shim linked into nix store. nix-snapshotter. fast and small. oci images spec allows to do nice thing with nix.

devenv has container support if you're looking for it, but you can completely ignore it if (like me) you don't care

Hi! Devenv user here:

devenv.sh autogenerates Nix specifications of container images for each of the development environments you define with it, and exposes them in a Nix flake for you. But if you don't explicitly ask Nix to build those for you, it doesn't.

My team's most complex devenv environment passes through some packages from Nixpkgs, exposing them in the flake, defines 3 or 4 development environments ('devShells') and also exposed (and includes in some of those devShells) some custom packages that just live in that repo and not in Nixpkgs.

Our simplest just uses all built-ins and defaults, exposes no packages, doesn't have a custom flake... there's basically nothing to it. It just uses devenv/Nixpkgs built-ins to add OpenTofu with some Terraform plugins to the path and plugs a Terraform linter into the project's pre-commit hooks.

There is some support for running long-lived processes via process-compose, which is inspired by docker-compose, but we don't yet need it for anything so we don't use it. (In the past I've rolled my own solution for this with Nix, direnv, and supervisord. Today I'd probably just use devenv.)

What I love about devenv is how cleanly it integrates with the wider Nix ecosystem in a no-nonsense way, as well as its great docs. I like that it

  - it uses the Nix module system for configuration, based on a standard implementation (flake-parts) shared by other projects
  - what it builds in, i.e., pre-commit hooks integration, exists in a separate repo also usable on its own via flake-parts
  - the docs and the tool both support flakes-based setups, non-flakes setups
  - if you want to let devenv automagically set up a flake for you, you can, but you can also use most of its features with a handwritten flake.nix customized to your liking
  - while it exposes some configuration via yaml for simple uses, the full power of Nix is always available
  - it takes care of all the little details you'd have to deal with if rolling your own (e.g., garbage collection, direnv integration) in a sensible way

For me, as a Nix user, it does a good job of keeping simple use cases simple, it has low lock-in, it uses familiar configuration interfaces, and it doesn't have too much magic.

I hope that helps you get a sense of what using devenv is like for simple use cases as well as what the more complex stuff might look like!

Yes, Nix can do that. Another (simpler IMO) alternative is vcpkg.

Yes, write either a shell.nix or flake.nix file

https://nixos.wiki/wiki/Development_environment_with_nix-she...

It's possible, just beware that other-distro versions of Nix are pretty aggressive at patching themselves into your shell profiles, which is often undesirable.

> is there some way to specify dependencies in C? Like requirements.txt in Python, where you put libraries names and versions and the user has to run a single command to install them (and without requiring root privileges)?

I'd describe this as "(language-specific) package management".

"Requiring root privileges" is more/less equivalent to "packages get installed in a system-wide location".

There are some C or C++ specific package managers.. but, essentially what's missing is the ecosystem/culture for providing a packages for those. -- e.g. With the Rust, packages use Cargo.

> Is this nix/tvix/guix an equivalent of pip?

Nix and guix are system package managers.

Since they're powerful tools, one popular use case is for providing dependencies for C projects.

Nix practically requires root for its installation. (i.e. installing it without root may be possible, but I'd think it's impractical). Although once installed, it can provide packages without needing root.

TVIX was a media player made by Dvico. I used to have one a few years ago. You may want to check on trademark issues around the name.

https://a.co/d/hOzOcvu

Do you put the hash at the end of the dirname in the store?

IIRC, Nix names things like "$unreadable_hash$pkg_name" which makes tab completion impossible in the case someone wants/needs to go spelunking in that directory.

Tvix is compatible with Nix, thus it follows the /nix/store/$hash-$name naming scheme.

Fuzzy completion makes it trivial, although it doesn't help with the unreadable part.

Spelunking in nix store dirs directly is usually unnecessary, though. You can debug/tinker with your derivations in nix shell (or, confusingly as always, nix-shell).

It's my understanding that the ${hash}-${name} order is actually intended to help with tab completion, but from a different direction; if you already know the first couple chars of the hash, you can quickly tab-complete it, instead of using the less-unique name. I mean, I don't think that's the order I would have picked either, but there is logic behind it.

Why would you remember anything from a hash instead of the infinitely more usable name+ version (a more-than-enough-unique combo?

If you have an actively used Nix system with a large disk, it's very likely that you have a large number of (unused) deriations with the same prefix in there. A derivation is also a lower-level thing than a package, so it's not guaranteed that you have a useful name & version in there.

I just checked on my laptop and I have, for example, over 400 stdenv-linux build outputs in there.

That still doesn't explain how 400 hashes are more useful than name+version+hashes

(this'd still suffer from the common fail of flat folders, should've used proper nesting, but not as much as hash-first)

To determine how useful something is it's important to look at the usecase.

My usecase for interacting directly with the store is that I want to look at some specific store path, so I already know the hash from somewhere and can autocomplete on the first characters with much higher precision than first completing drv names, and then moving on to the right hash.

You seem to have some other kind of use-case. What is it?

the usecase is described right there in the first comment

And 2_of_name + 2_of_hash is giving you higher precision in one match as it has more info to match against the full path vs just 2_of_hash

Is the language statically-typed? If not, why not?

Nix the language is untyped; I don't believe Tvix changes this, particularly since it has the goal of supporting all of Nixpkgs.

That’s a pity. IMO the lack of static typing is one of the biggest flaws of the language, and it persistently slows me down.

pure flake nix with nixos modules is strongly statically typed.

in this case typing is evaluation. so tvix promises doing typing faster.

I guess it does depend on what you consider evaluation-time to be e.g. “Haskell is a dynamically-typed, interpreted language.”[1]

1: https://aphyr.com/posts/342-typing-the-technical-interview

There's no need for more fragmentation in this ecosystem. Already package managers are heavily fragmented in the Linux world. This is making a bad problem worse.

Why do you think that? I've been following Tvix for some time closely and I've never felt they were introducing any sort of fragmentation.

Tvix developer here; Tvix is quite different from Nix and is a clean restart, in my humble opinion, it would be quite hard to integrate some features of Tvix in the current Nix because you have two targets: (a) getting the feature in the existing architecture of Nix (b) evaluating a good architecture for the feature itself in the ideal architecture.

You know what, you are right. Going beyond just the fragmentation, there’s too much (Linux) cruft on top of (Unix) cruft on top of (POSIX) cruft. Complexity leading to diminishing returns in every effort to modernize the system. Unironically, let’s throw away all the historical cruft and build a new OS, with a Nix-like package manager, TheseusOS-like architecture and only 6 processes visible when I run htop after a clean install, all with clear descriptions: Services Manager (systemd), Display Manager, Desktop Environment, Connections (WiFi/Bluetooth) Manager, Audio Manager, Power Manager. A desktop OS that is intelligible by its user.

It's worth noting that tvix explicitly[1] does not support flakes.

[1] https://lobste.rs/s/ypwgwp/tvix_we_are_rewriting_nix#c_zvtze...

This seems to be from 2 years ago, a lot of things could have changed.

He seemed quite opinionated on flakes, but the only bad thing that I can see to flakes is that you cannot pass args to them, for imo no valid reason, you have to use the file+file protocol and update the inputs each time to have a reasonable way to pass parameters...

(Tvix developer here) 2 years after, Flakes are still what they are, IMHO, a pile of layering violations.

I do not see a path forward for them in Tvix before we get to fix them layer by layer, which Nix is trying to do (slowly?).

At some point, once we stabilize a bunch of things, I have some plans to do what I call the right design of Flakes, but it does not involve modifying completely the core of the interpreter to leak this implementation detail everywhere, but more make this a library concept.

Most flakes make it a pain to cross-compile, too, especially to platforms other than the "big four" ({aarch64,x86_64}-{linux,macos}), as compared to overlays.

(That said, the tooling around flakes is so much nicer that I still use and recommend them.)

override input works well.

asking nix to accept arbitrary args in flakes like asking to so random patch of rust code during compilation. to patch use cargo patch. that is analogy of flakes, when you thing how flakes make nix statically typed.

I've had CL11242 open in my browser for a few weeks. The first thing I'm doing when it merges is evaluating my flake(s) with flakes-compat.

Tvix explicitly targets stable Nix features, so supporting Flakes is a non-goal.

Many users have a poor understanding of what Flakes _actually_ are: They are a bit of UX glue on top of existing Nix features:

- Input/output schemas for `flake.nix`

- A lock file format

- CLI features to work with the two above

It's entirely feasible to build out a Flake user interface & evaluation support on top of Tvix without making it a first-class evaluator feature. See https://github.com/edolstra/flake-compat for prior art.

The key point of Tvix _not_ having support for Flakes is to not make special snowflake evaluator features that are tied in with it.

> Tvix explicitly targets stable Nix features, so supporting Flakes is a non-goal.

Except that it rolls its own CA store[1], which is also not a stable Nix feature. One could argue that it has to roll its own store because Nix wants to own the store, but implementing a shadow version of an experimental feature makes the "they're just targeting stable features" part ring rather hollow.

> The key point of Tvix _not_ having support for Flakes is to not make special snowflake evaluator features that are tied in with it.

That's not really how I read the authors' defense of this choice. It seems like they made it because they disagree with the design decision, and I consider that more defensible than the position you offer. I just wish they would make this information more central, and I will keep posting it in news items about Tvix because no one else is going to. There is no indication that the "Nix" they implement is many years old now, and when they do[2] indicate it, they are vague about why.

[1] https://cs.tvl.fyi/depot/-/blob/tvix/castore/docs/data-model...

[2] https://cs.tvl.fyi/depot/-/blob/tvix/README.md#compatibility

Flakes are a misfeature that adds a complicated layer of abstraction over an already not very simple system.

I follow some chats with lots of Nix beginners, and the amount of people that are now stuck in a flakes tarpit and have no understanding of the fundamentals of Nix (and no path to get there, really, in the course of normal usage) is depressing. Just the other day I saw someone post a 40 line Nix code snippet using flakes, pulling in 2 git repos apart from nixpkgs (because the barebones flakes are barely even usable without support libraries), all to make a simple nix-shell with a single package in there - and it didn't even work and they were unsure how to proceed with debugging.

In almost all chats with flakes proponents it also eventually turned out that what they want is something like niv, but integrated into the Nix binary, not all of the additional stuff that is attached to flakes.

Anyways, adisbladis' point is that Tvix does not need to support flakes in any way. If you really want to add these complications to your Nix code, you can implement everything flakes do in pure Nix (using e.g. the flakes-compat thing linked above).

Sure, and this is all fine, but it feels like this information gets buried or smoothed-over, and I think you strengthen your position by leading with it as a differentiator. Nobody new to Nix knows what targeting 2.3 means, but they likely know what flakes are, and you aren't doing a lot to make it clear why they should prefer a fast language evaluator to something that can handle flakes.

PS: TVL is international, but a lot of the development will take place in our office in Moscow. Say hi if you're around and interested!

That's going to be a no from me.

Imagine if we had said the same about Nginx. Let's evaluate things on technical merits, specially if they are using open source licenses we understand. In this case GPL v3.

Yes, imagine if we had said the same about Nginx. Now let's imagine now that we're saying those things in 2024, during the first major land war in Europe since WWII, in a context where state actors, if they are not already Jianing Tan, have enormous motivation to do so.

Now, let's put the risk at the root of the build system, and leave the lead devs in a country known to coerce, threaten, torture, poison and kill its dissidents.

That's a no from me, dawg. Look outside, it's not 2015 anymore.

I’m afraid these wars are going to kill open source.

Unfortunately… if that’s going to happen, I’d rather have it happen without first having huge security breaches.

I'm responding to their suggestion to travel to Moscow. FWIW I am very excited to see a rust impl of Nix focused on performance, and I am completely understanding that a person cannot pick their nationality (to some degree).

Why?

I'm a US citizen and there is a significant risk that I will be detained and held hostage if I travel to Moscow.

I don't think is wise even if you had written guarantees from their state via the business, because how your state might look at it.

It's a war, things have changed.

That post is from 2021, pre-big war (but post-2014 one, which nobody noticed). In 2021, stopping by and saying hi wouldn've have been all that preposterous.

Said it downthread, but saying it again because it bears repeating:

This is a product whose major contributors live in a state prone to very persuasive forms of coercion. Worse, this is a _build system_. Moreover, it's one of the most reticulate build systems on the planet -- I doubt more than a few hundred senior engineers would be able to spot a Jia-Tan-style compromise in the Nix machinery, let alone a remix/workalike of it.

There is currently a war going on. Not a 'war on' drugs, or cancer, or a 'culture war', or any of the other phenomena that Americans routinely nowadays call a 'war', but an actually-actual conflagration involving grads, kinzhals, and, easily, the futures of a couple of well-intentioned engineers in Moscow.

I expect the creators of this product are sincere, serious, smart, kind, generous, funny, honorable people. Yet it is simply too plausible that the FSB could come by and lean on them (or their uncle, or mother, or sister, or...) and have a little something extra put at the very root of the tree, from where it could be folded into any (or every) part of the system.

Open source across international borders is a peacetime phenomenon, as we are all about to discover, methinks. I hope I'm wrong.

I really don't want to start irrelevant discussions in this thread, but if you think an extrajudicial secret service can't lean on any open source developer regardless of the country and your side relative to it, please think again. "This can't happen here" is not a helpful attitude, and also you are on the slippery slope here. Paranoia can disintegrate the community far quicker than any actual spooks. Without a certain amount of trust (backed by reputation and verification) nothing useful could be possibly done.

there are plenty of people with commit bits (me included) who don't reside in Russia, and would (I hope) be pretty alert to the Russian state coercing tazjin (the only committer currently residing in Russia) into introducing some form of backdoor.

Cool, now do Israel.

Reply on news.ycombinator.com