NixOps – Declarative cloud provisioning and deployment with NixOS
nixos.org
nixos.org
Currently, this works best for Haskell, for which there is a sizeable amount (relatively speaking) of Nix tooling available. This is largely because a lot of Nix(OS) users also use Haskell, perhaps because lend themselves to very similar modes of "declarative" thinking.
This is a thorough, excellent guide on how to develop Haskell projects with Nix, including managing non-Haskell dependencies (zlib, SDL, ncurses, and so on):
https://github.com/Gabriel439/haskell-nix/
I only started using NixOS about a month ago, and it's fantastic. The best part is that binary caches are available for almost everything, so the hitherto-painful step where one compiles Haskell dependencies locally has been completely eliminated. The best way I can describe how I work now is "Python virtualenvs, but for everything at once and just so much better".
PS. Gabriel Gonzalez is, in general, someone who makes great documenation. Check out the tutorial for his stream-processing library Pipes: the ASCII art alone is awesome :)
https://hackage.haskell.org/package/pipes-4.3.4/docs/Pipes-T...
We build everything with nix from sbt projects, go projects, haskell projects, purescript projects, docker images, and npm projects.
It's an incredible tool.
NixOps is also a good tool but there are a lot of issues with it we haven't had time to contribute fixes for. We also wanted it to be lighter weight and to decouple system specification from system deployment. So I wrote a tool that handles performing the same system deployment operations that NixOps does for you and you can simply pipe the nix path result from nix-build to it. We use terraform to specify and manage the cloud resources.
I work for Awake Security so keep an eye out on the company's GitHub org as we've been publicly releasing a lot of projects even though they aren't completely polished or announced yet. I'll see if I can do this for our deploy tool soon.
Basically the kind of person I would love to buy a beer any time because he does so much great work.
on topic: The binary caches are great. For work I'm currently evaluating https://github.com/typeable/stackage2nix and https://github.com/input-output-hk/stack2nix to generate nix expressions from our haskell stack projects. And then push the build artifacts to a shared cache. Such that nobody has to ever compile a dependency twice. Because nothing sucks more than starting your day watching the compiler churn away at compiling after version bumps, to then also see a collegue go to the same pain all over again.
However, i've found NixPkgs to be a bit messy. It's too big, targets too many platforms and it's too disorganised. But to be completely fair, I think it's getting a lot of good refactorings lately.
For large enterprise use, Nix is going to need to mature to a point where it's possible for enterprises to host a Nixpkgs cache locally, using a binary repository manager like Artifactory or Nexus, both for sheer cost (to save on bandwidth of thousands of servers downloading updates from the Internet) as well as risk reduction (in case the public binary caches are unavailable at critical moments). If this is doable today, it's not really documented.
I'm sure the project will get there... eventually.
You can even go further and set up your own build server to service all the architectures you need to support.
GuixSD is getting production-ready IMHO. See for example Guix getting adopted by some HPCs:
https://github.com/nh2/nix-binary-cache-proxy
Allows you to set up (using NixOps!) a caching proxy for the nixos binary cache in your local network.
It can also serve your own custom packages if you sign it with your own package signing key and tell your machines to trust it.
Is using Artifactory or Nexus a requirement? Why?
The binary artifacts are behind an HTTP server, which you can access control through whatever normal access control mechanisms, or leave uncontrolled and rely on the fact that it's not possible to retrieve artifacts from the server without having access to all the build inputs so you can hash them.
Reading briefly through the examples, it seems very oriented towards provisioning machine instances rather than provisioning behavior across networks or multi-machine services, which is something that Puppet really excels at. What features does this offer in addition to ones that Puppet provides?
That said, I've been getting tired of Puppet recently for some unrelated reasons, so I'll definitely give this a try.
NixOS treats immutable configuration as a first-class citizen at the OS level. Dependenices are not installed into /usr or /etc, rather specific versions of packages (identified by their hashes) are installed into /nix/store and then symlinked back to /usr or /etc. Different applications that require different versions of the same dependency, when packaged according to standard, harmoniously co-exist on the same server because the Nix packaging system symlinks the dependencies between each other in /nix/store.
You don't realize the full power of this setup until you make a mistake, make your system unbootable, and calmly tell GRUB to just use the last valid configuration - a few minutes later and you're back at a desktop. Conventional configuration management tool like Puppet tend to be worthless in those kinds of failure modes, and most sysadmins would be implementing backup restoration procedures.
I write this into my configuration.nix:
boot.kernelPackages = pkgs.linuxPackages_latest;
There are other small things I can do with this. What if I didn't want to boot this kernel to boot, just test it?
$ nixos-rebuild build-vm
$ ./result/bin/run-*-vm
---
Another: What if I wanted Tarsnap backups of something? (This may be out of date, but it's my old config, and I wrote the Tarsnap module)
services.tarsnap.enable = true;
services.tarsnap.archives =
{ nixos =
{ checkpointBytes = "5GB";
directories =
[ "/root" "/etc/nixos"
];
};
home =
{ excludes = [ "*.o" "*.hi" ];
checkpointBytes = "5GB";
directories =
[ "/home/a"
];
};
};
$ systemctl start tarsnap@home
$ systemctl start tarsnap@nixos
Lots of other small things. Binary caches, your own Hydra, remote/multi-architecture builds, etc. One time I "configured" my NixOS laptop install before getting my laptop: on a remote testing server, over remote KVM. Pushed it to git. I cloned my configuration.nix during NixOS installation on my laptop a few days later, and I replicated my laptop in 10 minutes. :) Combine this with Hydra (CI) server: You can even write tests and have your laptop configuration continuously integrated all the time?!?!?!?
That said NixOS isn't always the easiest thing to use, but it does come with some big advantages!
[1] I'm aware of DKMS (and have even worked on modules using it where it was fine) but I've still had it break once or twice. Maybe that was VMWare's fault :)
[2] If you share code between all your configuration.nix files, this becomes a lot easier.
Nix/NixOS is the big deal actually. NixOps is just icing on the cake. A NixOS installation is configured by building/copying over a system profile, and then activating it.
With just NixOS, you edit a configuration file on the server, then run a command that will build a system profile from that configuration and activate it.
With NixOps, you have configuration files on your workstation, and run commands to build the system profiles, copy it to the servers, and activate them. Of course you can do all of this without NixOps. It just makes certain things easier.
Puppet and others can do all of this, too. However state transition tend to be the weakest link (e.g. tell Puppet to provision a brand new server vs one that has run some other stuff, how sure are you that the resulting systems are the same?). Nix/NixOS takes care of that (most of the time anyway -- there are still things like data migrations that applications have to take care of).
With the nix system you can fully declaratively state your entire fleet of machines, from your highest level application configs all the way down to kernel patches.
When you run nixops, your machines become exactly as specified.
That this works reliably is a novelty: most other configuration management tools work like "action runners" that try in a best-effort fashion to achieve the desired state, but that doesn't always work (for example in Ansible, declaring a package as installed installs it, but NOT declaring a package will not uninstall it).
It is made possible because the nix system is designed from the bottom up to be declarative. That makes it different from Puppet.
Most other configuration management tools have no choice. I got my own [1] to the point where it cannot only provision files/users/groups, but also clean them up correctly most of the time. However, since it is not the final authority (the package manager is... or rather, root is), there will always be situations where the entity that needs to be provisioned will be found to be in an entirely unanticipated state.
You have basically two ways to solve this conundrum:
1. just throw everything away and start from a clean slate when the configuration changes (aka containers)
2. do configuration management in the package manager and encourage/force root to keep his filthy hands off the system partition (aka NixOS et al)
NixOps is the cloud configuration layer, it provisions cloud resources and uploads NixOS configuration to the machines. It's closer to terraform in that regard. It also has a state file (currently sqlite3) and uses a push model to propagate configuration changes.
What's missing right now is a third option where an agent is installed on the machines for a more dynamic configuration of the cluster as a whole. This role will probably be filled by Kubernetes or Nomad.
Here's a deployment of mesos where you can specify how many slaves and masters you want: https://github.com/wmertens/nixops-mesos
An object (called attrset in nix):
{ a = 3; b = 4; }
[ 1 2 3 4 ];
"hello"
''
multine
string
''
a: b: a + b;
{ a, b }: a + b;
One special type is the path, which has it's own literal that starts with `./`
To load a file, the `import` keyword is being used. In most cases the whole file is a big lambda function, which gets returned by the import.
Once you know that you can start reading nixpkgs and learn about the building blocks to constructs packages.
{
a = {
b = {
c = 3;
};
};
}
{
a.b.c = 3;
}
services.openssh.enable = true;
Also includes links to the only other resources I found.
With NixOS, my system is always 100% clean, and never broken.
I can install packages in my home directory without root.
Best of all, whenever I want to write any code, I can use nix-shell to make the toolchain and dependencies temporarily available to a self-contained shell. No more cluttering my system with libfoobar-dev for a quick build, and forgetting why.
However, from my ignorant viewpoint:
The deep agreement is that machine-level statefulness is evil. Trying to repair it in-situ is pointless when you can simply reset to known-good states.
BOSH does this at the whole-machine level. You get a stemcell (fixed target), you get releases (fixed configuration of software) and you get deployments (fixed mappings from releases into running systems).
When you make a change, you redeploy. BOSH recompiles any changed release packages and then rolls them out across the fleet, starting with canaries.
BOSH lets you define the number of VMs, networks and whatnot; I'm not sure if NixOps does.
Treating components as replaceable is the main point of departure from previous generations of configuration management, which largely evolved out of a world in which the mission was "make this single machine look right, starting from an unknown state".
NixOps appears to do something similar. I'm unclear as to whether it operates at the whole-machine level (à la BOSH) or whether it can perform transformations within an existing machine. The latter would, obviously, be a lot faster than replacing VMs wholesale.
That said, whole-machine replacement does have one accidental benefit: it's ruthless. You very quickly encounter the one-in-a-million edge cases for running services because BOSH will quite unmercifully steamroll VMs during a deploy. Any stateful distributed services that aren't ready to have running machines yanked from under them at short notice will quickly reveal themselves.
You'd be surprised how many systems that advertise themselves as tolerant to random machine loss ... er ... aren't. Not sure much how I can go into this, but some distributed systems do much more poorly than the stuff on the tin suggests, even after substantial engineering on our part to try and prop them up.
Disclosure: I work for Pivotal, so I use BOSH sometimes in my dayjob. But I'm by no means a capable operator or BOSH expert.
* NixOps works on whole machine + transformation of machine level: NixOps builds all system configurations of the whole network, then makes sure the required target hosts are available, then copies the builds to the hosts, and then atomically switches all hosts to the new configuration at once. NixOS is stateless, and a new config only means that the new service configurations are applied.
Rollback is a matter of switching all VMs back to their previous configurations, takes less than a minute in all.
So what I do is have all system level tools installed using NixOS: the OS, logging, systemd, Docker, sshd and so on. Then I have my applications built as Docker containers and provision them as systemd units also configured by NixOS files.
There are probably lots of systems running standard Linux packages with more complex configurations (i.e. Mailservers, maybe databases, proxies etc) where I could imagine it being useful? Or maybe their desktop work makes more sense than I thought–my desktop environment is much more customised than any server, and I've never tried if the setup scripts I try to maintain actually work.
I feel NixOS definitely gets some stuff wrong, and I hope at some point it gets some good competition. Mainly because the Nix language looks a bit awkward to me, and also because I don't fully get their package management situation. But the idea of having the operating system be configured from a specification is just a level up from old fashioned linux distros.
It's great to see NixOS and similar tools gaining ground. But personally I find that docker's configurations simpler to grok than NixOS's syntax. Plus it's interesting the granular level of control over system security and system access gaiened via docker (really mostly Linux secomp controls and process namespaces). It's kind of like having capabilities built into the OS on top of specification driven system configuration.
NixOS is capable of creating a Docker image given a Nix specification, by turning the Nix store into an archive. These images are quite minimal since they contain exactly the dependencies needed for your app.
NixOS also has its own container system, so your server configuration can simply specify a subconfiguration that will be spawned as a container. The containers reuse the Nix store from the host system, which means you get a fine-grained caching system.
I haven't used those things much so I don't really know how production-ready they are.
I sometimes use Docker (with Dockerfiles or DockerHub images) on NixOS as a way to run systems that I don't want to package with Nix. I've made a little Nix module for declaring Docker services as systemd services.
That's part of why I don't feel like Nix is a limitation. I have actually used NixOps to deploy NixOS configurations that include Docker services, so my feeling of Nix is that it kind of supervenes on everything else...
Still, Nix should be at least an inspiration for next real world package managers.
Most of the pain is building stemcells (the target disk image). Microsoft haven't quite made their peace with the idea.
If possible I'd prefer to use one tool for both OSes.
Other distros are also affected by these issues, so I think they'll be solved eventually.
https://github.com/NixOS/nixops/pull/634
I cherry-picked this into my local NixOps repo and have been using it for several weeks to deploy to Vultr servers with no issues, except one -- the author has hard-wired an assumption that you're using Btrfs for your root filesystem. I didn't want that so I simply replaced that hard-wired assumption with ext4 in my personal branch. Now everything is working great.
Certainly Nix makes it a lot easier to get reproducibility in practice, though.
[1] https://www.youtube.com/watch?v=QvH5qU1qBXY&feature=youtu.be