NixOS and Stateless Deployment
gfxmonk.net
gfxmonk.net
And NixOS is something so elegant and beautiful, so mathematically "right", that I subconsciously expect that it wouldn't actually work, and yet it does. I tell it what I want to get (relatively marginal configuration, btw) and it configures installation just right. I break something and it rolls back without a problem.
So the main question I've had ever since: why this stuff isn't popular enough yet.
It would have required a significant time effort from me to understand what is going on and get everything set up the way I want. I was just not willing to put that in at the time. I've gone back to Ubuntu for now.
I am doing more and more Haskell development so I'll take another run at it or, at least, set up nixpkgs on Ubuntu but I will probably give the project some time to mature.
Therefore, the culture seems to have all default package builds with all optional dependencies on. This leads to situations such as installing mutt and getting python too! (mutt -> gpgme -> glib -> python)
Last time I checked, if you installed git, you'd also get subversion, etc. Quite sad, given that nix is full of so many fantastic ideas. Hope it matures soon.
https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...
dev::hlinux { /home/sibi }-> nix-env -i --dry-run git-minimal-2.1.4
(dry run; not doing anything)
installing ‘git-minimal-2.1.4’
these paths will be fetched (11.69 MiB download, 69.16 MiB unpacked):
/nix/store/4qh8anh4phwx3gmrn2k49zh5xbkn49ir-git-2.1.4
/nix/store/4x0w8x70j27jxmy9wsn7sijrjjzj3dgj-pcre-8.36
/nix/store/g9qkr44yllgy5cb03vmfdksmh3pbmp1s-gnugrep-2.20
/nix/store/gwcvzrwyvgkgwlrgfyj4gqvsqn5syv1c-expat-2.1.0
/nix/store/rhghn8brqrz4j5ndkhwnm4afqjvk7zi8-python-2.7.9* If I removed something from the Ansible configuration, it stayed on the server unless I explicitly removed it manually. This created hidden dependencies. I solved this problem by creating a brand new server and running Ansible on it from scratch every so often. I have considered setting up CI for our Ansible configs by using Vagrant to recreate our server architecture, running Ansible on the virtual machines, and ensuring everything works.
* Our continuous integration setup requires Ansible to be installed on the CI server, so it can automatically deploy to staging using the same playbook (Ansible configuration) we use for deployment. Our staging server is the same as our CI server, and it was actually a pain to set up deploying locally as root. Also, I feel like allowing the CI software to use root is a security hole.
I also spent some time with NixOS a year ago, and I was very impressed with how it manages packages. The first problem I mentioned with Ansible seems like it wouldn't happen with NixOS, since not including a package in an environment means it won't be present. Second, it also seems that you could use Nix's declarative configuration language in restricted environments, which wouldn't necessarily require root, instead of having to install system-wide packages for a particular deployment. I am not sure how easy this is in practice.
Currently, I am using Arch Linux, and I installed the nix package manager to play with some more. In the future, we might be provisioning AWS servers in real-time to run simulations given to us by customers (we make simulation software), and in that case I am going to investigate NixOS more.
The solution in my experience is to change the state to absent instead of removing it from the playbook, and then refactor it out at some point in the future.
But NixOS is stateless from update to update as well. The key difference lies in the difference of scope between what is described in a playbook and what is described in a NixOS configuration file.
That's how I remove / add users or packages or configuration files. It also gives you a record of what you had on there before too, which has been nice in some cases.
Another thing that is very useful to do is to every 2-months or so, reprovision your systems again (which is something of an architecture refactor).
Really I wouldn't want untracked resources to be automatically removed - because someone might have had a reason for installing them.
Though, yes, the question of removal of cruft does come up.
However, if you are doing immutable systems, it's likely you are doing Ansible builds from something like Packer or aminator, and are doing red/green deployments that completely replace the OS.
I think immutable systems are likely the future for applications, they just take some extra work to get your brain around them, that may make them be viewed as "not worth it" in some capacities - those dealing with a lot of data, needing to "stay around", legacy applications, etc.
However, this is also why Docker is popular - because it's providing people a quicker way to get into the immutable system workflow.
People have tried to make a better RPM in the past - I worked at a startup that did this at one point (rPath, now defunct). Though, ultimately, I think people want to work in higher level primatives than package managers, and the use cases involved in migrating a system from version A to version B are often not entirely clearcut.
Configuration is also manageable by things like etcd - I think a new package manager is ... I guess conceptually interesting. But it's kind of the last thing I'd want.
To control what versions you install, it can often be easier to just maintain good software repos, rather than enforcing it on each individual system.
(Think databases).
disclaimer: wrote Ansible
However, nix still still really useful even when you have immutable servers. Once the components of an app are described as nix packages you have a lot of flexibility in working with them:
• deploy to a local sandbox for development
• deploy to virtualbox instances for end-to-end testing
• deploy to cloud instances for testing or production
I very frequently deploy to a virtualbox VM during development. With NixOS this often takes <10 seconds, and that still feels slow. I cannot imagine that you can do immutable deployment anywhere near as quickly or conveniently (but I'd be excited if you can tell me I'm wrong).
[1]: https://nixos.org/wiki/Hosting_providers_%26_NixOs
[2]: http://digitalocean.uservoice.com/forums/136585-digitalocean...
You can also get NixOS running on EC2, if you happen to have an account: https://nixos.org/wiki/NixOS_on_Amazon_EC2
* source: followed said instructions to get my linode vps running on nixos.
Just like every other package manager, from rpm to dpkg, Nix is responsible only for the aspects of the filesystem that it's specified to be responsible for. It's not going to remove garbage left behind by users or poorly-written post-install scripts, and it's not going to automatically undo other non-filesystem-related state changes on package removals. And the practical reality is that not everything on a system can be a native package anyway.
Any system is welcome to add it's own entropy, put it's own junk on - but when you want reproducibility, you do it by specifying how to get that reproducibility in your nix expressions.
Moreover, there's nothing stopping a Nix package author from writing a post-install script that causes state changes to be made out of bounds. The package manager isn't going to notice, and it's not going to clean it up at deinstallation time.
Every package in the store has exact dependencies - they reference the hashes of other packages only. When a package is built with Nix, only the directories for these packages are exposed to the chrooted environment in which the build occurs - so it is simply not possible for some randomly added junk in the filesystem to make it's way into a nix-defined package.
Contrast this to building software on another machine, where I might depend on "glibc" version "1.0". The combination these two values hardly represents a unique identity, as I could make any random package that fits those requirements. It's much more difficult for me to create a package which results in an hash collision though.
One thing that makes the other systems so unreliable is the presence of multiple repositories. If you were going to deploy packages from a single repository, then you can do careful planning in such a way that packages do not have any collisions. Assuming no user mutates the directories under control of the package manager, such system will also be effectively reproducible. Current mainstream distros work surprisingly well because they basically use this model, where a default repository provides most users needs. These distros quickly break down when you start adding third-party repositories which bundle alternative compilations of the same software that sits in the "official" one.
Basing packages from hashes (identity), rather than names and numbers, and making sure all of the files for each package is held in an isolated directory ensures that collisions won't happen, even if two different repositories provide the same software name and number, they are represented by different hashes, and will be treated as distinct pieces of software.
I'll continue to follow and learn more about Nix, but anxiously awaiting my epiphany.
[1] https://github.com/NixOS/nixpkgs/blob/1a504e3fb72fab10799cf0...
The thing is, "better" does imply "different." The syntax is weird and difficult, but it's the linchpin of the whole system. It's much, much better than your typical packaging system and worth the effort to learn it.
I take it configuration of the packages needs to go in nix as well? So what do you do when there is, for example, a change of upstream smtp?
Nix gives you tools for dealing with the concept of an immutable system - there's very few other working tools that do the same. Perhaps it can't possibly get all of it right - database setup and schemas, for example - but that's alright, you can use other tools to deal with them! (There's certainly a glut of schema management tools that would fit right into a Nix system.)
However, NixOS doesn't handle multi-system upgrades very well on its own - for example, you need to have some external way of dealing with "we need to upgrade the schema before restarting the application servers one at a time". I think that's a whole project on its own, and something that almost everything seems to do on an ad-hoc basis (Salt excluded).
Because the service definition has my migration tool as a dependency, it gets rebuilt when the tool is updated. That causes Postgres to get restarted and the tool to get run IFF there are new migrations. The nix configuration is parameterized, so that in dev or staging environments, the database is completely rebuilt and loaded with test data on each deploy, but in production, we just apply migrations.
With Nixops, this works even on a multi-system deployment. The full deployment typically takes several minutes, but most of that time is in installing the updated packages, which doesn't interrupt the running system. Once everything is installed, the cut-over only takes a second or two. Eventually, I'll have enough traffic to warrant more careful orchestration of the cut-over, but for now it's fine.
Don't get me wrong, Nix sounds great, but this is a poor article from an inexperienced sysadmin who is unable to really point out the pros and cons.
1: Actually, last time I looked at NixOS, you could also install things the imperative way, but that's a silly thing to do.
Actually not, because if you do, you will find these changes added to a file declaratively describing all the changes you have committed to the system (or user-context) as a whole.
It's a good way to build a config if you're not fully comfortable with nix-syntax.
I don't remember where the file ends up, but it is there.
My hunch is that it could scale well, but would require some integration work in order to use it nicely with existing orchestration tools. I don't know exactly what that would look like, or if it would be worth the effort of going off the beaten track.
This article is fluff because of this lack of experience; it honestly reads as another "looks at this cool thing I know nothing about!"
And you're done. Server up and running. done about 20 of them in the last twelve months. What's all this "package" stuff about? what's wrong with yum?
The thing with package managers is that it get's boring to do everything all over again for a large amount of servers. Also your brain memory does not have a reliable version control. You need to keep why and how you do certain stuff on your server somewhere else. First thing that comes to your mind is a shell script and people are already using something better ( ansible, puppet, etc. ). Now, with this article, I see that there is something even better than this.
By using Puppet, I never have to read my colleagues custom setup script because store it there. By checking in everything to version control, I see who has done what by reading the changelog.
If you are only managing your own servers and keep a strict discipline you will see less of the benefits.
All these tools have to deal with the problem of state. Most of them choose to handle it implicitly...the state of the system is the state and the tool tries to determine it at runtime. This strategy has all the problems listed in the article. Other tools attempt to keep a record of the state. This solves the issues from the article but has it's own set of issues (mostly state management and state corruption). Neither approach is objectively right and both have their tradeoffs.
I honestly agree with the original, down-voted, poster...the article is a bit preachy without much insight. The rise of DevOps has turned system administration into a CS problem. And, like most other CS problems these days, it all boils down to managing state. And just like there are many approaches to state management in application architecture (RDBMS, NoSQL, Paxos/Raft, etc) where no solution is objectively right, there will be many approaches to state management in DevOps where no solution is objectively right. The author obviously has his preferences, but his analysis is biased and incomplete and should be called out as such.
It sounds like it's mostly a provisioning tool, and doesn't really help with configuration management once your machines exist. From the "terraform vs chef / puppet" page:
> Terraform enables any configuration management tool to be used to setup a resource once it has been created. Terraform focuses on the higher-level abstraction of the datacenter and associated services [...]
So it sounds like like a very nice provisioning tool, but doesn't really compete with NixOS itself. Perhaps you could even use it to provision NixOs machines?
I'm really not trying to sound like a shill for Hashicorp, but we use a bunch of their tools and find them to be, overall, very worthwhile and focused on accomplishing a single logical task which makes them easily composable with tools from other vendors. I also don't want it to sound like I'm criticizing Nix or NixOS...they sound like excellent tools. My only point was that there are other ways to solve the problems expressed in the posting and that each solution has tradeoffs that DevOps needs to consider when designing infrastructure. Your blog struck me as being a strawman criticism of somewhat dated tools without consideration for newer options, especially since your discussion of Docker was so narrowly focused on the actual Docker tool without any consideration given for Fleet, Swarm, ECS or any of the host of orchestration options in the Docker ecosystem.
If you'd written it more from a position of "here's how NixOS has made my life easier," you'd probably find that people would be more receptive to it. But, instead, it had a "here's why NixOS is better than the alternatives" feel to it which is going to rub people the wrong way when it's pretty clear that you're not aware of all the alternatives. NixOS is one good option, but it's by no means the only good option.
> Perhaps you could even use it [Terraform] to provision NixOs machines?
You definitely can, so long as your servers are virtualized. Terraform is significantly less useful in a bare-metal world. However Terraform is really about provisioning specific machines. For example, you might write Terraform to provision 1 SMTP server, 3 web front-ends behind a load balancer and 2 database hosts. But it's pretty crude at doing provision-time tasks...it basically allows you to run shell commands. Where you'd do the bulk of your provisioning would be in a tool like Packer, Aminator or other such tool that creates VM images that can be deployed. That's where you'd start with a base NixOs image and then declare what's installed on an SMTP server, a web front-end and a database server. Terraform would just reference those images and size the machines.
I definitely have humble requirements in terms of deployment size, so (for me) any orchestration tool is likely to be way more effort than it's worth. I compared NixOS (not a deployment tool) to those other (small-scale deployment and/or configuration management tools) because that's what I and plenty of developers I know have used, and I think the comparison helps illustrate the issues that NixOS can solve. Hopefully those who _do_ have experience and need for larger orchestration software can tell from reading whether the problems NixOS solves are relevant to them.