Erase your darlings: immutable infrastructure for mutable systems
grahamc.com
grahamc.com
edit: If I get three more upvotes I'll do tomorrow.
edit: two more.
edit: one more.
edit: all filled up, it's going to happen!
edit: Some more context. Although I have a laptop (a 13" 2015 mbp) I don't bring it to work anymore, but have two desktops, one at home and one for work. All three run the same nix configuration, shared through a git repo. Whenever I discover there is some program I need I add it to the config and after switching machines run `sudo nixos-rebuild switch` - so the thing I added becomes available on the current machine there, too. All this just works (tm) and I'm fairly confident the re-install will be painless.
It seems really complex though, so I'm not entirely convinced it's a good idea.
Secondly, for an exactly deterministic install you'd also pin the nixpkgs to some commit (the package repository is a git repo). But I guess many don't do this unless they have specific needs.
All the big chunks of data are already isolated into redundant partitions, but it’s the system config that’s tough.
I have a time machine backup, but that’s still not the same as being able to say “I’m gonna wipe my hard drive and start over today”.
So does anyone have have good suggestions on maintaining a MacOS laptop in an immutable way?
Any immutable infrastructure lights up license keys and demo restrictions like a spotlight.
Take, for example, Apple’s desktop OS: You used to be able to drag-drop an application to install it. To uninstall you would delete it. Simple. Easy. Stateless. They talked about it a lot and so did Mac evangelizers. But it also meant you could walk in to an Apple store, connect a USB drive in your sleeve to a demo computer, drag, drop, and walk away with full versions of very expensive software. So, the OS fell prey to the same stateful pitfalls as Windows: places to hide keys, system hooks, etc, etc, etc.
Your best bet these days is likely to manage all software config through a system management tool, keep your data backed up in Time Machine, and “reprovision” your laptop every X days or months.
Then you can store this info in separate key management software (e.g. Unix pass for your local machine) at which point bootstrapping your system could be done relatively automatically.
EDIT: I should add that this is my view of how things should work in the-world-as-it-ought-to-be, which isn’t necessarily the same as the-world-that-we-live-in.
I guess to me that’s what the /persist drive was in the example, the need to customize system data beyond the basic install steps. Think of your Application Support folder, or apps that don’t sync to the cloud, or system kexts you might need to run apps.
The biggest problem isn’t software like Apple’s which use the Internet to authenticate how many systems are in use with generous limits, it’s Software like iLok and such that use various stateful properties of your system in an undocumented way so it’s hard to preserve them across installs. If that software uses timestamps, for example, it might be hard to preserve what it needs. Keychain might be another example, I haven’t fully investigated how that interacts with other chips like the T2 in these scenarios.
The problem is that outside of servers, it can be hard to distinguish between files I care about and files I don’t. What I’m hoping we get in the future is a filesystem and (on Windows) a registry, that automatically containerizes all saved state to the app and user it belongs to, ideally with some kind of historical log metadata. I know it’s asking for too much and would never be perfect, but it’s a nice thought. Right now we rely too much on apps to be well-behaved, but it’s the apps that aren’t well-behaved or that share data between themselves and other apps that are the issue. Apps asking for access is one approach, but I’m less concerned for privacy in this scenario than I am for state cleanup. The trouble isn’t reading anywhere though, it’s when an app shared state by updating data in another “container”, then you’d have to create containers to represent files shared between apps, or the container idea disappears and you’re left with metadata about file usage and global state as an alternative... At that point you’re limited by how much work you want to do to keep the system clean, and it might be easier to identify anomalies periodically than to keep a whitelist updated with files to persist...
Got any good examples? This is usually where I get stuck and can't find a good solution that works well with MacOS.
- https://www.usenix.org/conference/lisa13/managing-macs-googl...
Unfortunately that was 2013, and https://github.com/google/macops only has a couple commits, so those devops processes are likely very outdated.
The mailing list is dead, but as of mid-2017 this statement was made:
> The main google/macops project is still alive, but there is more development currently going on in the other projects we link to (like santa).
> We still use puppet, as do our Linux and Windows fleet, but none of us actually use puppet infrastructure. We are all working with a standalone (masterless) model.
santa seems to be the only actively-developed macOS management tool on the google github profile, but other tools listed in the talk are actively developed (puppet of course, munki, etc)
This is what makes Chromebooks so easy to maintain though: web apps and sandboxed Android apps can all easily sync to the cloud and compartmentalize their data.
All my Firefox plugins and their config (I guess this is technically data)
All the settings I've changed though the control panel (I can't figure out every file that changes when I make a change).
Any time I've changed a setting with nvram or defaults
Most of the settings of the default apple apps.
My wireless configs
For most of these, there is probably a shell script one could write that can backup and restore them, but I've never found a holistic solution.
Security permissions/demo versions are the proper solution to this, aren't they?
Demo versions (as in compiled to be a limited demo) are viable, but was counter to Apple’s image at that time (it doesn’t really show off the experience if you’re getting restricted or nagged).
Apple stores still need to allow people to connect their iOS devices and maybe USB sticks for photos, but they could have created a “read-only” policy and applied it to store machines.
Of course, these days they just distribute through the app store, and those apps put tentacles all over the place. No USB copies would function.
- Pretty much all my software comes from brew or brew cask
- All my files are stored in either DropBox or Google Drive (split is for legacy reasons mostly)
- All my work is in a remote Git repository
- My local config is also in Git
Anything else isn't backed up. I treat ~/Documents as scratch.
It's not ideal because it still takes some time to configure and log on to a bunch of cloud services, but maybe it says something about moving into management that I spent 90% of my day inside of Firefox, Mail, and Slack...
I saw that it now supports Whalebrew[1], which seems interesting. One of the issues that I have with Brew is that often my need is temporary and that I want to be able to clean up afterwards (similar to `git stash`).
Edit: Huh, only 90 packages. I was expecting more!
There is also Ansible, Chef, Puppet, ... but in my experience the overhead to write and test those config files is even hired than using Nix.
John Wiegley’s system configuration [0] is an example of this, although it is extremely ambitious and definitely not a good place to start for anyone who has used Nix before.
One of the interesting things that Wiegley does for further reproducibility is to have Nix manage and install DMG-based Applications [1] in addition to CLI apps, services, and other system-level things.
[0] https://github.com/jwiegley/nix-config
[1] https://github.com/jwiegley/nix-config/blob/bbad310daa1106f6...
If you have iCloud backups, do a final sync, then restore to the device you just bought. This has been the process for the last several iPhones I've bought for my family. At the Apple Store, they walk you through this process so everything is set up before you leave.
It's hard to tell the difference between your old and new device, in terms of software and data, for the most part.
It’s much better now than it used to be, but there are still time where I have to sacrifice data by uninstalling/reinstalling or resetting settings.
Top of my wishlist for iOS is Apple letting us manage (and potentially fix) our backup data.
I have an air from 2013; I moved (regrettably, given the keyboard) to a new air around this time last year and used the process as an excuse to force myself to specify everything well enough that I could recover everything I need from stock macOS from a short bootstrap script.
I use yadm (~git) for my dotfiles, Nix for everything Nix can do readily, brew bundle for a few mac apps around the edges, and 3 small restic backups for some project files and such. My dotfiles contain a longer bootstrap script that takes over the heavy lifting once yadm is installed. This script configures most of the settings I need on the way, and prompts me to do the few bits I haven't found a good way to automate. It also collapses ~/Downloads, ~/Desktop, and ~/Documents into a single directory, so that I don't have out-of-sight/mind places for state to hide.
It has been a lot of work, but it has bought me peace-of-mind that I can replace my system quickly (i.e., be ready to work on a new device in ~2h with maybe 20 minutes of actual babysitting?)
That said, I think the basics here are actually fairly low-hanging fruit. You can also make some really quick progress once you know you've got everything essential safeguarded if you're willing to take a dive like Graham has here and let minor losses roll off your back. Access to a second mac also helps a lot; iterating on problems in the actual bootstrap process is slow, and best done while you have a working system you can stay productive on.
I'm currently most of the way through building out the rough inverse of my bootstrap script--a script that audits as much of my "state" as it can; I hope to have it working for a living as part of my backup runs soon. It cleans up and auto-commits anything it can, tells me what is in a clean state, reassures me not to worry about everything I've worried about during past moves, and gives me an explicit checklist for anything that isn't buttoned up.
I think the main things left are forcing me to deal with downloads/desktop/documents as I go (with measures like Graham's being on the table...), and figuring out if there's a sane way to audit for drift in my macOS system/app settings.
All my Firefox plugins and their config (I guess this is technically data)
All the settings I've changed though the control panel (I can't figure out every file that changes when I make a change).
Any time I've changed a setting with nvram or defaults
Most of the settings of the default apple apps.
My wireless configs
This is always the part I get stuck on.
1. Safari is my daily driver, so I'd have to trawl other people's dotfiles/bootstrap scripts for evidence anyone's handling FF in a smart way. But, this might still help: Apple recently changed Safari extensions to be an adjunct to native apps. Before they made this change, I had a section of my bootstrap that used the `open` command to launch the Safari extension page for each extension I used, which made it pretty trivial to click install on each tab.
Another thing you might find tractable is looking for FF config/state files that you can back up and restore (narrowly). This is easier to play with if you have a 2nd system as I mentioned. I figured out, for example, how to restore my macOS Terminal.app windows from my previous install with all of my scrollback history.
2. These are rough, but you're already on the right track. It sucks, so maybe bitch at Apple about it--more voices might move the needle here. I have 3 basic approaches: find a defaults (and, rarely, another command) that does the same thing; figure out which plist file is implicated and diff it before/after changes until I figure out how to force the right change (broadly, I find it better to try preference edits with all involved apps closed); AppleScript it. I have applescripts in my bootstrap for stuff like setting my Finder preferences and enabling specific Safari extensions (now that they're bundled with .apps). If you haven't already tried it, UIElementInspector makes it much more plausible to actually write an AppleScript that has to do UI manipulation.
3. No direct comment on nvram as I've never fiddled with it, though I noticed it does have an option for outputting the current settings and a flag for setting them from a file. For defaults, I'm not sure there's anything better than just not doing it unless you add it to your script. (This is where I was wondering about whether there's a good way to diff my delta against my setup script...)
4. Too vague for me to give a good answer. If you can't find a way to set these from the CLI or hacking around with plists, I'm not sure anything shy of AppleScript will help.
5. What do you mean by wireless configs? Passwords and network names? I pull mine in through iCloud keychain and don't otherwise sweat these.
i was planning to try it out (currently i use homebrew), but with Catalina it seems things are somewhat complicated for nixos+macos.
my understanding is that NixOS wants to live in "/nix", and Catalina does not like that.
there are some solutions to this problem but they seem somewhat incomplete (you have to make an unencrypted partition for Nix etc..), though i haven't tried it so i might be wrong :)
https://github.com/NixOS/nix/issues/2925 https://github.com/NixOS/nix/pull/3212
I can't see running NixOS in production.
We're running 100% Kubernetes, including for databases and other stateful workloads. Kubernetes implements the author's pattern just fine - any OS state is defined within the container image, and any application state is defined within a Persistent Volume. Unfortunately, NixOS doesn't have a good story yet for service management (Disnix isn't nearly as featureful as the Kubernetes scheduler and doesn't see nearly the same activity / community buy-in as Nix / NixOS) let alone ensuring that networked storage is re-attached to the particular node that runs the service in the same reliable manner that Kubernetes offers.
IMO the way forward for Nix / NixOS in production is to:
a) develop a container runtime that would allow a Kubernetes node to run pods that specify Nix expressions directly in the image field, instead of the current workaround of creating Docker containers from Nix expressions and dealing with the overhead of external registries
b) improve the experience of running Kubernetes on NixOS such that ease of installation more closely approaches that offered by managed Kubernetes providers.
https://github.com/google/nixery
I'm the author of this, and here is a talk about it: https://www.youtube.com/watch?v=pOI9H4oeXqA
Why is there a need for an image registry? Part of the beauty of Nix is that Nix benefits from remote binary caches, but they are not required. Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache if possible and builds from source if not (with the caveat that production nodes should basically never be building from source)?
(Also Nixery is GCE-only and we're on AWS but leave that aside).
Mhm, there's no state that can't be thrown away and recreated, so I'd argue the overhead of running it is much lower than a full-blown registry.
> Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache
It depends on where you want to do this - Kubernetes for example has lots of opinions about images and how they're downloaded, so just replacing the runtime wouldn't be enough.
Nixery is an incremental step towards the end-goal, but there's a lot of mindset shifting that needs to happen first I think.
> Also Nixery is GCE-only
Nope, you can use a disk as the storage backend and then there's no dependency on GCS. S3 support would also be relatively easy to add by just implementing this interface: https://github.com/google/nixery/blob/master/storage/storage...
Additionally, once you use a custom runtime, now you have to deal with multiple runtimes in your cluster. You can no longer easily just run pods, you have to ensure they run on the nodes with the runtime for the images you want.
See https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/s...
Specifically "services.kubernetes.apiserver.enable = true"
My strong recommendation is to use nix to generate super optimised and amazing docker images then use them with k8 on a PaaS.
https://grahamc.com/blog/nixos-on-zfs
If using k8 (self host or PaaS), keep an eye on https://github.com/xtruder/kubenix as it'll blow your mind. Noyaml, infrastructure testing framework, deployment etc using nix.
Additionally, the channel ecosystem as it exists today does not allow you to choose your minor version of Kubernetes, which is another issue if you want to keep your underlying system up to date but also want to make sure that you're controlling when you adopt a new minor version so that you can deal with deprecations as necessary.
"Services.<service>.enable" is very similar to freebsd and /usr/local/etc except with standardised language to configure every daemon.
As for channel ecosystem and having control. Pin and override using layers — https://github.com/digital-asset/daml/blob/master/nix/nixpkg...
Then use https://cachix.org/ for stupid fast builds via caching.
Do you really want to give kubernetes the added responsibility of building your images?
Vagrant launches a bare-bones VM. Local files mount on /vagrant Chef-zero uses local mount to provision the systems and configurations necessary.
On every vagrant reload, this process repeats. Chefs idempotent nature means that any manual drift is automatically repaired.
In this setup there's no difference between `vagrant reload` and `vagrant destroy && vagrant up`.
It's possible to package this so that it's simultaneously "Infrastructure as Code" while also satisfying "Immutable Infrastructure". Our stab at this is now 6 years old and we're surely not the first to do it.
The beef I have with this article is the idea of:
> New computer smell
> Getting a new computer is this moment of cleanliness. The keycaps don’t have oils on them, the screen is perfect, and the hard drive is fresh and unspoiled — for about an hour or so.
In my observation (and in datasets that I have access to), computers systems tend to follow the "infant-mortality" curve. This means that if they run for a little bit, they're likely to run for a long time (and in addition, if you have many of them, they tend to die around the same time). My conjecture is that many computer systems have initialization routines which are not as thoroughly tested as the normal operating state of the system. Due to this, we tend to run into more issues in "immutable" systems than you otherwise would in "mutable" systems.
You couldn't just update anything easily. Well, anything is possible, but CoreOS made it very hard to do it the wrong way, with the readonly system partitions.
But it made upgrades really easy. And you have a second, backup system partition to boot from, if the update messed up things.
We had to move back to a 'standard' Linux distribution and now all those old habits are creeping up. It takes a lot of discipline (and enforcement) to avoid the applications of 'fixes' which get eventually forgotten.
That sounds very relevant: Why did you have to go back?
https://github.com/ghuntley/dotfiles-nixos
Steal away and enjoy.
The basic idea is to image with some lean/vanilla image, then run the script to put the system into the desired state. Kind of like Puppet, except far easier to understand and change.
Properly done, one can reimage pretty much at will, which is nice if there are lot of people making local 'root' changes on boxes.
edit: typo
Spot on. Much better than most CM tools.
The Nix setup is pretty hackish and does not track files changed by running applications.
What we really need is automatic version history on whole filesystems.
As a trivial example, one step in one's setup script might be "install package X". That can end up creating/modifying a lot of files, but many of those differences aren't necessarily meaningful or something one would want to carry into future reimages. And as things progress over time, those diffs might even be "wrong".
I liken it to whittling vs CNC. Either can be the right way to go, but usually at scale we end up doing better with CNC. And the best CNC program is a compact one.
It's not that much different than a one-liner that installs the package plus uses echo to write your config file. The magic is the "forced" reproducibility by default.
That said, unfortunately nix is a practical option for about zero percent of commercial shops. Sigh.
I did actually work at a place that tried it once. Summarizing, the introduction failed through some combination of politics and the perception that nix would introduce a lot of complexity vs a standard vendor distro.
That said, yes, something like that would be great. Using overlay + namespaces/cgroups could get us close to that. Then we'd have to detect which packages are completely passive (read only, shared dependency of many others) which are shared but have their own state, which are shared but their state is a mix of per-dependant states (so if multiple packages/apps depend on a DB service then the schemas/tables ought to be isolated), and so on.
No: it can only create a serialized sequence of changes that cannot be reordered. Also it's not aware of the context and meaning of a change.
> That said, yes, something like that would be great. Using overlay + namespaces/cgroups could get us close to that. Then we'd have to detect which packages are completely passive (read only, shared dependency of many others) which are shared but have their own state, which are shared but their state is a mix of per-dependant states (so if multiple packages/apps depend on a DB service then the schemas/tables ought to be isolated), and so on.
apt-get based distribution already track which files and directories are related to which package.
Also systemd unit files already can ensure that daemons are not making changes elsewhere.
Indeed, what's missing is tracking dependencies across applications and across hosts.
Linux distributions got most of it right decades ago (before the container dumpster fire)
Interesting...
I maintain a 'packages.config' xml file in a git repo that lists all the packages I decide I want to be installed and their versions. I intentionally don't list dependencies of the packages I want in this file, so I can remove a package later and I don't have to trawl through the huge list of potentially unnecessary packages now that I've changed what I want. I don't (typically) manage the version number listed in packages.config, I have a few scripts to help me do that:
install.ps1 has two uses: 1. install on a new system by cloning (or downloading the zip of) the repo and running the script, which will download and install the exact versions of the software that is declared in packages.config. 2. If I want a new package I can edit the packages.config to add the line then run install.ps1 (this second capability could use some UX work).
update.ps1 lists all of the packages that are currently out of date, gives you the option to update all of them, and then (regardless of which option you chose) rewrites packages.config with the currently installed versions. This allows you to use git to identify and manage the version differences between multiple windows installations and also upgrade all of them easily.
This doesn't configure any of the applications, but at least they're installed and they're the same version when you move from one computer to another. You can also quickly identify version differences between installations which makes debugging application version problems much easier. And you also don't forget what you have installed.
Virtually by definition if I was changing something, it wasn't in production at the moment, and I just learned my life was so much easier if I made sure my changes were really committed to state at the moment I made the change rather than learning it the hard way at 2am 6 months down the road, and desperately trying to remember what I had "fixed" and why.
Granted, these were still pets, but at least they were well-trained pets.
I rarely shutdown Linux desktop. I also keep desktop VMs with project context suspended to just reopen it next day or in a month to be at place where I stopped the work.
It's great because it separates /etc into vendor-provided defaults and your own customizations. It's not so great because it's not that automatic, you need to script it.
systemd/Lennart also explored this topic a bit: https://www.youtube.com/watch?v=pL0AMLiwPj8
https://www.freedesktop.org/software/systemd/man/systemd-vol...
takes discipline, but interesting.
If you want a TL;DR overview of NixOS then start here https://github.com/ghuntley/workshops/tree/master/nixos-work...
I'm just waiting for the day they can make the database immutable. It'll probably look something like git.
You could also call it "declarative infrastructure", because you declare in your configuration.nix how the system should look like: users, packages, file paths, etc.
The title says 'erase' rather than 'refresh', and does acknowledge they're 'mutable systems'.
Netflix had immutable databases back in 2012. I mean, as close as you can get I suppose. We could lose 1/3 of the nodes and keep running normally, and lose 2/3 and keep reading.
It was built with Cassandra and based on the Dynamo model.
There was also an open source, fully in memory database with the same reliability: https://github.com/Netflix/dynomite
I assume it is still that way, because it would be odd otherwise. But I just can't say for certain because I'm not there anymore.
But, more than anything this is an ad for nixOS.
Wouldn't event stores with CQRS effectively do this?
Additionally what if I want to look up a "row" of data that was modified 5 times on 5 different "columns" at various dates across 3 years? That's an aggregation job across 3 years of event data.
For event sourcing you still need to turn the "event" into an actual operation and record that database operation in a classic database.
Event stores just make the "event" the source of truth. It doesn't get rid of regular databases.
Traditionally other services read a single entity database/service and use that as a source a truth. Now a single button click records data across several databases and several services. It's not necessarily a better architecture, just different/buzz-wordy, and definitely more complicated.
Bottleknecks writes through an acid layer, makes reads against any single database connection (the value) immutable.
That's where the CQRS architecture comes into play. An event is recorded to the event store, the projection database is updated to the latest point in time, then any further reads are done from the projection database.
Me too. Mutable State is to Software as Moving Parts are to Hardware :-)
[link redacted]
It's called the point free style.
Look at every Dockerfile
FROM Ubuntu/Ubuntu
RUN apt-get update (or apt-get install) # BOOM — no longer possible to reproduce the build
Of course, they would be rebuilt when base layers change, but if you really want exactly the same image, you reference it by a digest, which will give you a point-in-time image.