Why Puppet, Chef, Ansible aren't good enough
domenkozar.com
domenkozar.com
Most of unix's standard set of tools (both the /bin programs and the standard C libraries) are written to make changes to state - but automation tools need to assure that you reach a certain state. Take "rm" as a trivial example - when I say `rm foo.txt`, I want the file to be gone. What if the file is already gone? Then it throws an error! You have to either wrap it in a test, which means you introduce a race condition, or use "-f" which disables other, more important, safeguards. An idempotent version of rm - `i_rm foo.txt` or `no_file_called! foo.txt` would would include that race-condition-avoiding logic internally, so you don't have to reinvent it, and bail only if anything funny happened (permission errors, filesystem errors). I does not invoke a solver to try to get around edge cases (e.g., it won't decide to remount the filesystem writeable so that it change an immutable fs...)
Puppet attempts to create idempotent actions to use as primitives, but unfortunately they're written in a weird dialect of ruby and tend to rely on a bunch of Puppet internals in poor separation-of-concern ways (disclaimer: I used to be a Puppet developer) and I think that Chef has analogous problems.
Ansible seems to be on the right track. It's still using Python scripts to wrap the non-idempotent unix primitives - but at least it's clean, reusable code.
Are package managers idempotent the way they're currently written? Yes, basically. But they have a solver, which means that when you say "install this" it might say "of course, to do that, I have to uninstall a bunch of stuff" which is dangerous. So Kožar's proposal is somewhere in the right direction - since it seems like you wouldn't have to ever (?) uninstall things, but it's making some big changes to the unix filesystem to accomplish it, and then it's not clear to me how you know which versions of what libs to link to and stuff like that. There's probably smaller steps we could take today, when automating systems. Is there a "don't do anything I didn't explicitly tell you to!" flag for apt-get ?
> it's not clear to me how you know which versions of what libs to link to and stuff like that
You'd typically link to the most recent version which you've tested against, and record it's base32 hash in your package definition. That is, a package by default contains exact identities of all of its dependencies - there is no "fuzzy matching" of packages based on a name and version range. The point here is that the packager of the application should know what he is doing, and by specifying exact dependencies, he is removing the "hidden knowledge" that often goes into building software. (in many cases, this is just ./configure && make && make install, but can be massively more difficult to reproduce a build, particularly if the dependencies aren't well specified.
The Nix build system knows which version to build against because there is only one version to build against in the chrooted environment where the build occurs - which is the one whose identity you specified in the nixpkg.
This is all rather new to me. Would it be fair to make the analogy? The build process is not a portable/cross-platform event, so you basically distribute a BuildFoo.exe with statically-linked libraries included.
You're roughly guaranteed that the BuildFoo.exe will run (they've got those libraries), and the user gets Foo in the end (either dynamically-linked or statically).
With the NixOS you get the additional advantage of configuration management and everything, including the kernel is handled by the package manager, which providers stonger guarantees that things should work as expected.
I mean specifically with regards to configuration management: that is, managing the part of software that developers intend to be modified so as to change the behavior of the program.
Maybe I just don't understand, but I don't see how this does anything to advance current config management dilemmas like how to merge a new upstream version of a configuration file with your site-specific changes; or how to deploy similar changes to large numbers of nodes at a time.
Modifying files in a git repo which are deployed to $ETC by ansible where modification triggers versus modifying files in a git repo which are used as "inputs" to a functional operating system seem like a largely cosmetic difference to me.
On topic: idempotency may be a red herring in this context. Unfortunately filesystems are designed with the assumption that every modification is inherently stateful. (It may be possible to design a different type of filesystem without this assumption, but every filesystem currently operates as a sequence of commits that alter state.) So installing a library or a program is necessarily stateful. What do you do if the program fails to install? Trying again probably won't help: the failure is probably due to some other missing or corrupted state. So indempotency won't help you because there's no situation in which a retry loop would be helpful. That is, if something fails, then whatever operation you were trying to accomplish is probably doomed anyway (if it's automated).
I think docker is the right answer. It sidesteps the problem by letting you create containers with guaranteed state. If you perform a sequence of steps, and those steps succeeded once, then they'll always succeed (as long as errors like network connectivity issues are taken into account, but you'd have to do that anyway). EDIT: I disagree with myself. Let's say you write a program to set up a docker container and install a web service. If at some future time some component that the web service relies upon releases an update that changes its API in a way that breaks the web service, then your supercool docker autosetup script will no longer function. The only way around this is to install known versions of everything, but that's a horrible idea because it makes security updates impossible to install.
It's a tough problem in general. Everyone agrees that hiring people to set up and manually configure servers isn't a tenable solution. But we haven't really agreed what should replace an intelligent human when configuring a server.
---
On the subject of what to do when something goes wrong: Sometimes retrying installing a package does fix the problem: if there was a network error, for example, and you downloaded an incomplete set of files, the next time you run it it will be fine.
If your package manager goes off the rails and gets your system into an inconsistent state, then you have a decision to make. Is this going to happen again? If not, just fix the stupid thing manually: there's no point in automating a one-time task. If it is probably recurring, then, you need to write some code to fix it (and file a bug report to your distro!). I do not believe that there is a safe, sane way to pre-engineer your automation to fix problems you haven't seen yet!
In the meantime maybe your automation framework stupidly tries to run the install script every 20 minutes and reports recurring failure. The cost of that is low.
Docker is awesome, for sure, and I'll definitely use it on my next server-side project. It isn't a magic bullet, though - you still have to configure things, they still have dependencies. Just, hopefully, failures are more constrained.
---
and on the point of upgrading for security fixes: the sad reality is that even critical fixes for security holes must be tested on a staging environment. No upgrade is ever really, truly guaranteed to be safe. I guess if the bug is bad enough you just shut down Production entirely until you can figure out whether you have a fix that is compatible with everything.
Since you originally outlined the requirements as:
Take "rm" as a trivial example - when I say `rm foo.txt`, I want the file to be gone.
then the file should be gone even if "the readonly bit" was set.
This is not only a contrived example, but a bad one, for system management. rm is an interactive command line tool, with a user interface that is meant to keep you from shooting yourself in the foot. rm is polite in that it checks that the file is writable before attempting to remove it and gives a warning. System management tools I would expect to call unlink(2) directly to remove the file, which doesn't have a user-interface, rather than run rm.
However, the system management tool doesn't start with no knowledge of the current state of the system, but rather one that is known (or otherwise discoverable/manageable). And then attempt to transform the system into a target state. They can not be expected to transform any random state into a target state. As such, the result of unlink(2) should be reported, and the operator should have the option of fixing up the corner cases where it is unable to perform as desired. If you've got 100 machines and 99 of them are able to be transformed into the target state by the system management tool and one of them is not, this isn't a deficiency of the system management tool, but most likely a system having diverged in some way. Only the operator can decide if the divergence is something that can/should be handled on a continuous basis, by changing what the tool does (forcing removal of a file that is otherwise unable to be removed, for example), or fixing that system, after investigation.
The other option is to only ever start with a blank slate for each machine and built it from scratch into a known state. If anything diverges, scrap it and start over. This is an acceptable method of attack to keep systems from diverging, but not always the pragmatic one.
I'm a fan of aliasing rm to move files into a trash folder
* override normal protection
* don't return error if it fails.
Having personally done this, in production, I'll give you my goto example.
set PACKAGE-DIR=/usr/local/bmelton/test/ rm -rf $PACKAGE_DIR
(actually rm -rf /)
Simply rm the file and handle the particular error case of the file existing by ignoring it. Other errors go through fine.
I've been doing this at work to try to wrangle a sense of control out of our various projects. I'm using Sprinkle, which is basically a wrapper around SSH.
What I'm finding is that most decent projects include idempotent ways to configure them. Apache, for instance, at least on Ubuntu allows you to write configs to a directory and then run a command to enable them. Sudo also has the sudo.d directory, cron has cron.d. Just write a file.
> Is there a "don't do anything I didn't explicitly tell you to!" flag for apt-get ?
I would consider this to be overly tight coupling. We should let dpkg manage the OS packages, and if the system's state needs to be changed, you can simply re-build it and run an updated version of your management scripts.
You don't really want to start getting into the game of trying to abstract over the entire domain of systems engineering. CM, in my opinion, should solve one and only one problem, moving system state between the infrastructure/cloud provider defaults and a state where application deployment scripts can take over. Every necessary change to get from point A to point B gets documented in a script. There are only two points on the map, and only one direction to go.
How do I differentiate to ignore one error and not others? Matching a string? What if this is supposed to be portable? Hard code strings for every version of rm ever made?
>What I'm finding is that most decent projects include idempotent ways to configure them
Those are modifications debian makes. Lots of software supports including files, which lets debian do that easily. But sudo has nothing to do with you having a sudo.d directory, that is entirely your OS vendor. And having that doesn't solve the problem. What happens when I want to remove X and add Y? You need to have the config be a symlink, so you can do the modifications completely offline, then in a single atomic action make it all live.
However, with the rise of virtual machines, that's a trivial operation in many cases. Even on bare metal hardware it's not a big deal, as long as you can tolerate a box disappearing for an hour (and if you can't, your architecture is a ticking time bomb).
In fact, starting from a clean slate each time basically makes Fabric scripts a declarative description of the system state at rest... if you squint hard enough.
The problem is that running commands via SSH isn't really a deterministic process. It's fairly predictable, but it relies on a lot of inputs that can change unexpectedly. So it mostly works, but building a server has to be cheap and easy, because you throw them away a lot.
I think my ideal system would be sort of a cross between this and NixOS. We build servers with NixOS-style stateless declarative configuration. But then instead of using NixOS mechanisms for functionally/deterministically modifying the running system, we treat running servers as immutable and disposable and build new ones whenever we need to change something.
That seems like it would give maximum maintainability and robustness.
I'm also wondering if NixOps can build AMIs the same way Aminator does, by mounting a volume on a builder instance, writing files to the volume, then snapshotting and making an AMI based on the snapshot. (In contrast to snapshotting a running system.) That would be truly awesome.
I've only been playing with NixOps a short while, maybe there's already a way to do these things, or they're easy to implement on top of NixOps. Looks very cool, though.
The author majorly misses out on what chef and puppet are, since both are declarative, while chef feels procedural because you are writing ruby.
These tools are based on the idea of declaring how the system should be, and a way to get there. They also have the advantage of allowing for a design that applies to multiple machines.
Maybe some of the ideas expressed are novel and will shape the future, but mostly I see a lot of hand-waving and complaining about things that are Pretty Good Ideas(tm) like FHS, which most people still don't understand.
At the end of the day, he's essentially proposing static compilation for everything. That creates an amusing amount of attack surface, but I'll surely find myself taking a look at Nix and maybe what the OP is talking about will iterate toward something sensible.
That we should want something for our desktop which is a strict match for what we want in server farms is kind of silly, IMO. See Pets v. Cattle.
It's not technically declarative, but in the limited circumstances for which it makes sense, it feels declarative as you're thinking in terms of what needs to be there without worrying at all about the current state.
While we're analogizing, it's like stocking up your fridge to an exact state. Recipe A is:
* Check shelf for residue, clean if exists
* For each existing egg, if egg is off, throw out
* If egg carton contains yolk from broken egg, obtain new carton, transfer each egg
* While egg count less than 12, purchase and add eggs
However, if we just throw out the entire old fridge and buy a new one, we get the recipe:
* Purchase a dozen eggs. Place on shelf.
If you can afford a new fridge each time you go shopping, I don't think it's a bad algorithm.
> For each node a list of recipes is applied. Within a recipe, resources are applied in the order in which they are listed.
So it reads in a declarative fashion, but it doesn't actually build a dependency graph (and a static representation of config) like Puppet does.
Anybody who's done enough systems programming can put together something similar to Chef--it's the same old imperative stuff everybody else has written, except with an idempotent sugar-coating and a client/server model to distribute the script^H^H^H^H^H^Hcookbooks. ;-)
I'm not putting chef down--the fact that it's plain old ruby instead of a slightly-odd DSL is a definite advantage in some cases, but I don't think that anything else out there is really in the same category as Puppet.
Of course, there are things about Puppet that bug me, but I really think that it's the Right Thing at its core. The idea that it's not "good enough" seems like a consequence of incorrectly conflating Puppet with the systems that it manages.
They would all still be relevant with this new packaging system.
For some reason, this came to mind: https://xkcd.com/927/
Well, yes, replace yum, apt, etc. But once you have a functional package management system, you don't need Puppet, Chef or Ansible, because the same stateless configuration language can be used to describe cluster configurations as well as packages. So build a provisioning tool based on that, instead.
That provisioning tool is called NixOps. The article links to it, but doesn't really go into detail about NixOps as a replacement for Puppet et al.
(Of course, whether something's a good language for expressing particular kinds of tasks is another question)
The Nix model is the only sane model going into the future where complexity will continue to increase. Nobody is forcing you to switch distros now, but hinting that you'll probably want to look into this, because the current models might well be on their way to obsolescence.
* I fail to see how baking configuration into packages can a) reduce complexity; b) obviate the need for configuration management.
* How do you diff filesystem images when there are unique PREFIXes in the paths?
A salt module would be fun to play around with.
How likely am I to volunteer to troubleshoot your unique statically-linked packages?
This article has little to do with config management. A better title would be "apt, yum and brew aren't good enough; we can do better".
`sudo apt-get install nginx` just works. Perhaps they're doing it "wrong", but there are thousands of people who are making sure it just works.
I have some of the problems described in the article, but it only happens when I can't use the package manager. It happens when I have to compile from scratch, or move things around or mess with config files, et cetera.
For me, phoenix servers and docker are the solutions. Maybe they're not as pretty as what he describes, but there is a solution that works.
True, but that is only half of the problem. Once you have nginx installed you have to configure it, which requires another layer of automation. Package installation is the trivial part of the stuff done by Ansible, Puppet, etc.
You could package up pre-configured .deb's, but you lose the "thousands of people" aspect and are back to "compile it from scratch". It's significantly less entertaining, particularly if you're trying to do it repeatably.
I think the real point of Nix is that you use the same mechanisms for both package management and local configuration.
Problem identified!
Nix doesn't just solve user-facing problems, it's first and foremost a solution for developers to package applications that should just work. You won't need thousands of people making sure a package works - but one. If it builds for him, it should be for everyone, by virtue of the fact he specifies exact dependencies and build instructions, and performs the build in a chrooted environment which should be reproducible exactly on a users machine.
- The repo is down - External network doesn't work - You are missing a dependency not in your apt-cache - It conflicts with another package due to a dependency
All of which are possible and happen.
upgradepkg --install-new /dir/*.tgz
apt-get ensure that application will get the right version of a library for its execution.
'Untaring' doesn't mean the application will work OK with their dependencies (since there's no verification).
So anyone could do a dpkg -i --force-all *.deb. Its the same thing.
dpkg -i -E -G -B --force-overwrite --force-breaks --force-conflicts --force-depends --force-depends-version *.deb
Apt-get gives you some insurance in that it (mostly) won't screw up your system if you try to do something wrong. But it also adds levels of complexity that can make it very difficult to get anything done, even if you know what you're doing. Both systems will work, but only one is more likely to do what you want it to do without extra effort. And just besides maintenance woes, it's much more difficult to recover a broken apt system than it is to recover an installpkg system.
If you ask an admin "What's more likely to succeed: rsync and upgradepkg, or apt-get", the answer is the former, because the level of complexity of the operations is so much smaller. As long as your packages are correct, everything else is determined to succeed. With apt-get, you have many more conditions to pass before you get success.
That's because dpkg does more. It's designed to ask you questions interactively when configs change and get you to look at things. If you ignore config files and just blindly install new binaries (even yum/rpm do this!) then you end up with an upgrade that "worked" except that the new binaries won't run at all because the config files are now invalid.
Failing silently like that is hardly better. I would say it's objectively worse.
This is one of the things that Docker solves, as you are able to test and deploy using exact filesystem snapshots.
apt-get install nginx=x.x.x
I'm not aware of a case where that would install anything but what you ask for.
Sure it's possible, but it's unlikely. There are loads of debian/ubuntu apt mirrors. apt-get (or aptitude) downloads dependencies, package maintainers ensure that there aren't those conflicts.
#define if(x) if ((x) && (rand() < RAND_MAX * 0.99))
Now you might object that C if() statements are far more commonly executed than "apt-get install". This is true, but to account for this you can adjust "0.99" above accordingly. The point is that there is a huge difference between something that is strongly reliable and something that is not.
Things that are unreliable, even if failure is unlikely, lead to an endless demand for SysAdmin-like babysitting. A ticket comes in because something is broken, the SysAdmin investigates and found that 1 out of 100 things that can fail but usually doesn't has in fact failed. They re-run some command, the process is unstuck. They close the ticket with "cron job was stuck, kicked it and it's succeeding again." Then go back to their lives and wait for the next "unlikely but possible" failure.
Some of these failures can't be avoided. Hardware will always fail eventually. But we should never accept sporadic failure in software if we can reasonably build something more reliable. Self-healing systems and transient-failure-tolerant abstractions are a much better way to design software.
Most of us happily use git, where if you ever get a collision on a 128-bit hash it will irretrievably corrupt your repository. It's just not worth fixing when other failures are so much more likely.
Clearly if Nix is immature, that is a risk in and of itself. But all else being equal, a declarative, self-healing system is far better than an imperative, ad hoc one.
Other risk factors don't make the difference "go away", because failure risks are compounding. Even if you have a component with a 1% chance of failure, adding 10 other components with a 0.1% chance of failure will still double your overall rate of failure to 2%.
This is not to mention that many failures are compounding; one failure triggers other failures. The more parts of the system that can get into an inconsistent state, the more messed up the overall picture becomes once failures start cascading. Of course at that point most people will just wipe the system and start over, if they can.
File hashes are notable for being one of the places where we rely on probabilistic guarantees even though we consider the system highly reliable. I think there are two parts to why this is a reasonable assumption:
1. The chance of collision is so incredibly low, both theoretically and empirically. Git uses SHA1, which is a 160 bit hash actually (not 128), and the odds of this colliding are many many orders of magnitude less likely than other failures. It's not just that it's less likely, it's almost incomparably less likely.
2. The chance of failure isn't increased by other, unrelated failures. Unlike apt-get, which becomes more likely to fail if the network is unavailable or if there has been a disk corruption, no other event makes two unrelated files more likely to have colliding SHA1s.
But, it's "yum install nginx" some places. And some package names vary by distribution and even versions of same distro. And there is post install configuration that has to happen. And there is coordination with the proxy cluster, the memcache cluster, the db cluster, monitoring system that has to happen. And many sophisticated users will need a custom nginx or extension which requires building locally (and all those dependencies) or private repo. And you're glossing over configuring sudoers so the user you are logged in as can even sudo in the first place.
tldr; Provisioning is fucking hard.
It's very rare I don't have to change configuration files for any software that doesn't come with the default OS install. It's also pretty rare I can use the default packages that come with Ubuntu, I very often need different versions or ones that are custom compiled.
I'm looking forward to Docker, but can't use it quite yet.
Query: Why are you screwing around? Response: XKCD 303.
I am basically a Perl developer by trade, and have been building and maintaining customized Linux distributions for large clusters of enterprise machines for years. I would still rather use shell scripts to maintain it all than Perl, or Python, or Ruby, or anything else, and would rather use a system of 'stupid' shell scripts than invest more time in another complicated configuration management scheme.
Why use shell? It forces you to think simpler, and it greatly encourages you to extend existing tools rather than create your own. Even when you do create your own tools with it, they can be incredibly simple and yet work together to manage any aspect of a system at all. And of course, anyone can maintain it [especially non-developers].
As an example of how incredibly dumb it can be to reinvent the wheel, i've worked for a company that wanted a tool that could automate any task, and that anyone could use. They ended up writing a large, clunky program with a custom configuration format and lots of specific functions for specific tasks. It came to the point where if I needed to get something done I would avoid it and just write expect scripts, because expect was simpler. Could the proprietary program have been made as simple as expect? Sure! But what the hell would be the point of creating and maintaining something that is already done better in an existing ages-old tool?
That said, there are certain tasks i'd rather leave to a robust configuration management system (of which there are very few in the open source world [if any] that contain all the functionality you need in a large org). But it would be quite begrudgingly. The amount of times i've ripped out my hair trying to get the thing to do what I wanted it to do while in a time and resource crunch is not something i'd like to revisit.
Lots of people design terrible, overly complicated systems. This is an argument for good design, not for ad hoc shell scripts for everything.
Would you write a build system in shell? Why not? Because "make" is far better suited to the job. (Even "make" has a lot of room for improvement, but that's a story for another day).
As another example, even Debian (known for its conservatism) decided that systemd was a net benefit over sysvinit, where the latter could quite accurately be described as a pile of shell scripts. Abstractions that suit the problem domain are far superior to ad hoc imperative bags of commands.
You could definitely write a build system in shell. In my opinion, "the job" is subjective and you shouldn't have to pick one tool for it. But I would prefer a simple one where I can see pretty much everything it's doing up front and not have to download source code or search through docs.
Debian picked up systemd because it was a popular decision, not because it was better. As far as i'm aware there was nothing broken in sysvinit that systemd fixed; there was just a vocal minority that wanted some new features and insisted it be shipped as default.
And in defense of shell scripts, nearly every single operating system you can think of other than Windows uses scripts to initialize (hell, even Windows has some startup scripts), and that is an abstraction that's suited to the problem domain. It's not like for the past 50 years people have been too stupid to come up with something better than shell scripts. The whole point of init scripts is to let anyone be able to modify their operating system's start-up procedures quickly and easily. If we didn't need to modify them to suit our individual cases they'd be statically compiled C programs or assembly (which, by the way, I have done in the past; it's not a good idea)
You likewise have to read docs or the source to know what your shell is really doing. Or search through sysvinit docs to know which shell scripts it will run and when. I have to look up how to use shell conditionals every time, and it's extremely easy to get it subtly wrong such that it's not doing what you think it's doing. There are subtle distinctions between shell built-ins and programs of the same name, and it's very unclear in some cases which is being called. The shell is far from being a beautifully simple and transparent abstraction. Perhaps you know it too well to be able to appreciate this.
The shell and sysvinit are abstractions over writing C programs that make POSIX library calls directly. It's all abstractions unless you are writing machine code directly, (and even then you have to read docs to know what the instructions really do). So given that we're all using abstractions, we ought to use the best ones; the one that gives us the most advantages and the fewest disadvantages given the problem we are trying to solve. The ones that have the greatest combination of simplicity, transparency, and expressiveness.
If you want to make sure that only one copy of a program is running, is it a better fit to write a shell script that creates a pidfile, checks to make sure there's a running process that matches it, and that handles a million cases about how the two can fall out of sync? Or is it better to be able to express directly "I want one copy of this program running" and then let the abstraction handle the hard cases?
You might say "well the shell script is easier to debug, because I wrote it and I understand it." Well of course, you always understand the things you wrote yourself. But if I had to debug either a pidfile-making shell script or a well-designed babysitter tool, I'll take the specific tool every time. If it's mature, it's also a lot less likely to have bugs and race conditions.
> Debian picked up systemd because it was a popular decision, not because it was better. As far as i'm aware there was nothing broken in sysvinit that systemd fixed; there was just a vocal minority that wanted some new features and insisted it be shipped as default.
I don't think you're very well-informed about this. Your claim of a "vocal minority" is particularly suspect, considering the decision was made by a vote of the technical committee. Though there was disagreement, almost no one was in favor of sysvinit. And the position paper in favor of systemd identifies specific things that are lacking in sysvinit: https://wiki.debian.org/Debate/initsystem/systemd
Your debugging argument is bonkers. You claim shell scripting is too hard, then say it must be easy to troubleshoot a "well-designed babysitter tool", which requires WAY more domain-specific knowledge of debugging tools! If you don't know how to write bash scripts, you sure as hell aren't going to have an easy time figuring out why your package manager's borking on an update of openssl.
Did you even read the executive summary of the position paper? "Systemd is becoming the de facto standard" .. "Systemd represents a leap in terms of functionality" .. "People are starting to expect this functionality [..] and missing it could [..] make Debian lose its purpose." They only want it because it has new features, and people are starting to expect it. It's a popularity contest, and systemd won.
You seem to have a really shell-script-centric view of the world, as if the shell is somehow a fundamental and "pure" tool, and everything else is a more complicated version of a shell script.
What you are missing is that the shell is just another tool, and does an awful lot behind the scenes to achieve the functionality that appears "simple" to you. Bash is tens of thousands of lines of C and its manpage is hundreds of thousands of words. Using the shell is not "free", complexity-wise. The shell is a programming language, and not a particularly well-designed one at that. Shell script programming introduces its own peculiarities, and it is known not to scale in complexity very well.
> Your debugging argument is bonkers. You claim shell scripting is too hard
No, I claim that the shell is not a particularly simple tool. There is a difference.
I write JIT compilers for fun. I don't shy away from things that are hard. But my brain has only so much room for uninteresting and arbitrary facts. The very pecular way that conditionals work in the Bourne Shell is not something my brain can be bothered to remember.
> Did you even read the executive summary of the position paper?
You have edited out precisely the parts that contradict your position: "It replaces the venerable SysV init with a clean and efficient design [...] It is better than existing alternatives for all of Debian’s current use cases." Yes, the momentum of systemd was clearly a factor, but your claim that it is only but a popularity contest is not a conclusion that a dispassionate observer would reach.
if ! grep foo /some/file ; then
run_something
fi
if [ $INTEGER -eq 0 ] ; then
run_something_else
fi
if [ "one" = "two" ] ; then
do_something
fi
Also, those parts you quote don't contradict anything. It's saying systemd has a "better design", which means it is a shinier, fancier new toy to play with. But the paper never once points out any flaw in sysvinit. But that's obvious; Debian had been chugging along for over over two decades with sysvinit without any problems. If your argument is that suddenly, after 20 years, someone realized sysvinit was some horribly flawed design that needed to be replaced, and it just so happens that systemd came along right when they realized it, I don't buy it. What the paper does spell out, though, is all the advantages of systemd for things other than system init. Basically it says "Hey, we want all these new features, and we need systemd to replace init for it all to work, so please just go along with it because it's a much better design."
FOO=""
if [ $FOO = bar ] ; then
echo "equal!"
else
echo "not equal!"
fi
test.sh: line 4: [: =: unary operator expected
if [ = bar ] ; then
One solution you sometimes see to this is:
FOO=""
if [ x$FOO = xbar ] ; then
echo "equal!"
else
echo "not equal!"
fi
> It's saying systemd has a "better design", which means it is a shinier, fancier new toy to play with.
You seem dismissive of new technology. If you want to keep using CVS while browsing the web with lynx and sending your email with pine, more power to you (after all, graphical web browsing is just a "new feature"). But the rest of us are moving on.
What's really funny about systemd is I think that all its features have tons of value, and I would definitely use them. But I also think its creators are completely fucking batshit insane for making it mandatory to replace huge chunks of the operating system just to get those features. You should be able to just run systemd as a normal user process and still maintain the same level of functionality, but for some fucked up reason somebody thought it would be a great idea to make it a completely non-backwards-compatible non-portable operating system requirement. It's a stupendously bad idea, and the only reasoning anyone can come up with for why they designed it that way is "It's Advanced!" Of course, I should add the caveat that I don't care at all about boot times, and so people who are obsessed with short boot times will find systemd very refreshing, in the way an Apple user finds replacing their old iPhone with a new iPhone very refreshing.
I don't feel that's a fair summary of the lengthy debate had about this. There is a long page here [1] listing the reasons for selecting systemd. There is also [2], a very good summary by Russ Allberry of the different init systems being suggested (including staying with sysvinit). Debian has rarely been accused of taking a major technical decision because it is hip.
1: https://wiki.debian.org/Debate/initsystem/systemd
2: https://lists.debian.org/debian-ctte/2013/12/msg00234.html
I believe solaris startup since solaris 10 uses smf which is configured in xml files.
I believe apple's init launchd is configured with Property lists.
Edit: OK, sarcasm aside, I can't see how this comment amounts to more than "I don't need anything fancier than shell scripts." Fine, you don't. But there are lots of people out there that do.
For several years, I ran a cluster that consisted of 10 or so EC2 instances that had to rebuilt and replaced every day. Shell scripts didn't cut it for me. Chef and Puppet were painful. I ended up using Fabric, boto and a pile of custom Python code.
And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.
Four years ago, I was going this with shell scripts, a little bit of Python, and a cluster management tool called Rocks (built in Python and shell scripts). This was for a 5500+ Linux server cluster processing data from the CMS detector at the LHC. I had the ability to do a rolling reinstall of Scientific Linux/CentOS while the cluster was actively processing data with no interruption to the end user jobs.
Everyone keeps reinventing the wheel, but in most cases, shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
I've written loads of both, and I realize the thread had already started down this path, but how is a shell script really different from a python script? If you're targeting Bourne, your shell isn't a separate installation, but it isn't onerous to install your favorite python version on machines you control. Sure, python has an ecosystem of libraries that shell doesn't have, but you don't have to use those libraries if you don't want to. What terrible consequences does someone who prefers python to shell scripting face?
Terrible consequences? Probably not. Shell script dependencies are easier to handle, because the commands you're relying on are built right into the OS. They're going to be there because they have to be. How often does sed, grep, awk, and cut change?
Regardless of if it "worked", I think mixing languages in an application is a bad idea, and should be avoided whenever possible. There are a few reasons, 1) maintainability - you're requiring anyone to know more things to maintain the application 2) Attack surface - lots of security bugs come in between boundaries between tools 3) Correctness - it's harder to automate testing
Did you have any sort of automated testing?
Just because shell scripts can do a lot of heavy lifting, doesn't mean they should.
Knowing Python or Ruby doesn't mean you should write every system administration task around it.
Who writes testing around 50-100 line shell scripts?
Just to be understood correctly, you're saying the exception proves the rule? Sidestepping the necessity of the task above, which I'm sure was rendered easier by Fabric or whatever, but it's not like that's a typical use case. You would have to ignore nuance to extrapolate that to generic VM instances from known-state images, etc. that might respond just as well to a bunch (not even a bunch) of scripts.
The GNU build system (+pkg_config) is really awful to work with when it doesn't magically work the first time. You'll often get errors such as m4 macros not being defined (but which don't hint at what dependency is missing), and more often than not you'll need to build several dependencies for the software you want, which are often poorly specified. It's not uncommon for one to waste hours or whole days trying to build a piece of software (and if you're lucky, he will blog about his experience so that you can spend less time when you inevitably google the problem, because that's always easier than trying to understand and debug the build scripts yourself.)
Nix reduces this by requiring exact dependencies and build instructions - there's no "hidden knowledge" that the software developer/package maintainer had which the user should not also have available, so at most, only one person needs to experience the pain of building the software and others can reproduce based on his precise instructions, rather than his blog post.
I can't honestly say that there will be no problems when using Nix, but a build error in nix is a bug in the distributed nix packages, not just some unknown problem on the users system which he needs to fix.
I agree with you that reinventing the wheel is usually bad, and you get a long way by reusing existing programs with shell scripts, but sometimes you need to throw away your assumptions about how things work because a new model is significant enough to rethink the old. (We didn't build aeroplanes by attaching wings to cars).
also, it's not always better to use existing systems. sometimes rolling something new is indeed better.
but i had to work in a place where everyone was a ruby fetishist. don't get me wrong ruby paid my rent for the last couple of years, but i too thought that using it for basic system administration tasks made absolutely no sense. at least what they used it for.
it's used in chef/puppet for one reason only. because it's easy to make a dsl in ruby, and it's good for that purpose actually. although i find chef terribly overengineered
Currently, I have automated system set up where basically a cluster of vm's in nodes of two communicate with each other to pass over where the other server in a node left off in its operation (or dies from timeouts) running nginx/gunicorn/django.
And those servers on each node are monitored by another server (running apache/php/mysql) that checks the progress/status of the operations and may send requests (reinitialized a node that stopped running) to the nodes where those bash scripts (concatenating files, finding a specific place in one of the files to help reinitialize a process in a node) are executed and piped through back to the monitoring server.
It is way more complex now, but surprisingly, I don't have troubleshoot not even close to as much as before since I automated that.
That's a misunderstanding. It's like saying "Java is a reinvention of Haskell". Neither is a reinvention of the other, they are two very different beasts. You don't have any OS whose entire state (package and configuration) is described at a central location and can be recreated by a single command (outside of research projects, possibly). You may disagree with the approach they take, but such a blunt dismissal dismisses at the same time any kind of original approach to a problem.
Speaking of which, you speak in favour of shell as a solution to automation. What you are saying is that you push the complexity on the side of the shell script developer. In my experience, declarative systems tend to have more complex innards, but it also means more maintainability, and a flow much easier to understand. This is the kind of tradeoff you make when forsaking sysvinit in favour of systemd.
I have the shell scripts to prove it makes you think more convultedly in order to get things done that are cleaner in other languages.
I think the complexity of automation is more a symptom of the problem space than the tools. It's just a hairy problem. Computer Science seems to largely focus on the single system. Managing "n" systems requires additional scaffolding for mutual authentication and doing file copies between different systems. It also requires the use of directory services (DNS, LDAP, etc…)
I like the analogy of comparing the guitar player to a symphony orchestra. When you play the guitar alone, it's easy to improvise, because you don't need to communicate your intent to the players around you. When a symphony does a performance, there is a lot of coordination that needs to be done. Improvisation is much more difficult. That is where Domen is right on target, we can do better. Our symphony needs a better conductor.
Thousands and thousands of these: (debian_6|debian_7).(role_postgres|role_postgres_exception)::
And ordering? Nah, just run it 3 times in a row. :\
As for the run 3 times in a row, this is an explicit design decision in CFEngine, because the state of the machine changes as you operate on it, so you cannot assume your plan remains valid during agent execution.
You can create atrocities in a lot of programming languages, and CFEngine is no exception to this rule. CFEngine is highly dynamic, but with great power comes responsibility.
I'll take a stab at this one too. I'll try another music analogy. I have a guitar with big frets. When I press the strings down too hard while playing chords, it causes the notes to bend out of tune. Playing in tune, requires a light touch.
Procedural programming is really familiar, even in the OO world where you attach these nouns to your verbs called "instances" of a "class". You call a procedure and it has a side effect or returns some kind of value. Familiar right?
Converging to a desired state is just different, because you focus on the outcome instead of the order of operation. If you get caught up on debugging the order of operation instead of the desired end state, it will take longer to troubleshoot a convergence failure.
In other words, loosen up your "procedural" grip on convergence. Playing in tune, takes a light touch ;)
Actually, most people don't know it exists. Poll your average "devops" person and most will believe that chef or puppet created the concept of configuration management. Cfengine is less terrible than puppet and chef, but it is still not close to good enough.
>Its utterly atrocious
If you write atrocious policies, sure.
It's a good write-up. The idea of a stateless, functional, package management system is really important in places like scientific computing, where we have many pieces of software, relatively little funding to improve the quality of the software, and still need to ensure that all components can be built and easily swapped for each other.
The HashDist developers (still in early beta: https://github.com/hashdist/hashdist ) inherited a few ideas from Nix, including the idea of prefix builds. The thing about HashDist is that you can actually install it in userspace over any UNIXy system (for now, Cygwin, OS X, and Linux), and get the exact software configuration that somebody else was using across a different architecture.
I've been looking into Nix (the package manager) recently, and it also can be installed on OS X and various Linux distros (I've seen Cygwin mentioned, but I don't think this is well supported).
If you're familiar with both, what does HashDist provide over Nix?
As an example, if you'd like to build a Nix system, you're going to need to compile or download an independent libc. HashDist is capable of bootstrapping an independent libc as well, but we default to using the system libc.
We choose to seat our "pure" system on top of the system compilers and libc, but by default, install our own Python and other high-level libraries. We also can integrate with other package managers, and there's an example branch in our repository right now showing integration with Homebrew.
Literally every single one of the tools he said we can do better than, could use his package manager with almost no changes.
You can specify the packages you want in Ansible, and it will abstract away everything else, giving you what you asked for. Ansible's config language wouldn't look any different if it were using nix (and it probably can), so what he says doesn't follow:
Ansible (et al) aren't enough => We improve a part of the underlying system => Ansible looks exactly the same, but is magically now enough.
There is a fundamental difference between Ansible and co. and NixOS. These tools take you from an unknown state, look at the part of this state you configured in the Ansible files, and apply the modifications necessary. NixOS will take its cue from a single file (possibly with includes) describing the entire machine.
Concretely, if you tell Ansible "package X should be present", apply the configuration and then remove this line before reapplying the configuration, package X will still be present, even though it's not listed in the configuration. I don't mean that as a criticism of Ansible, for my purpose it's about as good as it gets, but it's two different paradigms.
Ansible (et al) aren't enough => We improve a part of the underlying system => We build a new tool that makes use of the improvements and is therefor better than Ansible.
Individual users may benefit from versioning their config files, but usually not their programs. Big business might do both, or just as likely screw it up and version code but not configs (I'm looking at you, crontab).
I think the more useful way to think of packages and configuration are that they are necessary compliments. An unconfigured package doesn't do what you want, and configuration without a package doesn't do anything at all.
The theory is, they're both components of system state, so why not manage them together?
But to me it sounds like saying that text and layout are necessary compliments for a magazine, so why not manage them together? Because it turns out that is a horrible idea. Anyone who does professional publishing manages them separately.
"Do it all" tools rarely do it all well.
If you follow it, you'll get a package like "mycorp-nginx", which is an atomic piece of configuration management that not only controls nginx's configuration wherever you deploy it, but also nginx's version on those same machines.
Basically, with this strategy, instead of installing nginx, you only install mycorp-nginx; the config-package then installs the thing it's configuring as a dependency. That dependency is locked to a specific version, so it'll only upgrade when you push out a config-package that changes that dependency.
In practice, everybody screws something up, whether individually or part of a small/big business. The tools are typically not the problem; the humans and how they use them, are.
I think there are a few more steps in there. Assuming identical OS and arch, you've probably got:
* software packages that should be identical on all nodes in the cluster
* a set of software packages that should be installed on all nodes
* configuration that should be identical (e.g. 99% of an nginx config)
* configuration that is unique to the host (e.g. a hostname)
Or perhaps:
* packages limited to a single OS (traditional software packages for Linux)
* packages limited to a single organization (identical portions of configs)
* packages limited to a single node (hostname)
Anyway, I'm playing around with using Nix for all but the unique-per-host stuff. I see potential, but it may or may not pan out. I seem to always need a custom version of nginx, or ruby, and I find making custom Debian packages rather cumbersome, even with a fair amount of experience.
The whole point is that those aren't different things. Once you create a proper package management system, it is also a proper configuration management system. The article didn't do a great job of explaining it, but fire up a few VMs and give NixOps a try.
Doing updates of any kind to a running system is unnecessarily complex when we have all the tools to treat entire VMs/containers as build artefacts that can be tested as a unit.
How do you guys do this for Rails apps? Django apps? is this only an issue with Java web apps?
There are lots of tools within Rails and Capistrano that hopefully get you to a state approaching 'idempotent' deployments, but they fairly often aren't.
This is not a new idea - the "application directory" dates back to riscos as far as I'm aware. It's been carefully examined many times over the decades, and hasn't been widely adopted because it leads to massive duplication of dependencies, everything in the system has to be changed to be aware of it, and there are less painful ways to solve or avoid the same problems.
NixOS doesn't duplicate dependencies. Instead it makes everything read-only. Each dependency is to a specific version of the package, and everything that uses that specific package version uses the same copy. If you want to upgrade to a new version of a package, that implies a new version everything that depends on it as well. (Or at least the application that you want to use the new version.) It still uses more space than a traditional system, but not as much as duplicating everything.
Being able to online a new server and have it automatically install all the required software, setup all the configs just by its hostname is a beautiful thing.
They sit on top of the package manager. Ansible is more about doing commands en masse, Puppet is more about ensuring a consistent state en masse.
What happens if you want to ensure that your server farm is all running the same version of nginx? What if you want to ensure the configuration files are all in a consistent state?
You can script it yourself if you really want to, but it's a solved problem at this point. Puppet's mission in life is to notice when a server has deviated from your specified configuration, report it, and haul the box (kicking and screaming if necessary) back into compliance.
Manual scripting doesn't scale beyond 100 servers or so. You don't have enough hours in the day.
Its "playbooks" are by default meant to be idempotent -- you can run them over and over ensuring that a system is in a consistent state.
What happens when the configuration of nginx needs to be slightly different on each server?
What happens when the configuration of nginx needs to change?
What happens when you need to install a custom version of nginx that's not in your OS repository?
What happens when you need more than one instance of nginx running on the server?
rdist was a good method in the old days which scaled a little better than one-off, but we're in the pull vs push world now.
If you have a really large deployment, you can set up your own repository, or a mirror of existing repositories. If that is still not enough, you're Facebook.
By the time you're done putting all your apt-gets and your config files and whatnot in some shell script to automate it all away, you've reinvented a poor clone of puppet/ansible/chef.
apt-get is fine, it's just that at some point you need an automation tool to trigger it. A lot of chef recipes rely on the underlying package manager, that's fine.
Simple solutions regularly fail when you add zeroes.
I know, I know 640k will be enough for anybody, but is anybody's startup really failing because of nginx point releases?
Now Windows programs want to put stuff in C:\Progra~1\APPNAME, C:\Progra~2\APPNAME, C:\Users\Applic~1\APPNAME, C:\Users\Local\Roaming\Proiles\AaghThisPathIsHuge, and of course dump garbage into the Registry and your Windows directory as well. And install themselves on your OS partition without any prompting or chance to change the target. And you HAVE to do the click-through installation wizard because everything's built into an EXE using some proprietary black magic, or downloaded from some server in the cloud using keys that only the official installer has (and good luck re-installing if the company goes out of business and the cloud server shuts down). Whereas in the old days you could TYPE the batch file and enter the commands yourself manually, or copy it and make changes. And God forbid you should move anything manually -- when I copied Steam to a partition that wasn't running out of space, it demanded to revalidate, which I couldn't do because the Yahoo throwaway email I'd registered with had expired. (Fortunately nobody had taken it in the meantime and I was able to re-register it.)
I've been using Linux instead for the past years. While generally superior to Windows, its installation procedures have their own set of problems. dpkg -S firefox tells me that web browser shoves stuff in the following places:
/etc/apport
/etc/firefox
/usr/bin
/usr/lib/firefox
/usr/lib/firefox-addons
/usr/share/applications
/usr/share/apport
/usr/share/apport/package-hooks
/usr/share/doc
/usr/share/pixmaps
/usr/share/man/man1
/usr/share/lintian/overrides
Some of these directories, e.g. /usr/bin, are a real mess because EVERY application dumps its stuff there:
$ ls /usr/bin | wc -l
1840
I welcome the NixOS approach, since it's probably as close as we can get to the one-directory-per-application ideal without requiring application changes.
as far as the "stateless" thing, this could have been explained in a far simple manner IMO.
1) No library deps:
"all system packages are installed in /mystuff/version/ with all their libs, then symlinked to /usr/bin so that we have no dependencies" (that's not new either but it never took off on linux)
2) fewer config deps "only 4 variables can change the state of a config mgmt system's module, those are used to know if a daemon should restart for example"
So yeah. it's actually not stateless. And hey, stateless is not necessarily better. It's just less complicated (and potentially less flexible for the config mgmt part).
Might be why the author took so long to explain it without being too clear.
(speaking from second hand knowledge) They don't go into much of the really interesting detail in the paper. The awesome part of all of that was that everything an application required to function was under it's own tree, you never had any question of providence, or if the shared libraries would work right on the given release of the OS you might be using. And it worked from any node on the global network. This problem has been solved, most people didn't get the memo.
I only assume the reader has basic command-line familiarity, but I try to make the writing approachable for both newer admins and veterans.
Right now I'm working on a chapter on security, and another on Roles, and will probably publish an updated version next week.
I'm also working on getting one or two really good technical editors, as right now I'm basically doing my own editing as I go, one chapter at a time. Once I feel comfortable with where the book is, I'll be publishing on Amazon so people can get a hard copy.
[1] https://servercheck.in/blog/self-publishing-my-first-technic...
If you didn't have a checklist, could you remember all the steps to get the machine back to its current state? If not, you probably should have a checklist. Chef/Anisble/Puppet lets you make such a checklist, but that checklist is actually runnable so that if you needed to bring up a new box it's just one command line away.
So, are you missing out? Maybe. It depends on how complicated your setup is. My experience is that over time, the complexity tends to go up. It seems like extra work up front to make all your configuration changes in some weird extra layer, but as your changes grow and grow it becomes nicer and nicer to have them all listed and documented.
An analogy: it's kind of like programming with no source control. You can do it, especially if you're not collaborating, but as time goes on it becomes more and more cumbersome to work without it.
So his example is OK in fact (a variation on the SYN-cookie), I was wrong.
Generally the whole point of a configuration file is to allow administrative users to the change the behavior of the application. Treating the configuration file as an "input" is a relatively trivial difference and doesn't really address most of the problems admins face.
"OK, just set LD_LIBRARY_PATH."
It's in private beta right now, but if you're curious, let me know and I'll hook you up.
For instance, I have a bunch of disposable VMs, and I don't want them to download the same gigabytes every time I run init-deploy-test.
apt-get install apt-cacher-ng
And despite the name, newer versions work with yum too (although I found I had to disable the fastest mirror yum plugin to get reliable caching).
I had a look at Puppet/Chef.......wow those really look complicated for something that should really be simple.
Not to mention that a lot of stuff you would do with Chef/Puppet, on Windows are done directly from the "package" itself (create a default user with limited rights to run under, set up IIS for web apps etc etc). And when something fails....isn't it better to debug your code rather than a black box ?
But the technology is called Desired State Configuration which is PowerShell based.
This is nasty.
I like it very much, you can guess why...
I would suggest another blog post about monadic (you know, type checked, guaranteed safe) packages (uniques sets of pathnames), statically linked, each file in a unique cryptohashed read-only mounted directory, sorry, volume. Under unique Docker instance, of course, with its own monolithic kernel, cryptohashed and read only.
Oh, Docker is a user space crap? No problem, we could run multiple Xens with unique ids.
About packaging:
> Notice that output files of one software package are also inputs (as the filesystem) to the other software packages.
Yes, and no. If you're serious about producing the package to distribute it, rather than to install locally, you're going to build it in an isolated environment that has a completely controlled set of dependencies available. For deb packaging that's provided by pbuilder. This is comparable to what happens in nixos at build stage.
About deployment:
> Our tool needs to: - figure out what kind of package manager we're using on our Linux distribution
That's not a dynamic thing. This is solved using plugins and the only operations you have are install (including chosen version/upgrade) and uninstall. There are a lot of system-specific options you probably care about, but they come up because you're most likely running a general purpose system. This means desktop user or developer will want 'apt-get install mysql' to do everything and give you a running server at the end, while people doing automated server deployment will want it to install the binaries and stay far away from the config files or restarting anything.
> The problematic part of such system is the fact our tool had to connect to the machine and examine all of the edge cases that machine state could be in
If you're running at scale, no, you're not connecting and not figuring stuff out. You'll want to run everything locally. There should be no edge cases either. If something doesn't work, go back to your dev environment and figure out what needs to happen. (or to put it another way, why are there edge cases if all servers are set up from the same description to begin with?)
> No dependency hell. Packages stored at unique $PREFIX means two packages can depend on two different openssl versions without any problem. It's just about the dependency graph now.
And managing security patches is a bit harder suddenly. You have multiple applications possibly using multiple versions of the same library.
> Source and Binary best of two worlds.
Only if you can turn the source installation off for the whole system. I don't want package installation to trigger a compiler run on a live system. Ever.
> Rollbacks. No state means we can travel through time. Execute --rollback or choose a recent configuration set in GRUB
This is not true for any non-trivial system. Can you rollback your database? Maybe, but it may not be able to read the files anymore. Can you roll back your language runtime? Maybe, depends if your code is using new features. Can you rollback a library? Depends what has been built on top of it already. Rolling back the binaries is what existing packages already provide and it's the easiest part of the rollback.
So after all this thing about separating yourself from the OS assumptions and various states, etc. we end up with "mkdir -p ${cfg.stateDir}/logs" - why is that embedded into the package at all? What if I'm running on a R/O system and log over network?
The article does raise some great points, but then describes a system that either still has the same problems or trades them for something equally bad. Also we still need something to tell all the new servers what the "cfg.stateDir" and other inputs should be. And it will need to know that it's running on nixOS. There are tools that can do that: chef, puppet, salt, ansible, ...