DevOps from Scratch, Part 1: Vagrant and Ansible
kevinlondon.com
kevinlondon.com
So let me rant about it for a bit:
- Whatever that language is, it's not something invented on earth. I've given up on learning when to use quotes or how it wants its indentations and just copy&paste&destroy everything&restart when it doesn't work after 15 minutes.
- There are a few thousands tests in their test suite, which makes it quite a feat to introduce a bug that breaks what amounts to "passing variables to a subroutine": https://github.com/ansible/ansible/issues/17356. (2 hours today I won't get back). I seriously don't know any other software where I spend most of my time reading through github issues.
- It's a tool to manage hosts. Which would make me think that "reboot" shouldn't be a dirty word. But there's actually no function to do (a) ```/bin/sh reboot```, (b) wait for it to come back. First request I found is four years old: https://github.com/ansible/ansible/issues/513. The current request has a nice summary of the cottage industry of "blog articles that explain how to reboot & wait": https://github.com/ansible/ansible/issues/16186
- Projects are structured in a quite "sparse" folder structure. A small project usually has 25 files in 20 folders and 18 of them are "main.yml" which is probably a conspiracy to annoy me. The main feature an editor plugin for ansible needs is not syntax highlighting but "use second-to-last folder name instead of filename on tabs".
-
Ha! I've found myself doing that with Ansible way more than anything else we use. We've been using it since early 1.x and it's been rather buggy sometimes, but in general it's really great and I'm glad we went with it over the alternatives. New features have been coming fast that have made it worth the occasional bug.
Also, if you used the same step in two different puppet manifests, it would totally freak out and refuse to run, even if the invocations were identical. I got the feeling that Puppet is for giant enterprise environments with automated agents everywhere, and Ansible is better suited for admin-driven runs.
Chef has caught up, it still needs the agent but chef-zero [1] makes solo deploying much easier.
What made you think it is? It's not. It's somebody's application deployment script that grew too big and started being marketed as a universal tool. For managing servers its very architecture is too brittle.
CFEngine and Puppet are universal tools for managing hosts, though they don't work in (sometimes needed) synchronous manner, like well-prepared RPC daemon would.
- YAML is not a proper programming language
- It's not easy to create reusable "functions". You can include tasks but it can be painful to manage variables, you can depend on roles but everything ends up in separate folders and files.
In retrospect I would prefer to just use Fabric, or use Ansible modules through Python.
If you want to configure your hosts using a programming language, just use shell scripts.
--
edit: the acronym has been changed to 'YAML Ain't Markup Language', to separate it from documentation languages
Enough with the flapping and discrete solutions already!!!
I had always assumed these tools felt over-engineered to me because I don't work with very large deployments. Now I am certain they ARE over-engineered...
I run close to 72,000 servers and I won't deploy any code on them except shell code.
Enough with the flapping languages already!
Vagrant allows you to create reproducible development environments using shell scripts (by default), so it should be right up your ally. Docker is used to containerize applications. runC is the container runtime used by Docker. Kubernetes orchestrates containerized applications across a fleet of CoreOS (a minimal "Docker host" Linux) machines. OpenStack orchestrates your data center. They are discrete solutions because they do different things. Each piece of the puzzle does one thing and (hopefully) does it well. That philosophy should sound familiar to you!
Back to "writing shell scripts and encapsulating them in OS packaging": have you tried a configuration management tool? I used to be old school like you; specifically, maintaining a fleet of AIX instances using a healthy mix of Perl, ksh, and RPM. I too was resistant to the tide of new, quickly-evolving technologies. I started using Ansible about a year ago and it's changed my life. Don't be so quick to pooh pooh the idea that there just might be a better way to do things.
Flapping, you know, when a person who cannot swim is drowning and just flailing and flapping their body, most notably arms, all over the place in an attempt to stay above water so that they do not drown.
but the technologies in your list do not all address the same problem space.
I don't know about you, but for me to get a fully configured, ready-to-serve system, I need both provisioning and configuration management. With that being the case, provisioning and configuration management fall under the same problem domain.
Vagrant allows you to create reproducible development environments using shell scripts (by default), so it should be right up your ally. Docker is used to containerize applications. runC is the container runtime used by Docker. Kubernetes orchestrates containerized applications across a fleet of CoreOS (a minimal "Docker host" Linux) machines. OpenStack orchestrates your data center. They are discrete solutions because they do different things. Each piece of the puzzle does one thing and (hopefully) does it well. That philosophy should sound familiar to you!
I'm familiar with all of those technologies, and yes it does indeed sound familiar to me!
But, what is unclear to me is, why I would use a discrete provisioning solution like Kubernetes instead of using, say, imgadm(1M) and vmadm(1M) through parallel SSH?
Why would I use a discrete, proprietary solution like Vagrant when I wrote my own, simple variable expansion function which uses eval(1), and that's all that's really needed for configuration management? I'm far, far better off with a simple shell function then using Salt or Ansible. Simplicity, the core of the UNIX principle. eval(1) is much simpler and easier to understand than an entire provisioning and/or configuration management engine! Or do you disagree?
Other than templated configuration excerpts and self-assembly, both easily implemented with a few simple lines of shell code in a small shell library which OS packages can source, what more do I need to configure systems? Why make things unnecessarily complex? In fact, that exact same approach is what Chef uses, conceptually, when one specifies the configuration management policy, except that with my solution, the system administrator need not learn a proprietary Ruby dialect and need not employ a client-server architecture; they can just dive in and leverage their existing shell programming knowledge, with a zero learning curve. And because it's all just shell and AWK, the system administrator has infinite possibilities to configure the systems and do anything they need to do, without being limited by the framework of whichever technology they are using (like YAML or Chef recipes).
So why bother??? Why flap?
Why would I need runC and Docker to provision systems, if I can just install OS packages which configure the system, and then make a ZFS image?
Or, in terms of Linux: why do I need say Docker and Kubernetes, if I have everything in OS packages, and I just use KickStart to PXE boot, automatically install and configure the target node with OS packages?
have you tried a configuration management tool?
You bet your ass I did! But that's not the point, the point is, why should I learn a discrete, proprietary systems, when I can just write a simple shell library which leverages the eval(1) command to do variable expansion, and BAM!!! I instantenously have a template expansion engine so all my configuration, even code, can be parametrized using the same varaible evaluation and expansion technology which the shell uses, and which the configuration packages can call?
With that:
- I don't have to use proprietary, discrete systems;
- I can leverage technology already built into the operating system;
- I don't need a client-server architecture like Chef or Puppet, ever (ssh-sshd is just fine, thank you!)
Why would anyone waste their time?
You point to imgadm and vmadm as superior solutions, but these appear to be "proprietary" (to use your own word) to SmartOS. Why not add SmartOS to your list of unnecessarily technologies and apply the same critical thinking? Triton appears to occupy the same space as OpenStack (Triton DataCenter), Kubernetes (Triton ContainerPilot), CoreOS (Triton SmartOS), and possibly other components, and it appears to use the same kernel-level containerization facilities as Docker and runC. You're using a competitor's vertically-integrated solution, so no, you don't need another stack. If you weren't using Triton you would be researching these alternatives. Your argument is akin to saying you don't need a car because you have a Buick, people who drive cars are using an unnecessary, proprietary mode of transport, and they should all just drive Buicks instead.
That brings us back to configuration management tools. You keep calling them proprietary, but the cores of all these products are open source! And while I agree with you that the client/server model used by Puppet, Chef, and SaltStack is unnecessary—although it does have benefits at scale—it's worth noting that Chef Zero and Ansible just use ssh-sshd. These tools are making the same system calls as your shell scripts, they're just doing them from Ruby or Python instead. So just to recap:
> I don't have to use proprietary, discrete systems;
They're not proprietary, they're open source, and you only need one of the four.
> I can leverage technology already built into the operating system;
They do: Ruby or Python and system calls.
> I don't need a client-server architecture like Chef or Puppet, ever (ssh-sshd is just fine, thank you!)
Okay, so use Chef Zero or Ansible.
Finally, your other main argument, against abstraction, does not hold water. Everything we do is an abstraction. Shell scripts are an abstraction. Why aren't you writing your configuration management in C? (Wait, why are you writing it in C instead of assembler?) On one hand, you have a portable, open source tool with a domain-specific language. On the other hand, you have each company creating their own proprietary (this is the correct use of "proprietary," by the way) tooling, and an engineer has to learn it from scratch every time they change jobs. You may think any engineer can walk in off the street and instantly grok your homegrown shell- and AWK-based big ball of mud, but you are probably wrong. If your solution is as good as you think it is—if it's portable, extensible, and easy to maintain—you should release it and we can add it to the list of available CM tools. If it's not, why are you reinventing the wheel?
Really? What is it then? Let's see...
Kubernetes (commonly referred to as "k8s") is an open source container cluster manager originally designed by Google and donated to the Cloud Native Computing Foundation. It aims to provide a "platform for automating deployment, scaling, and operations of application containers across clusters of hosts"[1]
Automating deployment? I guess deployment has nothing to do with provisioning a resource... really, if you are going to argue for these tools, you should understand the concepts which they are trying to solve. Or are you one of the developers of one or some of these tools, and this is just rubbing you the wrong way, because it means that your tool isn't required? If so, just state that upfront, so everybody knows where you stand, and we don't have to discuss this any further.
[1] http://kubernetes.io/docs/whatisk8s/
Moving right along...
your comparison of Docker and Kubernetes to OS packages, Kickstart, and PXE is outright laughable. You clearly do not understand the various purposes of these tools.
Okay, then you explain to me why I need Docker to do what I can do with PXE, KickStart, and OS packaging? This ought to be really good...
Why not add SmartOS to your list of unnecessarily technologies and apply the same critical thinking?
Because SmartOS uses zones, which give one fully isolated, true UNIX servers running at the speed of bare metal, and because it uses ZFS, fault management architecture (fmadm(1M), svcs(1), and svcadm(1M)), performance and post mortem analysis tools, and because it has everything required for massive datacenter and cloud deployments, built right into the operating system from the beginning, and not as an afterthought, knee-jerk "oh shit, we have to scale Linux to massive deployments now" reaction. By professional system and kernel engineers with formal education in those problem domains. imgadm(1M) and vmadm(1M) in SmartOS are the functional equivalents of JumpStart (or KickStart, in redhat's jargon, and AutoYaST in SLES one). That's why.
They're not proprietary, they're open source, and you only need one of the four.
Proprietary means that someone owns it, or that it is specific to something or someone, not that it is closed source.
A product can be both proprietary and open source, and freeware all at the same time if the proprietor is providing the source code and giving the product away for free.
They do: Ruby or Python and system calls.
Neither of those programming languages are part of the operating system consolidation on a proper UNIX, and on Linux, they are provided in various versions by various people who are volunteers. And anything which is not part of the OS/Networking consolidation on a real UNIX is a third party and unbundled application. That's the only distinction we have ever needed, and the only distinction we have ever made.
This mistake was already made once on UNIX, when Perl was hip just like Ruby and Python are now: we are still coping with the fallout of ripping out, and replacing all the Perl garbage which has since been integrated into the OS/Networking consolidation. You haven't yet been busted by that on Linux, but you will be, because Linux keeps repeating the same mistakes which UNIX made. Except we learned from that, and on Linux it just causes more flapping.
Between Ruby, Python and shell and AWK, I will always pick shell and AWK (and not even bash, but traditional Bourne shell), because they are stable, the interfaces won't change, the syntax will stay the same forever, they are well understood, easy to learn, and they are small and fast, very fast.
I have regularly beaten Python with AWK both in code density and speed (latest: 487 lines of Python code in 183 lines of AWK, and no kernel hooks required like with the Python code). So if you're implying that Ruby and Python are ideal for massive deployments and configuration management just because they're hip right now, that's insane, and I'm not buying it.
Okay, so use Chef Zero or Ansible.
Why? What does the Ruby dialect of Chef offer me, that I couldn't do in %pre, %post, %preun and %postun in a RPM package? Policy? Policy is best solved with a throughly thought through change management process. Capability Maturity Model and all that, you know?
Why aren't you writing your configuration management in C?
And just what do you think svcadm(1M) is written in?
(Wait, why are you writing it in C instead of assembler?)
Who says I'm not? Use the right tool for the job. The problem here is, none of these Chefs, Puppets, Ansibles, Salts, Dockers or Kubernetes are the right tools for the job. Even Linux as the substrate for all of this is not the right tool for the job! But then again, judging by the other comments here, people are starting to realize that, and that's a good thing.
Also check out the code examples: https://github.com/geerlingguy/ansible-for-devops
I agree it being Flask is largely irrelevant.