The DevOps Phenomenon: An executive crash course
queue.acm.org
queue.acm.org
DevOps isn't a tool. It isn't a team, or a person. It isn't even a culture. It's a way, for lack of a better word. Like the way of the Tao. The DevOps that can be named is not the true DevOps. Everyone is doing DevOps, and no one is doing DevOps. That doesn't mean some aren't doing it better than others.
Some years ago, I attended the first DevOps Enterprise conference, and there was a phrase everyone was using... "We're not unicorns, we're horses". The point being, your company doesn't need to be a Netflix or a Google to do DevOps successfully. Elements of transformation can be adopted by anyone. Then again, as I said on Twitter at some point, "Some companies are unicorns, some are just horses, some are just donkeys, some are just stick horses, and some are just sticks". To go on another wild analogy, DevOps transformation is kind of like alcoholism. First, you have to admit you have a problem.
The next thing is, because DevOps is basically a Taoist problem... well, you can't buy the Tao. You can't hire the Tao. So companies that try to "buy the DevOps", thinking a tool will save them, or try to "hire the DevOps", thinking a consultant will save them, are doomed to failure. This isn't to say you shouldn't go out and try to hire experienced DevOps engineers, or that they can't help you, but the reality is much more broad than a single hire. Getting a good engineer is only as good as that engineer's freedom to act.
And tools do not DevOps. They only make the DevOps easier. You can DevOps even with antique tools. Back before some of you were born, I was doing distributed deployments with version control and A/B testing with shell scripts and a SQL database. It worked.
Wow, I can rant a lot about this. But what really matters? Commitment to becoming better. Actions, not words. Automating what can be automated. Recognizing that speed reduces rather than increases risk. Remembering that it's a process, not a goal. Being patient with slow beginnings. Measuring things. Remembering that not everything that is valuable can be easily measured.
> It isn't even a culture.
I tend to think it is a culture that recognizes and seeks after the tao. I've been part of cultures that think IaaS, CI/CD etc are varying levels of important. Some are zealots about CI/CD but then don't care at all about IaaS (they provision everything manually). I've also been part of cultures that won't do anything manually. So maybe it isn't a culture in itself, but it's definitely related to and enabled by a culture at the least.
> Getting a good engineer is only as good as that engineer's freedom to act.
That made me cheer in my brain. Absolutely true :clap:
If you're doing DevOps right, you'll have good culture and good tools, for sure.
The problem is the team building your product often doesn't want to do that work, even if they can...
True story, my title was Dev Ops Manager long before the word was coined because I knew a lot about both and kept bringing both groups to the table to solve problems so they made a position for me.
It would be years before I heard the term DevOps again.
* Do you need it now?
* Is there a simpler solution that gives most of what you want?
* IS there an iterative approach you can start with?
It doesn't all have to be done at once, it can definitely be ad-hoc and can grow as your application(s) grow. Don't start with k8s when dokku will work. Don't force everything to be ready before any features can be built.
I agree with this. Most times I've been interviewing for "devops" roles what the employer was looking for was, actually, a sysadmin to babysit the devleopers (in terms of being a "dedicate resource").
1. All talk and no support for automation. I want to be rich, handsome, and well-liked too but none of that happens without a lot of serious work if I’m just a pretty face with a horrible personality and refuse to admit that’s a big problem
2. Reactionary (vs responsive) culture - all the bad, none of the good of what allows agility. If there was vision and a real need to deploy more often, that would have been solved as a priority to help speed. Instead, all had been around for 5+ years cobbling together bad processes, tribal knowledge first, and burn-out of engineers related to the failing processes was the rule not exception.
3. Lack of understanding of how to grow beyond a certain engineering maturity level - ceremonies over results and design is common as a reaction. A lot of companies do “Agilefall” for example and watch quality and throughout drop like a rock. I’m not sure if there’s an equivalent portmanteau for “Devops tools and processes with none of the benefits of automation and all of the drawbacks of code”
Would be very interested in your feedback on an article I wrote recently dealing with this issue: https://calebfornari.com/2019/05/31/leading-a-devops-movemen...
https://www.hashicorp.com/tao-of-hashicorp
- Workflows, not technologies
- Simple, Modular, Composable
- Communicating Sequential Processes
- Immutability
- Versioning through Codification
- Automation through Codification
- Resilient Systems
- PragmatismFor Hashicorp to claim the tao like that, is the admission that they do not know it.
This in contrast to "sysadmin who can configure your Cisco and your Outlook".
The difference is crucial in data centers where you need automation for sysadmin tasks. (I.e., all of them.)
This also means, not just dev and ops skills. It's also testing, also security, also compliance. "pipeline driven" is what we are after. And this of course enables true continuous delivery.
"DevOps" is another Silo. Which is why we also have "DevSecOps" and "TesOps" etc...
I'm in the process of writing a new book about this new-old idea: https://leanpub.com/pipelinedriven/
Calling it "devops" has turned out to be a disservice. Teams are either made up of devs who don't know the sysadmin space and thus think the solution to every problem is "Write a new program in $LANGUAGE_OF_THE_MONTH", and balk when you tell them "Ok, but... did you consider this thing that's been around 25 years?", or it's made up of what most companies had for "ops" people -- button-clickers who can follow a checklist like "Open Control Panel, double-click 'Add and Remove Programs...'" but are beyond hopeless in front of command line.
In the wild, it's exceedingly rare to find people who can bridge both worlds reasonably. I appreciate that Google was able to formulate a functional SRE team, but... most companies aren't Google.
Virtually everything that could've gone wrong as devops hit mainstream has gone wrong. Docker is all wrong. Kubernetes is a massive barf-fest. The obsession with "cloud everything" is not only a grotesque waste of money, it's insecure from first principles, as the emergence of practical speculative execution attacks has clearly demonstrated.
crazypills.gif
I agree with this comment. Personally, I got my start doing sys admin work, and after doing it for years I became pretty decent at writing scripts in Python and Ruby, and I picked up some Java and JavaScript along the way.
In my experience, a lot of people who don't have a lot of experience tend to fall in one camp or the other: either they're operations people who can't code, or coders who don't follow best practices when it comes to operational work.
But here's the problem that I keep seeing:
Management, the ones writing the checks and doing the hiring, all they know about DevOps is what they've read in books and seen in seminars. Due to this, they have a bad habit of dismissing people who don't speak the DevOps "language."
For instance, I did a job interview a few months back, and everything in the job description was in my wheelhouse. I was a great fit. But the hiring manager kept trying to coerce me into talking about my 'vision for DevOps.' Clearly, he had read a book or attended a seminar, and he wanted me to have some type of religious experience with him.
But that's not my thing. I am too busy actually doing the work to read a 200 page book about mission statements.
1. Separation of duties
2. Process (and well, documentation, but let's say that's a part of process.)
Separation of duties boils down into needing separate people to develop, review, and deploy a piece of code. Okay, let's look at this from compliance's PoV.
Compliance team just wants an easy way to show and enforce "yes, a separate individual performed each of these duties." Preferably in a way that non-technical people can understand. Easiest way to do this? RBAC. If you are a developer, you simply don't have permission to login to production. If you're ops, you don't have permission to contribute. Clean, easy enforcement, clear lines drawn in the sand, easy reporting to auditors. It also has a chilling effect on the devops mindset. Which seems to be the issue you are raising.
How else can you satisfy your compliance team, follow regulations, and still practice devops? Well, instead of enforcing at the role level, enforce at the individual level.
A lot of VCS software offer the ability to disallow branch merges without a secondary approver. This can be your separation between development and review. Then for implementation in production, you can automate this piece. Think of a git commit, enforced code review, and then k8s automatically deploying your change from the merge to master. This allows your dev and ops team to work on tasks together.
We use Jira extensively (other tools work too), and certain important systems are tagged as SOX for a specific year. When an auditor asks "show me all changes to financially important systems", it's easy to pull up that year's changes, and then show off the 2 people signing off. We also have auto deploy (not k8s... yet) that is easy to show "hey look, this code was definitely not tampered with on its way to production."
Getting your compliance team on board is difficult. It requires team work because if you are subject to SOX, you are possibly subject to other regulations that touch your processes. Like PCI-DSS. Or SOC. If your dev or ops teams can pitch a process change in a way that's easy to digest, document, and audit, then they will likely be on board.
You need to explain "peer review" or "merge requests" functionality from github, gitlab, what have you.
Basically, you have:
- An Author - Tests - Reviewer(s) - Audit - Robots - Protective Monitoring
You should be able to explain that your average developer cannot bypass this, offering you greater assurance than the paper driven process before, and then point holes in the current process such as, people deploying manually can change the deployment after it's signed off.
You can also mention that working in this way allows you to alert and triage on all assumed roles that aren't a robot account, meaning nothing that hasn't been reviewed should happen on your production cloud account, for example.
Depending on the InfoSec team you might need to present your own risks, such as the CD environment, robot account setup, how they assume roles, the administrators, etc.
I've been explaining this stuff to old school InfoSec teams for around 5 years now and have got pretty good at it. As long as they're actually interested in the risks, rather than just exerting control, you should be able to explain it to them.
So whole devops is just working together not just throwing stuff over the fence. It works great with QA, security, ops. It is all about people willing to work with each other. No amount of tools will substitute that.
Let's use real words that tell the listener or reader what we're talking about.
More correctly, it's something of an unattainable goal. Can we get rid of on-call? Probably not. Can we drastically reduce the frequency and scope of on-call actions? Absolutely! Can we get to where every single time the on-call person has to take action, there's a post-mortem to understand what went wrong, and fix the automation and monitoring so it doesn't happen again?
Have you ever gotten paged for a transient error that went away by the time you checked it? Or paged for the same error 5 weeks in a row? These, and really most pages, are fixed by DevOps.
In a DevOps system, a feedback loop is used to address all alerts as bugs to be immediately and permanently fixed. After a while, if something breaks, it's because someone just changed something, so it's happening during working hours. Alerts pop up in slack and are acknowledged before a page is sent out. So nobody is getting called.
If your infrastructure is ephemeral and managed as code, you use CI/CD to deploy all changes, and you aren't resource-constrained (at this point, cloud-native infra is only constrained by budget) you shouldn't have stuff crashing randomly at 3am, so there should be very few pages.
Isn't "budget" just another type of resource?
Of course Ops is still needed but what they do has probably changed a lot in this case. And their responsibilities and how they work with Dev must changed.
1) let's fire all the DBAs
2) let's fire half the operations team, and outsource the rest
3) let's have the devs do the operational work
What could go wrong?!
Anyone have the German translation?
That's the Americanized acronym of the original German word.
It's a culture and a way of doing things that goes far far beyond just automating routine tasks. The automation can save time and help with the transformation but it's not the main ingredient so to speak.
As a consultant in the space I have to explain this to clients sometimes and they don't always want to hear it. I believe this is because many people don't know how to influence the culture of their company or team and so it seems far easier to just implement some tool than it is to actually change the culture in a meaningful way.
I actually wrote an article on this subject recently that some may find useful: https://calebfornari.com/2019/05/31/leading-a-devops-movemen...
There also seem to be some misnomers here, like the lack of a definition for DevOps (not true), or that continuous integration is "continuously testing software" (not really), to confusing the definitions of continuous delivery vs continuous deployment, to assuming 'measurement' only refers to components of systems and not the actual development of the systems, to the idea that you need automation to do DevOps (you don't).
I mean, what and the how and the why?
It's supposed to be a team decision. If the team decides to use packages, then at least someone has to be able to implement that into the workflow, release and deploy process. If the team decides to not do RPMs or DEBs, then they have to solve build/deploy/release some other way.
For example since Docker I haven't even touched spec files or fpm ( https://github.com/jordansissel/fpm ).
The big company referred here is certainly Amazon - this is public information that we shared a few years ago (when I worked there).
The way you do that isn't by "automating all the things" and "continuously all the things" and "measuring all the things". You do it by getting everybody to put their concerns out of their heads, off the whiteboards, and into code. Code is the only thing that can guarantee that everybody's concerns have been accounted for, including the concerns you didn't know about because nobody can keep 200 people and all of their concerns straight in their heads, while allowing for a release cycle short enough to stay competitive.
That's it! Everything else is execution. Scripts (automation) handle enforcement. Pipelines make sure enforcement is consistent and up-to-date. Metrics make sure your scripts are doing what you expect them to do. Dashboards help convince people that their concerns are actually being taken seriously and the engineering managers aren't just spouting voodoo. Speed is just a side-effect of people being focused on their work and not having to waste time arguing about petty details anymore.
Shell scripting has been around for more than forty years. Modern tools made it pretty, but ultimately you need leadership to make it relevant on an organizational scale.
(Not actually me, but that’s how people at big companies think)
The target audience here is a large enterprise that has value cost and stability for years and has separate Dev and Ops (often not in the same country) and not only follows all those standard processes but has also added a lot of their own written Word documents to the pile. There are check lists, approvals, delivery managers, product managers and all kinds of other managers involved in every release.
How do they even start with your advice?
"Most importantly, for DevOps to truly deliver value, it must include more than just tooling and automation—so simply purchasing and installing a solution isn't sufficient. As outlined here, DevOps includes culture, process, and technology."