Reflections on 10k Hours of DevOps
matt-rickard.com
matt-rickard.com
> The value of a CI/CD Pipeline is inversely proportional to how long the pipeline takes to run.
Priority #1 is : test your product properly. This means "if it is supposed to do something, then you should have a test to check that it does that thing".
Sometimes it just takes some time to put together and run such tests.
In my experience some people (usually my perception is they are younger, more impatient people) who have read somewhere that tests should run really quickly, will argue that Priority #1 above doesn't matter, in the interest of their patience being assuaged.
Long ago we had product builds that took hours and tests that often ran overnight, and sometimes over days. And we shipped sh*t that worked!
Slow CI/CD and slow releases are a big problem because they change your entire workflow. Critical bugfix that needs to be shipped? If your CI/CD takes minutes, not a problem: fix it, test it, ship it and have a 5 whys. If it takes days, well, you either have a side-channel process where all your senior engineers have to gather around and speculate about the impact of the change, then you have them sign off, and get the VP to sign off to approve the hotfix, then hope for the best. Or you just wait for days while your customers complain on Twitter.
Tests should, ideally, run really quickly. Not at the expense of correctness or completeness, of course, but in most cases I've seen, slow tests happen because nobody bothers optimizing test code, not because they're inherently slow.
Sometimes CI/CD is slow for even dumber reasons. At one place I was at (1000+ SV-funded hot startup in the mid-10s), tests got relatively slow enough (~40 min) that VPs got involved and a "test speed task force" with a mandate of reducing CI/CD time to less than 20 minutes. This was announced at a company all-hands with a ton of fanfare. In the first week, one of the engineers did some quick glancing at the logs and noticed that the very old build scripts were still hardcoded to deploy to two servers which no longer existed... and the DNS resolution for those hosts was taking 10 minutes to time out. Needless to say, the task force accomplished its goal, though in a bit of a hollow victory.
Instead of fretting about the unknown sides effects of the code you are writing, you can be confident the unit tests will flag it if something goes wrong.
Arguably any "hotfix" (other than 3rd party zero day security patches) required is a testing escape, and ideally those should be rare. And usually there's a subset of tests that can give you 95% to 99% confidence the hotfix will work.
I agree with all this. I'm not in any way saying you should do less testing because it makes CI/CD faster. I'm saying that there's a 95%[0] chance your tests are taking hours or days because they are doing something suboptimal that can be improved. Generally, it's a combination of
1. lack of overall test ownership and structure -- for example, every team maintaining their own UI integration tests and repeating an expensive login process unnecessarily, or repeatedly writing new test data to a database rather than working with an existing dataset where possible, or (worst of all) relying on shared resources like databases and servers rather than having ephemeral environments per-branch, resulting in queueing of tests;
2. people who write tests not giving a damn, and this not-giving-a-damnness accumulating over time. If you sneak in a unit test that takes 300ms when it should have taken 10ms into the build, it's likely nobody will notice. If you have a suite of 1000 tests and everyone does that, your test suite now takes 5 minutes instead of 10 seconds;
3. management and engineering leadership also not giving a damn unless it becomes really untenable. Usually devs just grumble about a slow build and nobody does anything, especially in larger corps where there's less feeling of ownership. It's very hard to have a business justification as to why you would pay people to investigate speeding up tests. By the time the test suite is taking 4+ hours, it feels "easier" to just add processes to mitigate this, like making the build server run nightly, rather than address the fundamental issue of tests being slow.
[0] There are likely specific domains where this is untrue, but rarely in the generic "CRUD software with background data processing & storage" case.
They kind of are though, since they don't acknowledge the trade offs inherent in reducing the duration of the integration tests.
>In most cases I've seen, slow tests happen because nobody bothers optimizing test code
IME they typically they come about because realism comes with a price tag. An actual UI is more expensive to load than a mock, an actual webserver is slower than a fake web server, a postgres database spins up slower than an SQLite, a real API call takes longer than a mocked one, etc.
Sure, but spinning up a postgres database or running a Selenium test should change your test times from being executed in seconds to minutes, not seconds to hours or days.
> They kind of are though, since they don't acknowledge the trade offs inherent in reducing the duration of the integration tests.
As I say in another comment in this thread, there is very, very, very rarely a legitimate reason for integration tests to need multiple hours to run. Even in the worst case, you can just parallelize the tests on different hardware; if you can't parallelize, that is a problem with your tests and not a real excuse.
I worked on a big old ball of mud once where business logic was smeared across 6 or 7 different microservices. The ONLY way to gain confidence that the app did the right thing was to test it end to end which, of course, took ages, and meant hours long CI pipeline (longer if not parallelized).
My coworker tried to replace some of those tests with "faster" unit tests around bits of those microservices but all they did was "lock down" the horrible architecture such that it became impossible to refactor and consolidate the business logic without breaking a bunch of unit tests.
In your story the team was given time and resources to actually do it, at last. If it wasn't, the problem might have continued for months or years still unnoticed. It's not that hollow a victory.
And I miss those days as well. I think the author's point was not a wholesale dismissal of the value of extensive testing, as such.
But rather - whole point of a CI/CD pipeline is that you're supposed to get the feedback ... reasonably fast, right? That's what the "C" is supposed to stand for, pretty much (along with adjacent matters of being explicitly tied to your commit/merge process). So they aren't saying there's no value of all that testing -- but rather, if it takes hours and hours (when it was supposed to be nearly synchronous with your commit process) -- its values does attenuate.
Or if you're spending resources making it faster - rather than figuring out what's wrong with your code or models in the first place - that's another tax to pay.
Great so your build finishes in 3 minutes but you have no idea if the product works. While my build takes over 30 minutes but I fully confident the product works. Which one would you rather have.
> Immutable infrastructure removes a whole class of bugs.
Story time. Back around 2010 I was tasked to figure out why a server application written in Java was running out of memory. It happened often enough that the application had to be restarted daily. I was not the one who maintained the application, and I wasn't the one maintaining the servers it ran on either. I was just someone good at solving these kinds of problems.
The odd thing was that the problem only existed on one server in production, even though the other servers should be running the exact same thing. I ran load tests on a staging server, also running the same code, and was unable to reproduce the problem.
With VisualVM and some heap dumps handy, I narrowed down the problem to database-related objects, specifically objects from the Oracle JDBC driver were taking the most memory. I tried checking what was different between the servers that would cause one to have an issue and not the others. Was one getting more load than the others? Were the processors different? Memory? Disk speed?
To make a long story short, I spent entirely too long debugging the problem and eventually found out that one of the servers had a different version of the Oracle JDBC driver than the others. And it seemed that particular version of the driver had a memory leak bug. It took me so long because I assumed that nobody would have been dumb enough have different library files on different computers.
I could kill whoever put that file on one server and not the others! It must have been done manually. I never did find out who did it or why.
Immutable infrastructure would have prevented this problem, but this was back in 2010 where we ran our own servers and immutable infrastructure wasn't a possibility.
If we had sufficiently reproducible deployments then we could have just redeployed the application and the problem would have gone away. Alas, the Oracle library wasn't part of the application code, but instead was part of library code that was installed on the computer and never changed (or so I thought).
It's a godsend to be able to deploy containers that not only contain your application code, but also your server software and their dependent libraries. Reproducibility is so much better that way.
Worked at a consulting firm doing enhancements to a legacy hotel reservation system, and we had just finished our latest feature, which was integration with another, even more legacy reservation system (Visual FoxPro, yum). This was old-school consulting style, so we developed internally with our test systems and no access to the real systems, then shipped over an installer .exe and let the client, who ran ops, handle the installation and deployment into IIS. We built it, they deployed with no issue, test reservation worked, easy.
A month later we got a report that about 20% of online reservations using this system were failing. Everything was working on our test systems, and it was working 80% of the time on their systems, so we assumed that it was a weird race condition with our booking service, which ran mostly asynchronously. We added a ton of additional logging to that service to try to figure out what was going on, sent them an updated version to deploy, and waited a bit.
A week or so later I went over to the client's site to look at their production systems with their ops person. They had 5 servers servicing requests. We looked at servers 1-4, everything seemed fine. Open up server 5... wait, where's the log file? Process was running, IIS was responding, but nothing was coming out. Check the permissions on the folder, and lo and behold: the fifth server had a typo in the permission ACL entry for a folder needed for the legacy integration, which by fortunate coincidence was also where we had set the log files to write. Fixed the typo, the problem instantly went away completely.
And being a consultant/client relationship, of course there was a lengthy legal dispute afterwards about whose fault this bug was, as that determined if it was covered under the original statement of work or we could charge more for it... one of the experiences that made me realize I needed to be working for a product company, not a consultancy.
I feel managing dpendency jars separately was weird even in 2010, I always do a shaded jar or at least an maven assembly folder as a deployable to make sure the server runs the same combination as on my machine.
For DevOps in general I have learned to KISS, always know how your tools work, and focus on observability and low hanging automation. If you can't observe or understand how your systems works, you're screwed. You need metrics, logs, statistics, in one place that you can easily build queries with. You should be able to see everything at the fleet level down to the innards of each machine with no troubles.
This is too broad a backlash against it. Where YAML is used in problem domains where the configuration complexity is strictly limited and "etched in stone" it works very well. Nobody tends to champion YAML in this use case because it becomes somewhat invisible. It Just Works and nobody complains, but nobody raves either.
In domains where you have to start templating YAML, it fails horribly of course. This is YAML being used in a domain where it either shouldn't ever have been used or where APIs of some kind should have been provided alongside to accommodate dynamic use cases.
What "works" as configuration is often very hard to know up front, but the above attitude is throwing the baby out with the bathwater rather than seeking the correct balance.
Turing complete code is inherently more difficult to parse, understand and modify clearly than a config file and much more susceptible to technical debt.
>Your Integration Tests are Too Long
This is also a wrong attitude, I think. Faster feedback loops are great, but an obsession with speed typically leads to a lot of very quick tests that test very unrealistically - often missing entire classes of bug.
I've worked on a bunch of projects which had test suites that completed in under 10 seconds and never caught any bugs. Meanwhile, the team would end up leaning on manual QA to replace "slower" integration tests and the feedback loop on that made 5 hour CI runs look entirely worthwhile.
I've always thought dev ops was where one is both a dev and ops, but formally dev, and then also responsible for operations.
Apparently that was not the case, there is dev, and then there are devOps. Apparently the ops part is more important for devOps, and dev is actually geared towards the ops end and not actually developing features.
I was confused because I design and develop features, I build ops tools, I build and deploy CI/CD pipelines and I go oncall for bugs that happen. I thought I was devops, but apparently this is not how devops is defined in the broader industry. And apparently everyone else have much stricter role separation.
1. DevOps: A dev who can do ops and will do ops (because their employer won't hire actual operations experts)
2. DevOps: Using more development concepts and tools in operations (i.e., an operator who can do dev, but often in more of a support/deployment than product development capacity)
3. DevOps (original): A way of working that promotes collaboration, rather than separation, between development and operations (among other things).
This one seems more and more common, even though CI/CD has no inherent relationship with development or deployment in any public cloud.
- The value of a CI/CD pipeline is the value of its output. Doesn't matter if it took 5 seconds or 5 days. I have worked on pipelines that were shitty and long but delivered enormous value, and quick ones that were noisy and useless.
- The "code is better than yaml" post misunderstands what YAML is, and then says you should use a general purpose programming language for configuration. But if you're programming, you're not configuring, you're programming. My point is, both code and YAML are completely wrong solutions to the problem. Look at configuration file formats for 20+ year old programs as an example of good configuration. (Hint: none of them are either YAML or code)
- "Release early, Release often" only works for certain products/services. It's not a good idea to deploy 10x a day to an airplane in flight. (No, contrarian who's about to reply, it's not a good idea, just shut up.)
- Declarative configuration isn't a thing. It's just configuration. Non-declarative configuration isn't a thing either, that's just instructions. Declarative doesn't mean anything. It's a red herring people repeat because they heard someone describe it and it sounded smart.
- Do not 'set -o pipefail' by default. It will only waste your time when your scripts start failing because some command in the middle of a pipe returned non-zero while still outputting the result you wanted, and you spend 3 hours adding debugging to figure out where the error was coming from and write some extra error handling code. Just check if the result of the pipe looks accurate. Same result, fewer failures.
- If you have to do a simple task more than 3 times, check if (T*N)>A, or if it's a value chain bottleneck. If neither is true, don't automate yet.
I take configuration as custom values for an environment, read at start time by the executable code, that are easily readable for humans (a non-programmer ops person could change) and easy to produce/read/manipulate.
From that point of view, coding is different as you say, while it gives you more power it's just not very readable.
All the config files I've worked on for 20+ years are a way of expressing KEY=value, with some nesting allowed to not repeat yourself (.ini, Yaml, json, even XML basically) so in my mind they are all basically equivalent.
These files support a wide variety of functionality beyond what a data-serialization format like YAML support. The functionality is designed for humans to input data germain to the configuration of the application in an easy way. That is to say, it both makes the admin's job easier, and tells the program what it wants to know.
Data serialization formats (JSON, YAML) are particularly poorly suited for configuration. JSON doesn't even support comments. YAML does, but then forces you to correctly format the document in a non-human-intuitive way, making the admin's job harder, and not necessarily telling the program what it wants to know. It has been well established how many problems humans encounter editing YAML files, to the point that people just take out features to make it less problematic, though of course some problems remain. They could eliminate these problems by using an established robust configuration format, or even making up their own, but modern programmers have a poor education in software design and avoid anything that isn't trendy or popular.
.INI is a decent enough format, with the caveat that it was never a standard, so parsing is always application-specific. At that point you might as well just make your own configuration format that serves the admin and programmer better than trying to force complex configuration into simple KEY=VALUE pairs.
XML is sometimes actually useful, but time has shown that it is not human friendly, and often not even program friendly.
Exactly how do you "check if the result of the pipe looks accurate" without having to get a human to check the log file?
The whole point of automation is to get rid of human interaction even if it requires some upfront work to implement error handling where appropriate...
Most well designed command-line programs output to STDERR if they have an error, and STDOUT if they don't. So the simplest possible method is just to see if STDOUT was 0-length or not. If it's not 0-length, then assume the command completed successfully. grep can help validate the output. Depending on the command, it may resist partial failures; for example, many commands that output a JSON blob will wait until they have all their data before outputting the blob.
If the script you're writing depends on always working 100%, then you can do a lot more work to check for errors. First you would enable set +e, because if you exit on error with pipefail, you can't even check what the error was from PIPE_STATUS. Instead you can inspect every pipe result and tell your user what part failed, and decide if you should continue, print only potentially partial results, retry the operation, or fail immediately.
Most programmers don't handle error conditions properly and just immediately fail on any error, which itself causes errors which needs a human to inspect a log. Worst case, the shell script without pipefail just keeps working, when with pipefail it would have failed a lot.
I'm torn, I like Makefiles, but I've seen a lot of unreadable Makefiles with a lot of interpolation and ugly escaping, including cases where what should have been a proper shellscript (his point 25) crammed in a Makefile. I think Makefiles should be dead simple to see what a target does
Overall I agree, but IMO people ought to drop the 10k hours Gladwell term, to be frank it's bullshit.
C++ : C :: Just : Make
More sensible defaults, fewer footguns, just as expressive (pardon the pun), but still nearly the same syntax and paradigms so you don't have to learn a completely new scripting language.
The biggest downside, however, is that it's not installed by default on many systems and it's not in the official repos of some major distros. If that's a dealbreaker, stick with Make.
In my case, if I need to scale to a more complicated problem, I would use Rakefile and Ruby (in replace of Makefile and Bash).
Not only that but it makes it easier to read the history. I haven't yet been convinced of the benefit of merging pull requests instead of rebasing them. The history ends up so convoluted.
I'd be a bit harsher on lock-ins, which are rarely worth the loss of control in the long run, but otherwise <3
I'd like to see some examples of this.
We run Bottlerocket (previously Flatcar, nee CoreOS) which itself is running SSM for "break glass," so: only about 3 directories (and children) can physically be written to, no package manager, no sshd, and in Bottlerocket's case even getting to a shell on the host is so complicated I have it pinned in our slack channel. I haven't tolerated "general purpose" OSes for production machines in over 8 years because jokers just love to "just this one thing to troubleshoot..." exactly like the story above said
While I don't use immutable infrastructure myself, I can see it being of use in some organizations.
Doesn't have to be an idiot - just a normal, fallible human.
We normal, fallible humans still benefit a lot from using good methods that take human error out of the equation.