> and one is running tests and packaging software.
and linting, and code quality checks, and compiling, and optimizing, and pushing the package up to package & image repositories, and deploying it to sandboxes
I agree there is room for improvement. Parallelism in particular is a step that could shave up to 30% off some of the build pipelines I've worked with. I disagree however that your example should "make your blood boil". I feel like "Blindly following "best practices" instead of using our $170k+ Silicon Valley brains" is inciteful language that ignores the reality of the situation. We are all working from the tools we've got. The majority of those tools don't have good models for Parallelism. CircleCI for example supports "test splitting" parallelism, where tests can be run in parallel in different containers, but there's no workflow parallelism that I know of so that style checking, deployment to ephemeral environment, publishing, &c can proceed once the bud artifact is generated.
Yes we should try to adapt & grow. The very name, build pipelines, bespeaks a linear process. The ability to change is definitely at hand, & there are legions of us who would pick up the torch on this one & move forward.
But I disagree that our failure to do so is "blindly following 'best practices'". Yes this is one of the "problems you can solve" but it's a bit of a big deal. Configuring build pipelines is a fairly advanced task, & to advance to parallelism & our maybe 30% gains we need not only the machine work to support parallelism, but also the language work to redefine more flexibly how work ought to flow through a system, to build new models for computation that users can author that are graph based, that let work spin up when it's dependencies are defined & satisfied.
"Will most companies let their developers do things that may not be 'best practices' to do that?" Let's start with smaller steps, eh? Should companies let their employees venture forth & tackle new non-linear practices for software assembly? Yes. How many? Every company? No probably not. What responsibilities do you want to put on the team beyond delivery of something that works for them? What documentation do you need? Who reviews the proposed architecture to approve this new best practice you are trying to create? Is this an internal system, something you are going to make a core competency and sell, or are you going to try to really make it a best practice & open source it and try to grow it openly? What best practice tools will you use to develop this async build system, or should developers eskew convention & use their "170k+ Silicon Valley brains" on this (& other) layers too?
There are of course some build/integration/deployment technologies out there that do support wider realms of parallelism. Jenkins with it's groovy syntax at least supports creating stages that can be run in parallel. This is still leaving possible optimizations behind, as all items in a stage have to finish before the next stage can start; it's not fully async. Compromises were made but it works pretty well, helps. I've seen 25% cycle time reductions from parallelism here, it helps.
Overall, to me, this issue feels not so pressing. Engineers scrap together an enormous amount of their own time & effort to do what they can with what is available. They are extremely resourceful. I do indeed want companies that better let their employees pursue process & systems improvements, that spend more time stepping outside of product development & more time improving the house they live under together. Most of all, I hope that we are learning & sharing, openly, publicly, #LearningInPublic, so that the state of the art can improve, so that there is more & better material out there to decide what a best practice might be & how to head there. And how to have it make sense 2 years down the road when all the original engineers have found other opportunities & the new less seasoned crew has to suss out how to proceed.
I also want to touch briefly on the original article, although I'm not sure how. Because it has such a neat exploration of so many interesting rich capabilities, capabilities and that let software manage software: software defined software, software defined systems. And in ways that are outside our newer conventions, where we manage polyglot systems via containerized processes (versus using powerful multi-system vm's to manage). The authors love of empowering the operator with online tools, to let them situate & operate from inside the running machine systems. This is such a glorious & important capability, about how we commune with machines, & author is so right to call attention to what we've lost, to how much less operators are online with their systems & how much more distance there is. Being able to use with the machines is important.
And yet, here too I think the situation has nuance & reason. That it's not only a matter of best practices socially gumming us up & social group think making us dumb. Or, if they are gumming us up, it is in response to another form of getting gummed up. Systems used to be dealt with as pets, as carefully groomed & growing buckets of bits that had acquired various adh-hoc operations & scripts & subsystems over time, as operators went about working on the system & working on systems to help them better work on the system- the author leads with going in depth on the ad-hoc deployment systems that had grown up over the years internally on their system, in just such a manner. When anything is possible & at hand, many things happen. Over time, it's hard for newcomers (& seasoned veterans) to understand what has been done, why things were done, hard to even see what pieces are there. A system that is never killed, that's always carried forward, mutated, ongoingly & organically becomes an unintelligible bucket of bits, an opaque intertwined illegible historical artifact. In contrast, immutable containers don't attempt to strike much of a balance to this situation, don't leave a lot of affordances for many of the wonderful erlang capabilities, but they insure that there is a well know & iterateably improveable way to get from code to deployed code that everyone can observe & understand, that brings predictability & understandability that often, in the past, had been sorely absent.
Please have sympathy. Things are complex. Figuring out not only how to move forwards but move forwards in a robust enduring well-thought-out & mindful manner is a big challenge, & retaining that hunger & appetite for moving forwards, for doing good things is equally important, & navigating ourselves while under this tension is something we have to learn to live with, hopefully peacefully.