A Pipeline Made of Airbags

I work with a project that takes an hour to build because it’s thousands of C++ files. What would you suggest I do to improve my build times?

The problem we solve by religiously sticking to "best practice" is the extreme engineering churn in "Silicon Valley" companies.

Not spending 9 months on-boarding might be worth extra minutes on deployment, IDK...

> and one is running tests and packaging software.

and linting, and code quality checks, and compiling, and optimizing, and pushing the package up to package & image repositories, and deploying it to sandboxes

I agree there is room for improvement. Parallelism in particular is a step that could shave up to 30% off some of the build pipelines I've worked with. I disagree however that your example should "make your blood boil". I feel like "Blindly following "best practices" instead of using our $170k+ Silicon Valley brains" is inciteful language that ignores the reality of the situation. We are all working from the tools we've got. The majority of those tools don't have good models for Parallelism. CircleCI for example supports "test splitting" parallelism, where tests can be run in parallel in different containers, but there's no workflow parallelism that I know of so that style checking, deployment to ephemeral environment, publishing, &c can proceed once the bud artifact is generated.

Yes we should try to adapt & grow. The very name, build pipelines, bespeaks a linear process. The ability to change is definitely at hand, & there are legions of us who would pick up the torch on this one & move forward.

But I disagree that our failure to do so is "blindly following 'best practices'". Yes this is one of the "problems you can solve" but it's a bit of a big deal. Configuring build pipelines is a fairly advanced task, & to advance to parallelism & our maybe 30% gains we need not only the machine work to support parallelism, but also the language work to redefine more flexibly how work ought to flow through a system, to build new models for computation that users can author that are graph based, that let work spin up when it's dependencies are defined & satisfied.

"Will most companies let their developers do things that may not be 'best practices' to do that?" Let's start with smaller steps, eh? Should companies let their employees venture forth & tackle new non-linear practices for software assembly? Yes. How many? Every company? No probably not. What responsibilities do you want to put on the team beyond delivery of something that works for them? What documentation do you need? Who reviews the proposed architecture to approve this new best practice you are trying to create? Is this an internal system, something you are going to make a core competency and sell, or are you going to try to really make it a best practice & open source it and try to grow it openly? What best practice tools will you use to develop this async build system, or should developers eskew convention & use their "170k+ Silicon Valley brains" on this (& other) layers too?

There are of course some build/integration/deployment technologies out there that do support wider realms of parallelism. Jenkins with it's groovy syntax at least supports creating stages that can be run in parallel. This is still leaving possible optimizations behind, as all items in a stage have to finish before the next stage can start; it's not fully async. Compromises were made but it works pretty well, helps. I've seen 25% cycle time reductions from parallelism here, it helps.

Overall, to me, this issue feels not so pressing. Engineers scrap together an enormous amount of their own time & effort to do what they can with what is available. They are extremely resourceful. I do indeed want companies that better let their employees pursue process & systems improvements, that spend more time stepping outside of product development & more time improving the house they live under together. Most of all, I hope that we are learning & sharing, openly, publicly, #LearningInPublic, so that the state of the art can improve, so that there is more & better material out there to decide what a best practice might be & how to head there. And how to have it make sense 2 years down the road when all the original engineers have found other opportunities & the new less seasoned crew has to suss out how to proceed.

I also want to touch briefly on the original article, although I'm not sure how. Because it has such a neat exploration of so many interesting rich capabilities, capabilities and that let software manage software: software defined software, software defined systems. And in ways that are outside our newer conventions, where we manage polyglot systems via containerized processes (versus using powerful multi-system vm's to manage). The authors love of empowering the operator with online tools, to let them situate & operate from inside the running machine systems. This is such a glorious & important capability, about how we commune with machines, & author is so right to call attention to what we've lost, to how much less operators are online with their systems & how much more distance there is. Being able to use with the machines is important.

And yet, here too I think the situation has nuance & reason. That it's not only a matter of best practices socially gumming us up & social group think making us dumb. Or, if they are gumming us up, it is in response to another form of getting gummed up. Systems used to be dealt with as pets, as carefully groomed & growing buckets of bits that had acquired various adh-hoc operations & scripts & subsystems over time, as operators went about working on the system & working on systems to help them better work on the system- the author leads with going in depth on the ad-hoc deployment systems that had grown up over the years internally on their system, in just such a manner. When anything is possible & at hand, many things happen. Over time, it's hard for newcomers (& seasoned veterans) to understand what has been done, why things were done, hard to even see what pieces are there. A system that is never killed, that's always carried forward, mutated, ongoingly & organically becomes an unintelligible bucket of bits, an opaque intertwined illegible historical artifact. In contrast, immutable containers don't attempt to strike much of a balance to this situation, don't leave a lot of affordances for many of the wonderful erlang capabilities, but they insure that there is a well know & iterateably improveable way to get from code to deployed code that everyone can observe & understand, that brings predictability & understandability that often, in the past, had been sorely absent.

Please have sympathy. Things are complex. Figuring out not only how to move forwards but move forwards in a robust enduring well-thought-out & mindful manner is a big challenge, & retaining that hunger & appetite for moving forwards, for doing good things is equally important, & navigating ourselves while under this tension is something we have to learn to live with, hopefully peacefully.

I've always thought that's precisely why PHP is bad.

It throws out the context baby with the state bathwater.

This is probable the reason for why it's so popular (meaning wordpress), but also why it's so full of security exploits.

Fail-fast leads to more stable systems.

I love both Erlang and PHP, but the philosophy of what to do with errors in requesrs is like diametrically opposed. For PHP, usually it's log, but keep on going as best you can. For Erlang, it's log and stop, and the request handler will generally be restarted.

I guess where they're similar is that an error is generally going to only affect that one request, the next request will have a fresh handler.

(Of course, these are generalizations, you can always do things differently in either language).

I’d recommending reading the post about the difference between erlang and php and reading the article again. This isn’t silently swallow errors. It’s saying start with fail aggressively but also don’t forget to identify situations resulting in failures in the first place and fix those bugs our handle the corner conditions properly. That’s diametrically incompatible with “log and continue” as at no point is that desirable according to this post.

Erlang language level immutability is a means to an end, it makes the runtime (especially garbage collection) simpler than a system with mutable state where you can have references to newer and older data.

At the systems level, immutability makes operating the system much harder. Especially when you consider Ericcson's basic design requirement of a telephone switch with a redundant pair of control computers; it's safer to deploy changes by making a temporary change to the live system than a permanent change to the backup system and switching over. In case of a bad deploy, you can live reload the old version on the live system, or if that's not possible, promote the backup system and restart the primary back to the known good state. If all goes well though, there's no period of reduced redundancy. In a restart to deploy situation, every deploy has a period with loss of redundancy, as well as a primary/secondary switch, and you can't deploy to degraded systems where one of the redundant pair is not working pending repair (these switches would most likely be deployed to towards the end of the cycle to reduce risk)

A large part of the philosophy is controlling state. Immutability in the small helps control where state lives. It’s difficult to control (or understand) state spread across hundreds of little objects (e.g. most OO languages) and be able to modify a running system. OTP Supervision trees and GenServers are all about partitioning state and organizing it. It’s similar in many ways to micro services, but more advanced (as the article mentioned regarding OTP Supervision trees and k8s).

Author here.

For the first system, it was deprecated without replacement, and just let to run by managers and people who had moved on to other teams (but used to work on it) who did the minimal maintenance required, and former employees were given emergency contracts in weird circumstances to deal with things. Roughly 3-4 years after, they finally replaced it after 2-3 attempts at rewrites that had failed before. The old design with minimal maintenance for years finally approached limits to how it could be scale without bigger redesigns; I consider this to be extremely successful.

It wasn't exactly bulldozed nearly as much as declared "done" and abandoned without adequate replacements while major parts of the business was just being rewritten to use Go and a more "standard" stack. Obviously these migrations always start with something easier and by replacing components that are huge pain points to its contemporaries, and you're left with more legacy stuff in the end that is much harder to replace done in the final pushes. I felt that the blog post would have veered off point if the whole thing became about that, though.

The people on these teams left in part because the hiring budget was redirected towards hiring on the new projects. The idea was that everything could be done in Go for these stacks (by normalizing on tools and libraries developed in another project and wanting to have one implementation for both the private and shared platforms), and the rewrites were to start with Ruby components.

You knew working on the Erlang side of things that no feature development would ever take place again, that no new hands would be hired to help, and that you would be stuck on call 24/7 with no relief for years. All efforts were redirected to Go and getting rid of Ruby, and your stuff fell in between the cracks. I was one of the people who left on the long tail there. After my departure, I was brought back on a lucrative part-time contract as a sort of retainer for years to help them in case of major outages (got 1 or 2 in 4 years) since that was the only way they could get expert knowledge once they drove us all away.

I'm still on good terms with the people there, it's just that "maintaining a self-declared unmaintained legacy stack without budget or headcount until we get to rewrite it in many years" is not where any of us wanted to drive our careers.

Interestingly, we tried very hard to add new developers. We wrote manuals, tooling, a book on operating these systems (see https://www.erlang-in-anger.com), wanted to set up internal internships so developers from other teams could come and work with us for a while, etc. Whereas our team was very willing, internal politics (which I can't easily get in a public forum) made it unworkable and most attempts were turned down. These things were not always purely business decisions, and organizational dynamics can be very funny things.

I agree with you. This article complains that generic industry practices are not as optimized as his niche solutions. Thats no surprise.

Of course a build and deploy system specifically designed for your product/environment is going to be more feature rich and performant, but once the initial developers leave or move on, all is lost. I have seen it happen multiple times.

Sticking with slower clunkier industry practices means you can hire someone off the street and they’ll have a much better chance of being able to understand the system.

If you want longevity, you use the standards. All engineering (mechanical, electrical, etc.) agrees on that.

> I'd like to hear from the other side of this - I'd like to hear the stories of the people who pushed the redesign away from the systems.

I'm certainly not one of those people, but they'll tell you things like how they standardized the new acquisition onto "best practices" and eliminated the "cowboy deployment" culture that was previously running rampant. I was part of an acquired Erlang team that went through this, although I mostly didn't deploy into the acquirer's systems, I managed the legacy systems, until they were mostly retired, then I retired.

They won't tell you that deploy times grew massively, requiring a larger team to manage (and that increasing the team size requires increasing the team size, because of increased time spent in coordination).

They won't tell you that because they've built their system to handle unreliable environments, they've neglected to make their environments reliable, and the system redundancy factor had to be increased to 3x instead of 2x because the churn rate on systems is too high to expect at least one out of two systems in a pair to be running. Of course, increasing the number of systems, increases deployment complexity, too.

I'd argue the biggest reason for a graceful restart would be reliability and latency.

Kuberenetes is a leaky abstraction that by default allows the deployment of sloppy unreliable apps if you don't know what you're doing. If something is broken and it just restarts automatically and no one is paying attention, it will do that in perpetuity. Ideally, you would see those issues and iterate to make your app more stable, but it makes it easy to just forget about it. The additional layer of complexity is also harder to understand easily for most devs that haven't spent the time to learn kubernetes and even with devs that do spend the time it can still make understanding how everything works hard. Letting tech debt like unstable apps grow into mud balls can reduce a team's velocity in the long term and in some cases the short term as well.

Kubernetes is great for easily sharing and executing recipes, but it can make it tough for people that want to become chefs that are able to use reason and intuition to build and understand complex systems.

When you have two systems that need to talk to each other, and one system expects the other system to be reliable, it’s difficult to completely toss out all graceful shutdown and handover. Usually this logic is handled at the service discovery or load balancing layer with the deployment bits, so people don’t notice.

Kubernetes does a really good job at hiding this.

When it hides this, it also hides a couple other things with software — like failing containers, or apps that require churn or fail. I mean this is literally the reason why things like monit were built in the older days — things broke if you didn’t restart them when they broke.

In kubernetes, it also is easy to accidentally to change the system and make that behavior change (at a global level), that results in stuff like your app not being restarted as often, which results in elevated latency, and subsequent failures.

> But what is the business argument?

Complexity costs money.