Unpopular opinion: CI/CD engines are an awful idea

My philosophy is keeping the CI/CD workflow definition as simple as possible, and passing off the task itself into a shell or python script. The only thing I typically perform in a task definition is secret management, tagging & uploading build artifacts, managing distributed build agents, etc. and leave any complicated build process and success/failure criteria for the script to decide.

Jenkins works great for this purpose. It's fairly sketchy and held together by duct tape, but I find its developer experience to be much better than equivalent commercial options for a small scrappy team.

My playbook for a small team building a variety of software (Robotics software, web-based UI, AI, mostly running in Docker or built to static executables):

- Spin up a Jenkins master on a cheap VM

- Spin up platform/project specific VMs or physical machines as build agents, leave developers in charge of dependency and tooling installations on the build machine for their product. Don't bother with 100% reproducible tooling until you need to scale up to more than a few build agents.

- Maintain all build and deployment scripts inside the repository, next to the code, managed by developers. Store build pipeline/task definitions in the repository, and keep developers responsible for their service.

- Keep some boilerplate and convention around for pipelines and enforce it. Make sure all build pipelines use a similar git scheme. Doesn't matter as long as it's easy + reliable (ex. deploy every commit from develop w/ staged rollout? "Git flow"-based release workflows, etc)

- provide SSO access to the whole team via a plugin ASAP -- Avoid managing individual credentials to the machine

- Limit Jenkins plugins used for build actions! never use a plugin doing some action interacting w/ cloud resources, eg "AWS S3 Upload". Just asking for a security breach, and eventually turns the Jenkins UI into a giant mess of random UI controls. Instead, where possible use native tools eg. the AWS CLI instead of a Jenkins AWS plugin.

The concept of yaml config also has another issue: inability to test locally.

When I come onboard to a project I always see git commit history that goes something like this:

updated gitlab-video.yaml Fixing bug in yaml Trying again Trying again Fixing another bug in yaml Trying again …

What I’ve instituted in my projects is the yaml is only for specifying the docker container, and passing in secrets to shell scripts which define the build.

Keep in mind that it be be true that CI/CD engines are an awful idea (for some people and orgs) while at the same time that CI/CD engines are needed (for some people and orgs). Not everyone can spin up Jenkins on a cheap VM, write scripts to do a build, and most importantly maintain it for years.

You can't hire people with experience on your homegrown scripts. You can hire people that have experience with a commercial CI/CD engine. Having worked on projects that used homegrown scripts, I can attest to it ending poorly and blowing just as badly as a CI/CD engine can end things poorly.

I want to shill a bit for Concourse- https://concourse-ci.org/

All the plugins are docker containers that read json on stdin, and write json to stdout. It's pretty neat.

Man, that's crazy, I had no idea CI was like that. I mean, it's been like that at all the companies I worked for, but I figured it was just because we were clueless, haha.

well, yaml based DSLs seems are mainstream now (I don't say they are perfect), they have all the mentioned issues, however if they are substituted by pure programming languages, we have another issue - we tie ourselves to a specific language ... I don't know what choice is better ...

I guess the problem grows the more complex the pipeline. I’ve found gitlab CI to be mostly manageable for basic build test deploy sequences

In what universe is this an unpopular opinion...

There are a number of companies out there trying to figure this out and gain traction in this space

Let’s get rid of car factories and hand-build every vehicle and see how it goes