Improving maintenance of your Jenkins(file) pipelines
robertnorthard.com
robertnorthard.com
I've seen some places configure Jenkins jobs for a project in the repo itself, so running a specific type of build depends on it being defined in that revision of the repo you're using.
There are 4 basic ways to manage the initial job configuration: By Hand, Save The XML File, JCasC, and Groovy init files. JCasC is of course the most ideal, but also the least intuitive, and requires further server setup. Putting a Groovy file in an init directory is the most straightforward way, using code. And if the Jenkins server is already running, you need a specific interaction with some API call to actually update the job in the running server.
You can also do more horrible things, like Jenkins Job Builder, but I think making someone manage that is against the Geneva Conventions.
A job that would run that and create the other jobs.
Incidentally, a similar (but native and more streamlined) approach is taken by Buildkite: job that processes a DSL that then generates other jobs (could even be recursive if you like that sort of thing) [1]
Because then you can also run it locally to produce the same effect.
Jenkinsfiles/Groovy DSL/XML files should be the 20 other things: job dependencies, notifications, VCS configuration, retries, coverage reports, etc.
Same with anything: Travis, CircleCI, etc.
Over the years I've worked with most of the major players, most recently Gitlab CI. The one thing that makes it hard to go back to Jenkins is the poor quality of the plugin ecosystem. So many are abandoned or buggy that any upgrade comes with a breath and a prayer.
I'm probably gone for good. Appreciate everything they did for the industry but there are more robust choices around.
I don't think they went back to Jenkins.
Developed shared libs that serve full enterprise organization and do complex deployments. If project follow same rules, it is easy to encapsulate logic into shared lib and have jenkinsfile to call it with few params.
e.g. Jenkinsfile ``` @Library('common-libraries') _ deploy(product: 'xyz', environment: 'exyz') ```
Then, updates in shared lib propagate to all projects.
Not that Jenkins itself is bad but keeping the base and plugins up-to-date is really painful. There are a lot of plugins, yes, but many are basically inactive or poorly written (some are really good tho).
The good thing is that there are some alternatives to Jenkins, even very good ones if you have money to spend.
It's also easy to run locally despite what some people have said in the comments. You can literally download a war file and run it with java -jar jenkins.war.
Jenkins is powerful, flexible, and easy to set up. Just avoid most plugins and run your own Groovy/python/shell scripts. Easy peasy.
Also - what are people running in place of jenkins? The only tech I've seen mentioned is gitlab.
Getting Jenkins to load a pipeline definition from the source branch is extremely complex, so much so that I’ve yet to see it done well.
run 'sh ./build.sh && ./test.sh && ./deploy.sh'
You can also call individual steps, which I do from a Jenkinsfile. This way you can still see a nice pipeline woth different steps.
All this runs in Docker. This is integrated almost properly with Gitea (only thing missing is an icon in Gitea on success/failure).
Jenkins is fine behind the firewall. Its not terribly elegant but its very flexible.
That doesn't really help people handle the software well.
I worked with Robert and told him I wasn't a fan of the shared libraries approach because it abstracts the workflow away from the team, when the reality is we want more people to understand the pipeline, not less.
Jenkins itself is a bit of a hodgebodge of legacy crap.
- Plugins that do next to nothing. Hard to reproduce this.
- Inability to run locally; gives you the rope to hang yourself.
- Bit of a security mess
- Upgrade and admin story historically pretty poor; lack of infrastructure as code, etc. Conflicting versions when you restore.
- Unclear story on relying on state. Why isn't my build environment clean?
Jenkins is actually a pretty good tool, especially when you use it well. But once you're been around the block a couple of times and deal with teams failing with a tool, for a variety of reasons, you start to wonder if maybe the complexity of the tool itself is the culprit.
Personally it was easy for me to switch from travis because all my CI scripts are actually bash scripts, so the .yml CI file is just a few lines pointing to those scripts. All I had to do with gitlab was make a base image to be the "CI image" with all the tooling necessary + docker-in-docker.
Upgrades to GoCD have been very predictable, in contrast to Jenkins :-)
One reason why Jenkins is so widely used in enterprise companies is that within traditional corporate structures person hours are readily available whereas software purchases have to be approved by someone with budgetary discretion.
It's generally much more expensive to customise and maintain a brittle Jenkins installation than it is to buy a comparatively low-maintenance turnkey CI / CD solution.
However, the people who can do that customisation and maintenance work usually are there already anyway. Therefore people rarely bother with finding the most appropriate or most cost-effective solution but settle for the one that comes with the least resistance.
There's also a way to get CUDA working in Docker by passing the device paths, which gitlab-ci-runner can also use.