The worst thing about Jenkins is that it works (2019)

Gitlab CI is still the best CI in the game IMO, but GitHub Actions gives it an incredible run for it's money because of how easy action re-use is.

I meant to make a blog post about this, but here's a good a place as any: GitLab absolutely innovated many hard parts of CI/CD as a platform-native piece, but it feels like they lose slightly to GitHub on what GitHub does best -- social virality for developers. The problem with losing slightly there is that the advantage compounds; if developers find it easy to make and share, then they make and share which makes more people make and share and generates tons of value for a platform.

It may not be safe, but being able to:

    - uses: some-developer/some-repo@some-branch

Is absolutely amazing. Everything else (generally) GitLab has, and (likely) did first, but the social viral stuff GitHub just gets right in a different way than GitLab does.

No one would accuse GitLab of being a "copy" in recent memory like they did in the bad old days, but IMHO they need to just copy the shit out of when GitHub gets social things (like this) right.

Jenkins is not perfect by any means, but I can say it is much better than existing solutions (taking pros and cons after trying and using many of them). Sure, it doesn't have fancy UI or flying buttons, nor is it rust/PWA/react/whatever-rage-is-these-days, but it does its job - it works, is free, and has been very well-maintained for ages. Download a single file and run it. Want to extend it? There is a plugin for everything under the sun (but be careful, you can easily bloat it).

IMHO, I don't want bells and whistles from the CI tool. CI tools are dry; I see them as a car drive shaft - it must work properly, and I don't expect it to be pretty. If you need to showcase it, you are probably doing something wrong.

And one more thing: I expect Jenkins to be around and free in the next 10 years. I doubt that will be the case for "insert-your-favorite-ci-from-some-company".

All of the test runners have their issues. I've had as much problems with Jenkins as I've had with Github Runners and similar solutions.

The best way out of this is to subdivide responsibilities:

- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`.

- Put all permissions and code checkout and artifact publishing credentials in the test runner, but no logic. At most you would put processing of test output like making junit/coverage/tap more web-readable.

That's it. Split the efforts cleanly, and things fall into place. Also, you can switch runners easily - less needs to be reimplemented in whatever runner config DSL is picked.

Having worked with a huge Jenkins deployment at a large company somewhat recently (>10,000 jobs), I found it worked okay enough that company leadership never felt it was worth the pain if switching. But all the friction points added up to a system that was rarely touched by anyone who hadn’t learned where all the bodies were buried. Over time that meant as an engineering org we were underinvesting in CI; there was a lot of quality-oriented stuff beyond unit testing that never got implemented in CI because the typical dev was unaware of how to change it or fearful about trying to change it. The relative accessibility of Github Actions (and other config as code alternatives) transforms the average dev’s relationship to CI from consumer to owner/developer/maintainer, and I think that is extremely worthwhile even if these tools bring some new problems of their own.

Interesting, the side snipe about Docker

> One of the mostly-false promises of Docker, as it was sold to me by the true believers who introduced me to it, was that, if you do it right, you can run the same docker image, and therefore have basically the same environment in production, in CI, and on your local development machine.

I literally have never worked anywhere that hasn't used the same container in dev/test/prod.

People who have had to deal with processes that rebuilt images for each environment, can you tell me what was different in each image?

> The head of SRE championed Gitlab CI. I resisted this idea because I, the relatively inexperienced manager of a nascent team, was daunted by the prospect of trying to supplant Jenkins, Github, and JIRA all at once.

I think this is a really big scoping mistake. There is a clean glide path, and I have no idea why you'd think you need to replace Jira.

You can use Gitlab Runners in Github (https://docs.gitlab.com/ee/user/project/integrations/github....). Then with your mirrored repo you can try out the Gitlab features and realize it's better to use Gitlab for your repo if you're aso using it for CI. Having switched back and forth over the years (with tens of engineers using the repos), it's really not a lot of work to migrate between Github and Gitlab.

Finally, if you like Jira, just use that? Sure there are advantages to having everything in one place, but if you're already using Jira+Github then you'll get an equivalent experience with Jira+Gitlab.

I think this is one of those things that if you had actually prototyped the migration, you would find that it's much easier than you thought it was going to be.

I truly, truly dislike Jenkins but haven’t found a better locally hosting CI system that is repo agnostic.

My issues with Jenkins:

1. Groovy is a really annoying language to configure stuff in. I’ve never written it for other things but it has so many little idiosyncrasies that catch you out.

2. Jenkins doesn’t have much in the way of resource management. It’s easy for a job to take over the entire node and Jenkins doesn’t have the smarts out of the box to monitor this and put things in a different node.

3. So many random job killing bugs that boil down to poor job isolation and resource tracking. Every now and then killing one job can take down another.

4. The UI and UX is bad. I know there are better interfaces for it, but they all just have such limited interface exposure of state beyond “running” and “failed”. Estimate time is useless because it aggregates failed jobs.

5. Inter job dependency management and tracking is poor and annoying to setup. It’s doable, but really should be a better first class citizen.

In many ways, I start looking to other tools to fill my CI need that aren’t meant for CI: Render farm managers.

Film farm rendering is largely similar to CI at a structural level. But they have much higher throughput of individual jobs and have built around making the most of that. There’s better inter job dependency tracking and resource management because of that, and they’re built to expose more information up front to help non-tech savvy artists or render wranglers who need to see status quickly at a glance on a few hundred jobs at once.

There are a large number of quirks I’ve had with the Kubernetes runner on Jenkins causing flaky behavior, but when it works, it’s mostly fine after figuring out Groovy. The 1 in 50 time a job needs to be kicked off again due to some strange error is annoying though.

More often than not, I end up creating a sandbox folder in Jenkins and cloning workflows to work on them by hand without having 90 commits of misc errors and typos. This has helped speed up some iteration.

Sort of off topic from the article, I think where almost every single CI engine falls short is the extremely limited scope they cater to - that every job ever made must be solved with the cargo cult ideology where every CI job mustn’t ever have human input, everything must be perfectly automated CI. This is wonderful when it is possible to do, but sucks when it isn’t feasible.

Jenkins’ strength is that it’s actually a workflow engine masquerading as a CI tool, and that gives a lot of flexibility for a number of bespoke processes.

Do you want to wait for someone to click a button to grant permission to run a job because the test might spin up extremely expensive resources and you’d really rather someone baby sit it? Jenkins has you covered.

Or maybe it just isn’t possible (or worth the time) for your team at a point in time to automate all input to a job, so spin up a Jenkins job that lets someone manually specify parameters.

Maybe your flow can trigger a release of software but you want a human to come behind to do a manual once-over as a final check, then also capture who/why approved it.

More CI tools need to accept the reality that not everyone can implement the perfect ideal of CI jobs. Some jobs will require a human step somewhere, be it custom input to a job or waiting for a response.

Jenkins is rock solid. Been using it for three years to deliver highly critical software. The development experience is the worst of all ci tools since no one wants to setup jenkinspipelineunit to test the pipelines. Dagger might be the salvation here, but even that tool didn't support Jenkins out of the gate. And for those that are reading this, the statement about pipeline groovy being a trap is exactly right. Avoid writing groovy at all costs. No one on the team wants to get good groovy and if you do, you're weird. The way we do it is to just use the Jenkins file to call the scripts (or tools) with bare minimal script code.

Edit, forgot to add that I recently interviewed a bunch of candidates for a high paying job at a ai hardware company and every one of those candidates stated they worked on or near a project that involved moving from Jenkins to Gitlab CI.

Does Jenkins let you test workflows locally?

That's the thing I hate that about CircleCI, Github Actions, etc... having to "commit, upload, wait" to test every minor change =/

What would people recommend for a self-hosted CI system in 2023?

I'm currently leaning towards Jenkins, because neighboring teams are using it (with some pain due to their scale, but theirs is 10x-100x ours), and I have some ~10y-old experience with it.

(Our builds are compiling for embedded targets, with a Makefile-based system)

The blog perfectly sums up the feelings of anyone who has worked deep in jenkins. It works, but goddamn it's hard not to hate.

Dagger is the answer to jenkins woes. It's saved my sanity and made CI development tolerable again.

* Real language support. No groovy, no yaml.

* Debugging available in your native IDE tools.

* Clear docs in any language of your choice.

* Reproducible builds you can run locally. Just need docker.

This all comes with the bonus that it can actually just consume GHA yaml with some of the tooling folks have made.

Jenkins is amazing for one thing in particular, and I've not really seen it replicated well elsewhere (like gitlab ci): common job execution. Letting people run jobs on/against shared equipment, rather than having people run scripts locally.

I wouldn't recommend it for general CI other than that. Think of it like a butler, able to do small jobs, not as your complete home assistant.

> “everybody should be writing and maintaining their own CI jobs”

And, of course, everybody should be using whatever language and version they want to, and run their software on whatever cloud they want. The notion of this makes me super uneasy, but at the same time it's a good explanation why many big orgs have 100s of engineers when they can probably easily do with 10s of them.

I had a gig (circa 2014) where Jenkins was used in atrocious ways where it was basically running the show. Yet there were no problems with Jenkins itself, it was running smoothly

> Your editor won’t syntax highlight the Bash inside Groovy. You can’t run “shellcheck” (or any sort of Linter) on the Bash inside the groovy. You can’t very easily execute your shell commands to test them.

this lands, and not specifically for jenkins. CI generally is inscrutable and difficult to test

every CI has its quirk

github CI feels fundamentally worse than gitlab because job artifacts are native in gitlab -- this makes a million things easier. github's own action for attaching things to releases is deprecated. the upload-artifact action doesn't work in nektos/act (the only way to test locally)

The real failure here was the failure to recognize the sunk cost fallacy. My team attempted the pursue the "supported" path with Concourse for a recent project, and I watched them bang their heads for a week, realized it wasn't going well, set a meeting where we all discussed and decided to move to GitHub actions. Now everyone is happy and the project is back on track and I don't have a write a blog post like this one about Concourse.

The kubernetes plug-in for Jenkins works well for us, and as the checkout/clone happens in the pod you don’t have the workspace bind mount problem. We are using docker-in-docker at the moment but looking at buildah. We do parallel windows container builds as well.

Jenkins is a bit like Perl IMO: there are ways to use it that avoid the majority of its sharp edges (eg putting logic in bash and just call bash, avoid groovy) if you know the perils that lurk.

Concourse CI is great, it felt like it would be easier to write Concourse resources for the custom things we needed rather than fighting Jenkins anymore.

I don't mind Jenkins pipeline limitations because my philosophy is that CI should be as simple and contain as little build logic as possible.

Every job should be a few simple commands or a python script or something I can run locally. This simplifies reproducing and debugging issues caught by CI -- there's nothing worse than wasting time trying to reproduce what happened.

I really resonate with this post after working with Jenkins for ages. In my experience I wasn't very happy with Bash in the end either and moved to Javascript for reliability, performance and tighter integration with the Cloud were deploying to. It also allowed us to run locally and avoid having to deploy to Jenkins to test.

The discoverability issue is also a massive issue, declarative pipelines massively over promise its capability. Pipeline configuration was mostly still done in the UI because extensions had 0 documentation about how to declaratively define its configuration. Searching on the internet for the magic incantations as the author puts it is such a waste of time and I just compromised to get it running.

I'm glad this post is out there to articulate these common challenges when you invest so much into Jenkins.

the worst think about Java (in general, Jenkins is Java) is that it solves a real problem that companies have.

I put, however, that only companies have the problem, not programmers, and not even regular (sized) engineering teams; except when they have a fast employee turnover speed.

I do not have a very clear idea, but roughly speaking I'm saying that java does really well in the "too many engineers are working on the same code but not at the same time" problem space

The complaint about a long feedback loop is true of every CI system I've used (jenkins, travis, gitlab, github actions). None of them really have a good way to test a pipeline locally. The best solution here is to put as much of the build logic as possible in scripts and your build system.

The part about using Docker is wrong. You can use the docker plugin and just run any image you want, and then run the code inside that specified image. You can pass additional parameters to also run it within the context of any particular docker build stage.

I use the same Dockerfile for production builds and for running all the CI related tasks, without any problems. And the same single pipeline is used for PR hooks and releases.

Post also mentions Blue Ocean UI, but that one is deprecated and should not be used at all.

Jenkins is far from perfect, but unless you can use Gitlab (and sadly you can not in every case), it is the most feature rich self-hosted tool out there. Just make sure you set it up with Jenkins Configuration as a Code plugin out of the box.

Jenkins is the worst CI system except for all the others. Everything feels like a pile of hacks upon hacks, maintaining it is tedious and unrewarding, new functionality is always bolted on and undiscoverable. But every alternative implements the first 20% of features that provide 80% of the functionality and then never gets around to implementing the other 80% (and interestingly every attempt to "modernize" Jenkins, such as "Blue Ocean" or "Declarative Pipelines" as mentioned in the article, suffers from exactly the same problem).

I wish there was a nicer tool that worked. But there isn't.

As somebody who has spent his share of time interacting with Jenkins as a user, and messing around inside it as a developer, I am definitely biased. For or against, I'm not sure tho ;-)

But the problem with Jenkins is the problem with all CI. Our (software in general, not Jenkins specifically) build process are all godforsaken nightmares that would put HP Lovecraft and Rube Goldberg into rubber rooms. You can't add a bunch of extra automation to any of these systems and get something better, let alone a system that needs to be able to automate so many different and unique snowflake horrors.

Needs a (2019)

Richard was my manager in early 2019, and was awesome.

I’ve long had a very similar argument for why commuting in the Bay Area* sucks: because it’s just tolerable enough that people keep doing it. It’s not unmanageable, but it drains your life little by little every day.

* This, of course, applies to all long commutes everywhere, like how it applies not only to Jenkins, but lots of other software too… especially Jira.

This is my favourite quote:

    The worst thing about Jenkins is that it works. It can meet your needs. With a liiittle more effort, or by adopting sliiiightly lower standards, or with a liiiiitle more tolerance for pain, you can always get Jenkins to do aaaaalmost what you want it to.

This is great, opinionated writing!

I'm so much more happy with GitLab CI. The only thing that is easy to implement in Jenkins, but an outright horror in anything else, is the usual "helper scripts" stuff: sync the production database to integration (and remove PII in the process), run reports and data analysis... the best way I've figured out to implement this is to create an empty branch (git checkout --orphan helper_xyz) and then run pipelines against this branch either manually or on a scheduler, but it sucks.

And this is what keeps Jenkins afloat, despite it being a consistent source of CVEs both in core and in its myriad of plugins that don't bother to follow semantic versioning so every upgrade is playing Russian roulette with your company.

at my company this is all very easy. there is no version control of any kind and also no automated testing. solves a lot of problems.

I stayed away from jenkins for a long while because of its bad reputation, but after diving into it for a recent project I was surprisingly impressed how easy it was to set everything up (with the docker image at least), and it seemed that there was a solution for just about every convoluted build scenario I could come up with. In contrast, I've tried out other tools and encountered: confusing uis, too much focus on docker, broken interop with forges, poor documentation, too much yaml, ...

I guess the flexibility with jenkins is probably both a blessing and a curse.

Of everything I've tried, gitlab co is my favourite. It's just a shell script, and that makes it super simple to replicate yourself locally, and - I think most importantly - it's really obvious what it's doing. I think that's my biggest problem with Jenkins and GitHub and such is all the plugins that do something, but without digging through the source code you cannot work out what they really do

cloud is so good now it’s hard to justify not doing something bespoke. ec2 spot is insanely cheaper than turnkey cicd, and better in almost every way.

i’m delighted to pay 30% over infra cost for convenience, but not 500%. and it better actually be convenient, not just have a good landing page and sales team.

this month i learned localzones have even better spot prices. losangeles-1 is half the spot price of us-west-2.

for a runner, do something like this, but react to an http call instead of a s3 put[1].

for a web ui do something like this[2].

s3, lambda, and ec2 spot are a perfect fit for cicd and a lot more.

1. https://github.com/nathants/libaws/tree/91b1c27fc947e067ed46...

2. https://github.com/nathants/aws-exec/tree/e68769126b5aae0e35...

> Warning – I’m going to rant for many, many paragraphs. My advice is to skim

I had a good chuckle at this.

I agree, Jenkins is awful. We also started with Bash scripts, but eventually moved to using Ansible playbooks, which is much nicer than Groovy. Alternatively, I recommend using Earthly as well.

I agree with most of this. I end up with a lot of

      sh 'make all'
      sh 'make test'

In my `Jenkinsfile`, but there's not a great way to test things locally.

While we are plugging ci tools, I enjoyed buildkite at my last gig

Wow, this title sums up in one sentence what we all hate about Jenkins. It's like the equivelent of timeouts being worse than fast fails.

Github actions works great Even AWS codebuild is good.

The best part about jekins is that it works. The worst part is that many people dont want to work with jenkins.

Well, to be fair, pretty much each of these points apply to all other CI tools in existence.

>The head of SRE championed Gitlab CI. I resisted this

Well there's your problem right there.

Jenkins has improve a bot. before github action has self host runner, Jenkins has worker node years ago.

using jenkins declarative pipeline, almost everything can explain in code.

updating jenkins is a walk in the past.

the most tedious part is dealing with no longer maitenance or support plugin.

make sure to never expose jenkins out. use a relayer to bring webhook traffic in

Really like the one line dissmissal of gitlab ci - haha, what a n00b.

Been using git hooks with in-repo Makefiles and small bash scripts to good effect for some years now.

It's fast, simple and took a couple hours to set up. Rarely ever needs attention. It just works.

The most interesting part IMO was at the end when op was noting his new company doesn't have teams/engineers create and maintain their own jobs. I've worked lots of different places now and something that's always frustrated me is how reticent engineering orgs are to reevaluate their processes and policies. Like it looks like his first org could have moved on from this problem by adopting the second org's policy. There might be other tradeoffs, but IME no org even wants to hear about this stuff, and they definitely don't have metrics for evaluating anything such that they might say "huh, this isn't working."

We're using Jenkins at work. For building artifacts, it's alright as our use case is simple. But then we're also using for our automated tests, which were alredy kind of a dumpster fire in terms of reliability (both because of mistakes made when automated testing was put into place and because of issues inherent to our product) and I don't think Jenkins is helping a lot here, even after consequent investment in dev time.

> On a previous team I had used Concourse CI to some extent, but I wasn’t really blown away by the experience. Travis and Circle were mentioned. I was a fool. I should have committed to seriously researching some of the contenders and making a more informed decision, but I lacked the willpower and the discernment.

The whole post can be summed up as he had very little CICD experience. Made lots of beginner mistakes, which is easy to do in Jenkins. Then decided to write a post where all his complaints about Jenkins are not only wrong but are the issues that plague all the other CICD tools.

> So instead of writing Bash directly, you’re writing Bash inside Groovy

Why are you doing that? You have a fully featured programming language and you are running `sh('npm install')`. You could do this instead https://github.com/DontShaveTheYak/jenkins-std-lib/blob/mast... . How is bash inside of YAML better?

> The trouble is: Groovy is a much, much worse language for executing commands than Bash. Bash is interpreted, has a REPL that is great for experimentation, does require a ton of imports, and has lightweight syntax. Groovy has none of these things.

Groovy has a language server, linters and a vscode IDE plugins. They are probably not as stable or full featured as the bash ones, but they are available and very few take advantage of them. Again, how is YAML+Bash better?

> The way that developers test their Groovy steps is by triggering a job on the remote Jenkins server to run them. The feedback loop is 2 orders of magnitude slower than it is for just executing Bash locally.

This is a rookie mistake. For about 60-75% of pipelines you can run them locally in a docker container on your local machine. You can even set up hot code reload so as you change your pipeline the Jenkins reloads it. You can also configure the job to kick off a build when it reloads the code. When Jenkins is configured correctly it has the fastest feedback loop of any CICD tool on the market. GitHub actions comes in a close second since it can also be run locally but you cant run a "clone" of what you run in production, like having the same secrets, so it gets second place. Beside Jenkins and GitHub actions, I dont know of any solutions for the other tools.

You can run a GitHub action on Jenkins. It's a very deep and complex system. It's like an iceberg and so many engineers dont leave the surface before deciding it sucks and one of the YAML CICD tools is better. Sure the YAML alternatives are EASY to get started with and to do basic stuff with. But they are Terrible at anything complex. While Jenkins is not easy to get started with, once mastered, you can build complex pipelines with ease.

I get that I'm a Jenkins fanboy. Most of the things I mentioned above, I either contribute to or I'm the author of. I know Jenkins has issues. I know it has hurt lots of people, I read the complaints online. But it's still the best out there. The best software in the world is not written in bash or yaml and the same is true of the best CICD pipelines in the world. It's a shame very few people get to see/use those pipelines.

Avoid Jenkins, and if you can, try https://www.jetbrains.com/teamcity

"TeamCity Professional is free – even for commercial use – and has no limitations on features, number of users, or build time. It allows you to configure up to 100 builds and run up to 3 builds in parallel, which is more than enough for most projects"

Aside from the docker snipe I've already commented on, everywhere that I've had to deal with Jenkins it's been an issue because it's always a pet.

There are always issues, and like the author, it's "only" about 25% of the time. A quarter of all builds fail due to things absolutely not related to that specific build.

> oh, we were resource constraint at that one second, retry and it'll probably work

> oh, we can't replicate that error message from a tool that you have no visibility over, retry and it'll probably work

> oh, we have no idea why something timed out, retry and it will probably work

Now when I have a failed build I don't actually trust the build before I don't trust that there was an edge case in an integration test that failed. Because it's almost always something outside of the code.

[Edit] downvotes but no constructive responses? The issue isn't necessarily Jenkins, it's the type of people who use Jenkins to roll their own everything.

Jenkins works...

No it dosen't or maybe it does for java which I don't care about.

The ui looks like from the 90s.

Finding anything in bigger projects/companies is impossible... I have browser bookmarks to find anything.

The log view is annoying at best.

I have nothing good to say about jenkins, it feels like it is stuck in the 2000s and refuses to change.