Why on earth did we choose Jenkins for 2019?
rookout.com
rookout.com
Just seems strange to have to justify something that "just works" for so many years. It isn't even really "old" compared to many tools you use every day.
Why run a command in a shell build step when you can install a plugin that abstracts everything away in a non-intuitive manner?
Don't allow your Jenkins Master to create the Jenkins Slaves -- instead create the slaves via the same tooling your create the master, and register them with the master via the same (or similar) discovery mechanisms you use elsewhere in your environment.
Don't use plugins that configure your build tool of choice. Yes, it's initially spiffy that you can click around and change ant's behavior, but when things break you can almost never repro manually.
Shell steps. Shell steps. Shell steps. Make sure it's trivial to manually execute any step in the pipeline. It should be rare that you actually manually execute the step, but it helps you avoid landing in situations where it's impossible to debug the Jenkins build, because your plugin or non-shell step insists on cleaning up after itself in a manner you can't manually override.
Finally, read this before even getting started: https://queue.acm.org/detail.cfm?id=2841313
Once you've read it, read it again.
When we want to add a plugin or whatever we just add it to the plugins repos, git add/commit the repository and re-kickstart the Jenkins VM.
The Jenkins kickstart then just restore the previous git config/logs back into /var/lib/jenkins. Everything comes back as it was, but the new/upgraded plugins are available.
Easy peasy. But then again we don't have a huge number of jenkins jobs or a massive cluster of Jenkins masters/slaves so yeah.
You could feasibly do it without re-kickstarting the machine, but we like to show off how the Jenkins server can rebuild itself with a self-hosted Jenkins job.
> eight years in
Maybe they chose to stick with Jenkins in 2019 and not discard their 8 year old Jenkins setup?
1 years ago, it doesn't even have a support way to call `scm checkout` and get revision. You have to go all way around by calling out to shell to get `git rev` write to a file and read it back. Groovy DSL supports it now but given backthen I feel like even Jenkins team didn't push Jenkins hard enough.
Configure Pull Request Builder, Github Hook Trigger and Build Context are a hit or miss with Groovy DSL too :(.
I think in 2019, it has no reason to continue use Jenkins other than we are too familiar with it and don't want other approach. Concourse CI has a steep learning curve but it's way better and a right approach to me. At least, you don't have a UI where people go in and trigger build. Jenkins make it easier to be messed up.
The only reasons to still use Jenkins in 2019 are SCM polling and e-mail notification. For manual builds I just use python with fabric for remote builds.
Also we use scripted pipeline all the way
Basically, Consul would monitor the Jenkins master for liveness. If it discovered that the master had gone down, it would spin up the cold standby machine, first attempting to use a recent disk snapshot and then by re-running plugin installation, JJB, and copying in a secrets store file from Vault (and essentially starting the server fresh again). Then all the slaves would self-configure using Consul to figure out which "master" node was actually master.
It was gross, and I hated it, but it gave us Jenkins failover in under a minute in most cases. We only lost all our job history once in a year and a half, and this was in a flaky-ass openstack environment.