Untangling Jenkins
conjur.org
conjur.org
It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture).
Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is one of the best tools out there for its role as a prettier cron/queuing system.
A few tips:
* if you can afford it, stop ASAP using the UI for configurating jobs; look into Pipelines or the JobDSL, based on your needs
* once you have more than a few jobs, stop running anything on the master node, for increased stability
* use the minimum number of plugins and tie as few things to Jenkins as possible; script as many things and put the scripts into version control
* backup the global configuration; you can try to version control it but it's tricky; at least back it up periodically and most of the time it is enough
That's it for now, for more details, $200/h :p
Seriously? Downvotes? On a holiday on a plane and I get down votes asking for a little help? I did try and google and didn't find the stuff.
https://jenkins.io/doc/book/pipeline/syntax/#compare
Here are details on the Groovy Sandbox (provided by the Script Security Plugin and installed by default when installing recent Jenkins). The Groovy Sandbox is on by default for Jenkins Pipeline:
https://github.com/hyperledger/ci-management/tree/master/jjb
for instance
I am asking this because I don’t see a problem with running turning compete scripts if the application is most likely to run arbitary code anyways, which has access to syscalls any many other dangerous stuff anyways. You need to trust your employees anyways and if you can’t trust them with not dumping the staging environment in purpose, you should not have hired them. It is possible to include harmless looking code which can be malicious and still be considered a bug if found during code review’s.
Instead just invoke something outside of Jenkins groovy framework like maven, make, etc. Makes you much more portable and upgradable.
Jenkins should just be your execution/workflow environment not a dev framework.
Also never mutate the worker boxes, everything should be completely isloated from itself and each other.
Jenkins is like any other software architecture.
Finally doing too much in Jenkins makes it impossible to test Jenkins locally, which means you can neither debug builds locally without pushing new code to build, nor can you test alterations to Jenkins itself locally which makes it much much more fragile.
How do I know? I'm a sr dev coming in and fixing a mid sized engineering org (400+ devs) jenkins 4 years of accretions. These are major issues we've identified in just a week of looking.
Build systems should be simple, scalaboe, agile, and composable, not monolithic. Keep the monolithic to the individual builds not cross company.
If you would have used Jenkins pipelines, you would know that you can debug it directly from the UI. I can make my builds hard to run locally on configuration-based systems as well, this is not really an argument against Pipelines.
> How do I know? I'm a sr dev coming in and fixing a mid sized engineering org (400+ devs) jenkins 4 years of accretions.
You haven't used Jenkins Pipelines though, if you would have used it, you would know about the debugging part I mentioned above.
> Build systems should be simple, scalaboe, agile, and composable, not monolithic. Keep the monolithic to the individual builds not cross company.
These have nothing to do with the actual topic, these are just generalisations.
> The point is that you get mu hich better reuse by not allowing arbitrary groovy and building up interface contracts.
You can reuse the groovy code you write, which means you actually get more reuse.
> This is exactly like any other coding. Using groovy is akin to allowing arbitrary SQL from your web devs intead of apis (from an interface pov not security).
That is a bad analogy, a correct one would be something like: writing NodeJS build scripts in JavaScript instead of using Grunt or Gulp, or Groovy vs. Maven/Ant.
> Jenkins should just be your execution/workflow environment not a dev framework.
What is a "dev framework" in your mind?
> Also never mutate the worker boxes, everything should be completely isloated from itself and each other.
What does that have to do with using a programming language instead of a config DSL?
> Jenkins is like any other software architecture.
Fail to see how that vague statement supports your argument.
I have used jenkins pipelines. However I didnt use it from scratch so no, I couldn't do what you said due to all of the other issues. My point is that it does let you hang yourself, I prefer the config based systems as it's harder to hang yourself.
Your NodeJs anology is more apt but who cares you got the point. This is a discussion not a literature critique.
A dev framework to me is the type of stuff you've built up around a typical compile of a given "stack". For instance, doing ruby on rails, you're likely to have a bunch of build tooling you've built up to make that easier. I would prefer to distribute that stuff more generically and unrelated to jenkins vs some other build system, as opposed to building it all into jenkins.
As for mutating workers... you seem to be under the mistaken notion that I'm discussing the one thing you are discussing. But in general, if you have a turing complete language with complete system access like you do in the groovy, you can do anything and anything isn't always a great idea. I like the config based systems as it doesn't allow you to do anything, it allows you to do things we've deemed as good for building software. Jenkins boxes end up being shared resources in a large company so it's very difficult to audit what everyone is doing and side effects they are causing.
>Fail to see how that vague statement supports your argument.
Ok
>You can reuse the groovy code you write, which means you actually get more reuse.
You can but most people don't know how to or dont. It's a break from your typical dev loop and it's yet another framework to learn. I don't think it's effective unless you dive in and most people don't dive in. So to me it's a bad model. This is opinion.
Even worse than Apache Groovy being an imperative DSL (compared to Travis ci and Amazon pipelines being all declarative) is that the most declarative portions of Groovy (i.e. its collections API) are disabled when used in the Jenkins pipelines.
If you don't buy expertise in the short term, it may cost you a lot more in the long run.
At my current job I started as a SRE for the team supporting Jenkins and other CI/CD tools for the rest of engineering (around 50+ other teams), I moved back to a SWE position but as an insight: don't run a single master Jenkins for all of your teams, make it federated (per-team/project or whatever suits your organisational structure) AND do all of Jenkins' configuration through scripts.
Never rely on backing up the old configuration (problematic in cases of rollback to previous Jenkins versions, brittle for recovery), just start it fresh every time you need to recover and re-run your provisioning scripts, that also includes NEVER maintaining jobs through the UI, only through JobDSL, JJB or Pipeline scripts.
It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like becoming an archeologist).
Gosh, can't someone just build a lightweight ci/cd system ontop of e.g. k8s jobs?
Or should we just give up and do managed ci/cd?
While I don’t think Drone currently supports k8s job, it is very lightweight in comparison to Jenkins. Most of our developers would agree that it isn’t a perfect product, but it has reduced friction points by a large amount.
It's declarative and everything needed to bring up a pipeline can be stored in git alongside the software it supports. For an example pipeline definition, check out the one for bosh-gcscli[1] and its accompanying tasks[2].
Now, there is the downside that upkeep is a hassle because it requires buying into the BOSH ecosystem to deploy Concourse. That pays for itself by being explicitly anti-brittle, with support for migrations and canary rollouts builtin.
Disclaimer: I worked with Concourse at my previous position but currently have no financial stake in its success.
[0] https://concourse.ci/ [1] https://github.com/cloudfoundry/bosh-gcscli/blob/c908176/ci/... [2] https://github.com/cloudfoundry/bosh-gcscli/tree/30316d6/ci/...
It’s really nice, but expensive if you exceed the limits for the free version (3 agents and 100 build configurations). Luckily we are a fair way off that so can go along fine without having to pay yet.
* only use Jenkins stable releases
* snapshot the system before updating or installing anything
* make regular backups of your Jenkins server
* never install a plugin unless it's actually neccessary
* always update Jenkins before installing any new plugins as otherwise you may end up with an untested mishmash of old and new plugins
I like having control over my CI, but it does seem a lot harder than it should be.
On setting up Jenkins Job Builder using a 'seed job'. This setup allows you to add a .jenkins directory to you project, define a yaml based jenkins job inside that directory, push to github, and Jenkins will create the job you defined - NO web UI required.
In other words you make Jenkins behave like Travis.
Set this up with your first job being a job to backup Jenkins to S3 and things aren't as bad to manage.
https://git.sr.ht/~sircmpwn/wlroots/tree/.build.yml
More complex example:
https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml
Builds run with full virtualization (KVM) on any number of distributed (optional trust) build runners. The whole thing is open source:
https://git.sr.ht/~sircmpwn/builds.sr.ht
It's also self-hosting, it builds and deploys itself and the rest of the *.sr.ht network:
If anyone wants to give it a try, please get in touch (email in my profile) and I'll hook you up with an account on my hosted instance. I could use more people to kick the wheels and help me find the pain points.
https://www.youtube.com/watch?v=GkflqV6KenM
I'd love to hear more about what problems any of you are facing with Jenkins so I can be sure to address them in the work I'm doing.
Jon
People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well.
That's still an option with pipelines, but it feels much more discouraged (to e.g. write a shared pipeline job that runs a multi-line shell script).
It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen (as opposed to 20 jobs that did the same thing for 200 repos).
Unfortunately, it also seems like declarative pipeline is limited enough that we end up writing a lot of Groovy. I sorta get why Groovy became the official scripting language, but it's like pulling teeth getting people to learn even basic Groovy.
I realize, again, that a lot of this is a cultural problem. Mostly just trying to give a counterpoint that pipelines don't always look as clean as the article implies, especially for orgs with a lot of repos (where committing the same 'test.sh' file to each repo doesn't make a lot of sense).
But inevitably I end up writing that shared groovy code (because I have the most familiarity), and so when things go wrong, or it needs to be changed, I end up being the one having to debug it. Definitely a huge bus factor.
It'd be nice if it was a slightly more common language (e.g. Python) so it wasn't such a pain.
Making a Python language plugin wouldn't be hard though.
I never look at the main screen. You can create tabs that segment those 200 jobs. If your job names are consistent (as they should), you can do the segmenting by regular expression (I’m not sure whether creating tabs requires a plug-in)
As to doing (almost) the same thing for different projects: you can POST XML job definitions to the web interface, so if you have the rights to create jobs and to run code on your local system, you can script job creation. In my experience, that’s the way to keep job definitions consistent. To figure out what the XML to POST should look like, grab it from your browser by GETting job/jobname/config.xml (https://support.cloudbees.com/hc/en-us/articles/218353308-Ho...)
Yes, that duplicates lots of stuff in the Jenkins job definitions, but you shouldn’t treat that as source, but as the output of your job creation scripts.
I also don’t like having to use yet another language (or two, counting Groovy and the Job DSL separately), but that, I could live with.
Also, when you say that your developers don't understand groovy, you could teach them how to ding make targets inside of their docker containers. That's the pattern we use at DoorDash... very minimal Jenkins scripting which exists to ding make targets inside Docker images.
Also, regarding logs the site says this:
"Diagnose problems instantly and say good-riddance to endlessly scanning through logs."
Won't believe till I try it but it's a good goal!