New UX for Jenkins: Blue Ocean 1.0
jenkins.io
jenkins.io
This is just the first release of what we hope to be many more in the coming weeks and months. The surface area of Jenkins is _huge_ and we may not have all your use-cases covered - please send us your feedback and feature requests by signing up to https://issues.jenkins-ci.org and submit a new issue under the 'blueocean-plugin' component.
A few things that are coming up soon:
- Support for Github Enterprise
- Full read/write from the Visual Pipeline Editor for any Git repository (Github is supported today!)
- Visual Pipeline Editor feature parity with Declarative Pipeline
We want to run one job to build things, then after we deploy, we want to trigger another job that runs a bunch of integration, performance, etc. tests against the new code.
We decided on creating a new job of type pipeline, and pointing it at a script named `Jenkinsfile.deploy`. It has a few parameters, namely `BRANCH`, that can be manually set, or passed in from another job. We can then move our deployment steps from the main Jenkinsfile into this new one, and still version control the lot.
Major caveat is that the deploy step is not compatible with organisation scans (multibranch/pr), but that's fine for our use case. We usually want to trigger our builds manually based on the branch.
https://gist.github.com/HRMPW/9b2a3ccbdc370e0a7c9cb541be229d...
This is a Job-DSL seed job written in Declarative Pipeline that pulls in your Organization based on a build parameter and creates two jobs for every repo:
1. A multibranch Pipeline job that is based on the Jenkinsfile that will work across all branches and PRs.
2. A deploy job for the repo that is defined in a "jenkinsfile.deploy" file in the repo. This isn't a multibranch job but can be run on any branch based on a parameter for the branch.
This is a very simplistic implementation but could be expanded on further.
Referring to this example https://gist.github.com/i386/5bacc640574c6d79bb72cd9a1181a50... if I have a branch called 'feature/cool-feature' then only the stages Build and Test would be run. If the branch is called 'master', all stages in the Pipeline would be run.
'when' also supports environment variables and expressions and even multiple `when` blocks - see the docs at https://jenkins.io/doc/book/pipeline/syntax/#when
Do you want everyone of these jobs in every branch that you create? In Pull Requests? Do you want all of these jobs to automatically trigger on every push the repo?
It's possible to do all of these steps in a single Pipeline but I'm curious why you would want to separate it?
With everything in one file, we would have to add the logic to our Jenkins pipeline to handle when to trigger deployments (which is currently done through a service we built that looks at GitHub commit statuses of a certain prefix/listens to Slack slash commands).
It has improved a lot since I last tried it. Although my team is moving towards Travis for everything except server deployments. Pains me to say since I especially appreciate self-hosting, but Jenkins is still massively behind in everything outside of Blue Ocean. :/
For lots of things that are now basic in all CI/CD systems (email/slack/irc notifications, github login, etc), Jenkins still requires external plugins and/or a ton of setup from the really clunky UI. It's too much :(
What's the plan to resolve this? Is blue ocean going to spread to the other areas of Jenkins?
One thing to note about Blue Ocean is that as we bring in features (For example, better slack and HipChat notifications) we will be making the required plugins dependencies of Blue Ocean. We don't want you to have to think about plugins unless you are going outside our curated experience.
You don't know the pain we suffer from this plugin ecosystem. Installing a bunch of plugins which depends on a bunch of other stuff, there's no real dependency management for this and it's thoroughly painful when one of the teams decide to upgrade one of their plugins and break their whole setup because of it.
It has happened before even with basic plugins like "Git", one upgrade completely broke more than 50% of our instances (we are running about 50 instances right now) because a lot of other plugins relied on it and then needed to be upgraded, these upgrades then broke other plugins which depended on them and this started a domino effect.
Also, some plugins are just released and never taken care of, we have to maintain about 10 forks of plugins because there's no curation about them.
We came to the point where Jenkins flexibility by itself is a liability.
I know that Jenkins is wonderful for smaller teams but there's a point where the lack of governance really hurts and it's the main burden about it.
But of course, that's a given for any flexibility, too much of it and you have to design boundaries to contain problems... I'm just ranting because after this stint I'm burned about Jenkins, to the point where I don't want ever in my life to have to work with or use it.
Our current discussion internally is about where to set their boundaries, remove administrative access and curate a set of plugins we are comfortable with maintaining (and that makes sense for most of the teams).
That problem is called Java package management, Maven just plainly sucks compared to composer and npm.
At my company we solved the problem relatively easy though: a Docker image that compiles in all the plugin jars (it downloads them via mvn). Upgrading is usually easy (you just write the new version numbers from the "Available updates" screen into the pom.xml), but sometimes does takes one or two hours of wrestling when you have plugins that introduce new dependencies (which mvn cannot detect).
It's like kernel panics: if an application causes the kernel to panic, then the more fundamental bug is in the kernel, not the application. If upgrading a plugin through the UI can cause other plugins to break because of mismanaged dependencies, then that's not really a bug in the plugin, but in the plugin management scheme.
That Jenkins does not forbid administrators from upgrading through the UI plugins which are fixed dependencies of other plugins without at least a warning message about the possible damage to the instance, is a critical defect, one that has gone unfixed practically since the creation of the tool. And so it is a defect which should immediately disqualify the tool from consideration in any large environment where the need to maintain critical updates (security etc.) while keeping uptime high is crucial.
I assume that the teams are still using the pre-pipeline jenkins jobs where they have to make the changes on the Jenkins server directly right? If you could switch to Jenkins pipelines with a single master, you could give that same freedom to your dev teams to manage their own build without having to manage a dedicated Jenkins master for each one.
We have more than 500 engineers, it's seriously impossible to run a single master in this environment. We run both pipeline jobs and JobDSL (and some teams still rely on JJB).
There's nothing in Jenkinsfile/Pipeline that would help to mitigate any of the problems of scaling a master to hundreds of developers and tens of teams.
The dev teams all have control of their jobs through source control, and already had before the federation of instances.
We've made a curated distribution of key integrations and plugins for Jenkins and this distribution protected our users from the breaking upgrades related to SCM API 2.0, which includes the Git plugin upgrade you mentioned here.
If you're interested, send me an email (tkennedy@cloudbees.com) and I'd be happy to talk to you more about the problems you're experiencing with the plugin ecosystem.
Community plugins can be a godsend, but they can also be the exact reason why a software service falls apart.
The big ones: * Many don't work with pipelines (scm sync config plugin for example). And you don't know this until you try * Upgrading a plugin can and often does have unintended, and hard to detect problems (only way to know is to run every job and find the failures) * It's hard to automate plugin installation
Does the 1.0 release offer much that wasn't in the previous version?
Edit: Answering my own question. No, the 1.0rc4 -> 1.0 is just a version bump.
We've shipping a lot of bug fixes in the last few releases and the integration of the editor into the Pipeline creation process when you choose Github, which landed in rc1. I recommend upgrading your Blue Ocean ASAP :)
Jenkins: 2.46.1 Blue Ocean: 1.0.0
I click on "New Pipeline", enter the GitHub token, select organization, select repository (without Jenkinsfile in it) and then after a while this message shows up:
-- Pipeline Creation Pending... Pipelines are still waiting to be created. You may now return to the Dashboard to check for new pipelines. --
I don't see any errors in the log file or in the browser console.
Any insights?
EDIT: In the traditional UI, I see a new job was created for my GitHub organization but it has no repositories (probably due to missing Jenkinsfile?)
I had to click the "Show more" button on Blue Ocean 10x until I reached the end of my job list (when I was looking for this new pipeline that would have been created).
If you delete the multibranch project and try again, does it work the second time? It would be good if you could capture a HAR file [2] of your browser session too.
[1] jdumay@cloudbees.com
[2] https://support.cloudbees.com/hc/en-us/articles/204498690-Di...
Deleting it and following the "New Pipeline" flow again has the same result.
There are other minor issues I've come across, but since this one is extremely annoying I thought I'd mention it here.
You can watch this ticket for updates https://issues.jenkins-ci.org/browse/JENKINS-38523
Clicking through the gui to create jobs is error-prone, so I'm using jenkins job builder at the moment to create those, but it would be great to just use the one file in tree.
Just in general, the documentation for a 1.0 release seems really insufficient.
There are a few major things we need to get done with the visual editor before it could be called 1.0; the backlog in the Jenkins JIRA has much of the plan laid out, but not all of the design done to see.
Right now, the visual editor allows editing _declarative_ pipelines and only allows saving and loading from Github. Support will, of course, be added for the rest of the branch sources but that wasn't ready for the 1.0 Blue Ocean release.
In order to use it today, you'll need to set up a Github repository as a branch source and you'll get edit buttons on the branches tab as well as top of the run details.
From the pictures the new jenkins looks much more pleasant on the eyes but how would you rank it's competitiveness with Travis when it comes to getting started with your testing the first time?
Will reconsider Jenkins in my next project as it seems like you are taking UX seriously!
* If i use non core parameter in pipeline, BO ask me to move to classic interface
* If i do not use Github like pipeline, interface become partially useless. It means empty branches, empty PR columns etc. Where is good old custom pipelines?
But anyway it is good move forward!
PS: pipelines itself also has some issues, like really small number of useful steps, literally only sh step used everytime. Also in case of custom pipeline it is still unclear how to abort build in code. And using groovy collections methods is veyr inconvinient (like trying to use .collect to create command like options for next sh step)
It looks like Blue Ocean generates Apache Groovy DSL code for the pipeline. From their pipeline doco [1]: _In order to provide durability, which means that running Pipelines can survive a restart of the Jenkins master, Scripted Pipeline must serialize data back to the master. Due to this design requirement, some Groovy idioms such as collection.each { item -> /* perform operation */ } are not fully supported_
I wonder how feasible it is to make any edits at all to the DSL and have the new code read in successfully by the Blue Ocean UX. Perhaps the collections methods aren't the only restrictions.
[1] https://jenkins.io/doc/book/pipeline/syntax/#differences-fro...
Do you plan on supporting Bitbucket Server like you are with Github Enterprise?
Yes, we do plan on supporting Bitbucket Cloud and Server but I can't give an ETA on that right now.
I still think Jenkins is a pretty huge beast to run and configure. We're trying to keep things light (install in a couple minutes with `pip install`) and intuitive (simple YAML files to describe pipelines).
We're writing a post about our take on automation/continuous delivery and how this impacted some of our design decisions for Pipelines. I'll probably dive a bit more in this version of Blue Ocean before so.
I haven't set up an in-house CI server for a long time, and if ever needed one again, I was planning to evaluate GitLab CI. Otherwise I just use CircleCI, and TravisCI for open source.
But with this new UX, I might give Jenkins another shot.
IMO a build server should only be managing it's triggers (time, checkin) and then calling the build script (make or whatever) with the correct parameters for that build configuration. Instead it seems to be trying to take the role of both.
Instead, we now use Jenkinsfiles everywhere, or pipeline scripts where we have scheduled stuff. That's working well enough so far for our small team.
The things Jenkins provides can be difficult to find elsewhere. For example, we spin up an EC2 slave for most builds, but sometimes need to then take the result and deploy it from another (static) node because of network restrictions. Stuff like git access, the credential store, moving stuff between nodes just works everywhere.
Instead, we now have a very basic Ubuntu-based AMI with Docker installed, and otherwise stopped depending on the host environment as much as possible. Jenkins spins up an EC2 instance when needed, and shuts it down again after some period of inactivity. 95% of our pipeline scripts do work inside a container.
It also switches your default GitHub links to Blue Ocean. If you only want to try it out, you'll have to tell your users to switch their Notification URL back to Jenkins Classic in their user configuration page. (If there's a way to do this for everyone, I'd love to hear it.)
I do see the option in my user settings page but I don't understand what it's for. What is this "notifications URL"?
[1] https://jenkins.io/doc/book/blueocean/getting-started/#switc...
As for the Github links, as you mentioned a user can set their preferred UI (Blue Ocean or "Classic") in their user preferences easily. We do understand that this might be difficult for some administrators to swallow and are looking at a global option to turn that behaviour on/off [1]
I hate to use HN for bug reports, but since you asked about roadblocks (thank you for asking!):
For one, we use build triggers to ensure downstream dependencies are tested as well as the current project. I would have expected some kind of pipeline integration between those in BO (displaying the triggered build pipeline in the parent pipeline as a whole), but instead BO actually makes it more difficult to get to downstream builds.
Classic:
https://jzila.keybase.pub/classic_downstream_build.png
https://jzila.keybase.pub/classic_downstream_build_detail.pn...
Blue Ocean:
https://jzila.keybase.pub/bo_downstream_build.png
Note that in BO it isn't clickable, so I have to either manually type the URL or go back to Classic.
Some display bugs in the pipeline view:
- Issues with display of pipeline nodes in running builds that make what's happening pretty confusing: https://jzila.keybase.pub/running_build_display_bug.png
- Issues with display of pipeline nodes in finished builds (falsely reporting node failure): https://jzila.keybase.pub/finished_build_display_bug.png
Minor grievances:
- Nodes run in parallel don't have their hierarchy preserved: e.g. parallel(a: {}, b: parallel(c: {}, d: {})) all get collapsed into a, b, c, d parallel nodes in the BO pipeline view (as you can see in the bo_downstream_build screenshot above).
My contact info is at https://keybase.io/jzila if you need more details.
Minca, gi' fiada ora.
Then, in english: Wow! About time! Kudos to the whole team!
The UX and UI is by far my least favorite thing in Jenkins, so for basic usage Blue Ocean is miles ahead, even though it doesn't nearly have the feature parity of the old UI. That said, I cannot wait for Blue Ocean to grow and get better.
Uuuuuugh. It looks nice, but all my colleagues whine about how poorly it works. Then comes all the problems caused by regular Jenkins multibranch pipeline limitations...
:-/
Jenkins does allow 'parallel' jobs but that is the same thing we want.