Hacker News Reader

Comment by zug_zug | Hacker News Reader

Since everybody seems to be hating on jenkins so much, I'll speak up. IMO jenkins is one of the most valuable tools at any startup and every good engineer uses either jenkins or something very similar.

There's no comparison between managed/logged/permissioned/distributed jobs that jenkins provides for free vs building an overwrought service or an insufficient crontab. However it's a power-tool and I think a lot of people go in expecting something dead-simple and pretty and are put off by something that they need to invest in learning.

Just for example, in 2 days I built an uptime checker that ran every 60-seconds via jenkins and triggered slack alarms/sms on problem, and it just worked (< 2 days total maintenance), for years. An equivalent service (Pingdom) quoted 10k a year.

View on HN

I've been using jenkins for over 10 years.

There is no finer system for turning quick CI hacks into entrenched technical debt.

Jenkins gives you the freedom to do things without the burden of asking whether you should do those things. Jenkins permits you to munge a whole lot of separate domains into a single, creaky system.

There are better tools available for most aspects of CICD and automated testing these days.

> There are better tools available for most aspects of CICD and automated testing these days.

It's easy to say this and not cite examples.

I rock perforce and windows agents, requirements of the environment, and we struggled very hard to find something that beats Jenkins.

One guy even bought an Octopus Deploy license for a year (at the tune of a few thousand dollary-doos) and it did everything worse than our Jenkins setup.

I actually dislike Jenkins, I'm open to suggestions here.

Jenkins to me is great for prototyping certain process activities in a very arbitrary set of conventions, but it provides a lesson as to why the Java world moved on from tools like Apache Ant (XML-based Make) to Maven (convention over configuration as a tooling value). While 10 years ago such processes were not really standardized nowadays people are used to conventions like GitOps with well documented processes for how artifacts are built, deployed, and operated with container primitives after we've spent a _lot_ of time now trying to manage Jenkins installations that become incredibly brittle over time and become more of a liability than a force multiplier for the organization. So people are adopting more specialized tooling instead of trying to have Jenkins do everything under the sun from building artifacts to deploying them to deploying infrastructure with CD conventions.

Things I've found better than Jenkins once an organization lets go of over-engineered processes:

- TeamCity - Concourse CI - Circle CI - ArgoCD - Tekton

For Windows build agents I've met plenty of people successful with Octopus Deploy that have hated Jenkins and your data point is the first I've seen that went back to Jenkins.

Perhaps it's worth looking at Team Foundation Server / Azure DevOps?

Octopus Deploy and TeamCity are horrific. Jenkins is hilarious better despite all of its flaws. Concourse CI has a higher learning curve than Jenkins (at least for users, not necessarily admins), but once you get it, it's definitely a good contender. CircleCI is great!

Buildkite might be an option in a regulated environment.

They don't get access to your code (BYO runners, they manage pipeline config / logs / notifications). You run their (open source, easy to build yourself) agent on infrastructure you control.

It's really nicely put together.

Buildkite's problem is they only support git, unfortuantely.

https://buildkite.com/docs/agent/v3/hooks indicates you can override the builtin git support with your own checkout system. Sucks having to write a plug-in but it should be short.

Somebody already has, for perforce:

https://github.com/improbable-eng/perforce-buildkite-plugin

I've looked into this plugin before (we also use perforce). This does techincally let you build from perforce, in the same way making `p4 sync` your first build step does.

What are your requirements? The most recent CI/CD system I've used was Concourse, and using it was great - it's very flexible, if a bit barebones. (I didn't have hand in the initial installation for it, so this is only for using it)

https://concourse-ci.org/

Gitlab CI.

I think you missed that I'm using Perforce; gitlabCI would be a super hack job.

I've used Jenkins for a long time, long enough to have used Hudson (Jenkins is a fork of Hudson due to Oracle drama).

The alternatives listed in reply to your thread are all much better than Jenkins, from system design, to usability, to cost of ownership of the personnel needed and tech debt incurred. Here's my own opinionated list, having used or evaluated them before:

1. GitlabCI 2. Circle CI 3. Travis CI 4. Github Actions 5. TeamCity 6. Concourse CI

Jenkins just happen to have been around a long time. It's like Jira -- everyone uses it, but nobody likes it.

Jenkins has an important niche: "overgrown or enterprise projects that were initially created without ci in mind and require a ugly, hacky pipline to build"

I think it still has no rival in that area and it's also why is getting a bad rep.

Almost 7 years ago, we set up our first build pipeline and I was a bit anxious about Jenkins because I simply hated it and it would have been the only piece of Java technology in our stack. Our infra guy basically told me "look, there’s nothing more powerful and widely known out there" (which was pretty much true at the time, at least in open source) and had already set up the VM when Gitlab came out with their CI/CD.

Since we just had set Gitlab up as our source control platform, this was the logical choice and we immediately ripped down the Jenkins server. Since we were still starting out, integration and rewiring was done in just a day.

I‘ve never been so happy playing the early adopter guinea pig. Actually we never had any real issues with it and aren‘t looking back.

For me, powerful equals "footgun included". CI should do one job and do it well imho. Huge bonus points for tight coupling into the source control workflow.

Your mileage may vary.

I think Jenkins is a very powerful tool but I would strongly disagree that it's quite valuable in a startup environment. Jenkins has struck me as a philosophical sibling as C++ - it offers a plethora of foot guns to the user that can also be used to accomplish rather good things. I would advise Jenkins to companies with projects that don't require agile responses to user requirements where the functionality desired at the end is very clear from the beginning - since the times I see Jenkins fail the most are when the configuration of jobs on it are being updated and reconfigured rapidly.

Jenkins tends to suffer from extremely poor state management and experimenting with things can cause a permanent loss of value to companies if backups are not properly configured - compared with a system like ansible where recipes will grow and be regularly committed to a repository Jenkins doesn't have a nature version control system and as a result strikes me as extremely brittle.

I almost wish the UI for jenkins simply didn't exist and it actually was just a whole mess of config files and shell scripts so that it could be locked down to a much better degree.

I don't care to unpack most of that. But what I will say is that I recommend automatically backup up jenkins nightly, I accomplished this simply by commiting jenkins_home to a git repo and pushing every night.

These days you can also use the Configuration as Code plugin and you can automate the setup, version control it, document it, etc: https://www.jenkins.io/projects/jcasc/

But did you try to restore from backup? I did and was flawless and smooth

As someone who runs a large build cluster at a major corp, Jenkins is clunky and outdated. It's a pain to deploy and keep maintained. I'd rather stub my toe than figure out why some event isn't being properly processed in the bowels of Jenkins.

However, when coupled with Kubernetes, Jenkins is more powerful than almost any other CI tool.

You can orchestrate any iteration of build types or actions while easily providing the underlying resources transparently to your users.

Notice that almost every other CI tool mentioned here is either 3rd party hosted or severely limited in some fashion when compared to Jenkins.

If you are already running in kubernetes, how is any other tool severely hampered if it also runs in kubernetes?

It's partially because I'm not as familiar with other tools but they don't expose the ability to interact with an entire pod within the build.

Gitlab Action Services for instance is closer to Docker Compose.

Seems to me people hating on Jenkins are not hating on the idea of a central automation service/task runner, they hate on Jenkins itself.

You could use a tool desgned for it and probably be better of, since upgrading jenkins and plugins, as everyone knows... is not always exactly pain free.

Regarding your batch scheduling solution - take hashicorp nomad for a spin: a single go binary and your scheduled job can be declared in 10 lines of yaml. Won’t miss a beat.

This isn't a batch scheduling problem. It's a monitoring and alerting problem. OP needs a monitoring and alerting solution.

Stop ~~~ engineering ~~~ things and choose the right tool for the job.

"Solution" and "Job" may be overstating things here. If all op needs is monitoring and alerting of a couple of machines, then Datadog/New Relic/Ops Genie/Pager Duty/Server Density/Pingdom/AppDynamics/Loupe/Sysdig/Dynatrace (just to name a few in the crowded space) are all likely overkill and not worth the cost.

A large portion of the cost of many of these tools is spent "choosing the right tool for the job" Figuring out what they do, what they do well, where they overlap, where the company that makes the tool is headed, how hard its going to be to swap that tool out for a better one (or cheaper one) - thats a lot of expense in labor and training.

A script scheduled to test something and notify? Hardly ”engineered”.

I claim using pretty much any tool other than jenkins is easier to maintain over time (as illustrated in this very article).

Don't abuse CI/CD system as workflow automation tool. There are better and much easier alternatives.

Such as?

I'd also like to know. Many of them are <<clunkier>> and have smaller communities, less active development, etc. Far from a clear win.

(crickets)

We don't use Jenkins for CI/CD. It's probably too overkill for those basic tasks.

But as the middle ground between crontabs and something like Airflow it fixed all our problems without creating new ones.

Are there really any alternatives out there? (And no, we don't need Kubernetes, thanks.)

I think the question itself begets more questions. For example, I've seen Jenkins jobs replaced with an Apache Airflow DAG or two, but obviously that's not the right call for continuously deploying applications to K8S clusters, which is where I'd use Tekton, Harness, or Argo CD. If we go way back, companies used to churn out Perl and Ruby scripts and eventually they were replaced with anything from Python to Go to Rust. It always goes back to _why_ someone finds deficiencies in what Jenkins jobs provide. Ask 10 people that hate Jenkins and one is likely to find 10 different products / solutions appropriate for them.

Most folks I've seen replacing Jenkins for a job runner wind up using Ansible, Salt, or Rundeck if they're not in a giant enterprise (there's also the old HP Operations Orchestration system but that's buried now unfortunately). All of these have their own holy wars brewing and deficiencies, sure, but I personally prefer those warts over adding more maintenance issues by adding in Jenkins. I'd also suggest StackStorm for a more modern, async approach to orchestration workflows. The workflow software ecosystem is built around business domain specialization and it's kind of silly to try to go against Jenkins toe to toe as a company, so this is what we've got sadly. After all, Jetbrains makes TeamCity which is essentially a Better Architected than Jenkins system but it's definitely not very popular either.

For us, the Jenkins is:

- Website with access control which supports identity provider.

- Properly authorized user can start a job from their browser, and can specify arguments as well (using checkboxes, input boxes, multi-choice boxes etc...).

- Job works by allocating few machines from the pool and running some shell commands on them. The number of machines, type of machines, and commands to run are all customizable and can vary based on parameters user has entered.

- Once job is running, you can view text logs (in real time) or hit "cancel" button, and it'll properly cancel job and release all the resources. If the job fails for any reason, it sends notifications.

- From web interface, you can view the list of all past jobs, their status, logs, outputs, test results, etc...

- The machines can be physical on-prem ones, or allocated from AWS. For physical machines, all you need is plain linux with ssh access. For AWS machines, machines are created/destroyed in response to load.

- The whole thing needs no infra other than a single master machine which is hosted on premises. As long as you back up the master regularly, you can recover from complete meltdown, the only problem will be that some jobs will get cancelled.

------

Jenkins is actually pretty bad at this task, and we need a fair amount of Groovy code (ugh...) to get it to do what we want.

But there seems to be very little alternatives which can do all that. For example, Apache Airflow is missing authentication for web ui. Tekton seems to require Kubernetes and I (based on reading docs) has no UI controls. etc...

Jenkins can be relegated to simply launching tasks in other systems and having almost no state or configuration in itself then for your needs. One can lock down Airflow access to solely from Jenkins agents or setup an nginx proxy in front of it to protect it from prying eyes.

Jenkins is architecturally an artifact of enterprise software kitchen sinks for the 2000s and the way around it is to cut away at anything one doesn’t need out of Jenkins and to instill discipline in engineers to think about requirements carefully and to avoid hacking more and more into Groovy Jenkins libs than one would spend time actually doing work.

For a lot of what you’re describing Rundeck was built for this common case as essentially a central job portal, and at the least it supports more integration options than Jenkins. It’s difficult to wholeheartedly recommend it still given its surprisingly low rate of adoption. The market now is that for an OSS product a decent web UI with authentication is basically something to charge like it’s an enterprise product, and so lots of start-ups even today will begrudgingly fire up Jenkins in 2021.

I’m of the opinion that the time fighting Jenkins jobs and writing Groovy for common, trivial tasks in most CI systems is not acceptable in a start-up situation where strong focus and minimizing distractions / non-core work is so important for success. The kind of code to define a build doesn’t make sense for orchestration (it’s not just a digraph but about how to react to events and edge transitions) but it’s exactly how Jenkins DSLs work out.

Rundeck comes to mind

What do you have in mind? We have a number of workflow automation tasks on Jenkins and while it works, it requires constant upkeep.

An ability to share machine pool with a CI system, or having an integrated CI is a plus.

Someone, already, mentioned Argo tool family. I have got very good experience coupling argocd and argo workflows. Its not so advanced (from workflow side) as Airflow, but works as pretty good. There is always option to fallback to Rundeck and folks from older times

Reply on news.ycombinator.com