Hacker News Reader

If you're putting lots of logic into ANY CI server you're doing it wrong!

Script your build so that you can run it with a single command on any build server or any developer machine. Then have the build server invoke said scripts (with arguments for any build-server specific paths etc).

Having spent a week fixing problems with inter-plugin dependencies I do agree with your point - Jenkins just isn't very good, like a lot of the free software from that generation (FitNess etc).

Absolutely, tasks/system commands should always be stored in VCS as a shell script, so that feature branches can exercise changes in the task itself. Unfortunately theres the other 90% to worry about, and thats where the highest complexity is, such as configuring a build chains, access control, reporting, etc.

Glad to see jenkins is moving towards a travis-ci ".travis.yml" kind of format, but I'm not hopeful to see it catch up to anywhere near TeamCity, even Bamboo in user or developer UX.

Could use something like this if you are using github https://github.com/groupon/DotCi Job configurations can be version controlled and reviewed like everything else.

For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin

JJB is also a nice tool that seems like it allows similar functionality (I haven't worked with DotCi, so I can't say how they compare)

http://docs.openstack.org/infra/jenkins-job-builder/

Well...

Jenkins 2.0 will provide support for build chains via the pipeline plugin which will be included as a core plugin.

Access control can be configured quite nicely with the Role Strategy Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Pl... + AD/LDAP integration.

Reporting automation is covered by the pipeline plugin. Custom reporting is quite easy to implement on top of Jenkins, its APIs cover almost everything you need.

You have simple problems to suggest simple solutions. CI logic can gets complicated:

What about cascade builds? The pipeline can't be handled in your build script.

What about reporting? (list of compiler warnings in the build? List of tests that failed? In general extracting information from the log)

What about efficient email alert? (with blame list, list of changes, etc.)

What about IRC reporting?

Jenkins is not very good (and written in Java :() but the other available free alternatives are just not better.

Huh? I really don't understand your argument. Jenkins is better than scripts because you can do more in Jenkins than in scripts? O_O

You can do a heck of a lot more in scripts; you've got a full language with all of the libraries it comes with in order to do stuff.

We've build our current build pipeline with Python scripts:

0. Bootstrap the build environment (installs everything you need to build). 1. Build projects order, order is derived from the intermodule dependency graph. 2. Execute unit test sets based on build parameter + save results in output. 3. Execute integration tests based on build parameter + save results in output. 4. Build distribution packages + save in output. 5. Deploy distribution packages to server based on parameters. 6. Run UI tests against the deployed packages + save results in output. 7. Publish the bits if they're OK.

Most of these steps are in different scripts, which can be called either from orchestration scripts or one-for-one in Jenkins.

All we have Jenkins do is invoke the script(s) then process and report the results.

We've basically got it wrong with the design of our CI tools; they do too much. They should just provide scheduling, invocation, reporting and dashboards over the build process. Expose that as an API and let scripts integrate with it.

So, let's say you have 8 projects. In each of the 8 projects, you want to run the same static analysis tool. If the violations go up, you want an email sent out. If any critical errors are there, you want the build failed.

In any project, if the build fails, you want the latest commit automatically reverted.

Now, do you just copy and paste to each script, or do you have a chain of build processes and logic in the build system?

I was basically answering "If you're putting lots of logic into ANY CI server you're doing it wrong!".

There is a separation of concern: the CI server logic is a refactoring of things that are common to any project. If your project have scripts that sends email on build failure, I think you're doing something wrong.

You mention at some point "orchestration scripts", this is exactly what Jenkins is supposed to provide help with: orchestration.

The important thing IMO is that the boundaries of responsibilities are clearly defined between what a project does and what the orchestration does.

This is a critical component of releasing production software. I would not expect to see "free" in the conversation here.

There are good reasons to prefer free and open systems, even for "production" software. Those reasons are part of why people choose to use GNU/Linux, nginx, and all the rest of the open infrastructure.

It was not clear whether it was free as in beer or speech. I do understand there are benefits to using open software.

Too often I see engineers bias heavily toward inexpensive, free/cheap, or home-built solutions simply because they personally don't think things should cost money, when the understated anchoring costs of building and maintaining, plus the opportunity cost of valuable people time, might shift the toward other alternatives if a wider business context is taken into consideration.

Yeah, I parsed them as wanting a free system as in one that's open source, but I could be wrong.

If we're talking about hosted solutions, I agree it's mostly silly to refuse to pay... unless we're talking about open source projects, where many CI vendors do have a free tier...

Anyway, I also see an opposite bias toward costly hosted proprietary solutions and a refusal to even think about how one could accomplish one's goal with, say, a couple of shell scripts and cron jobs—or a self-hosted open source system.

That bias can be very sensible, but it can also be a hindrance, especially when it turns out that the proprietary solutions aren't quite what you want anyway, but since you're opposed to writing your own scripts, or running a more flexible system on your own servers, you settle for a suboptimal setup...

This bias I'm talking about is also a way of undervaluing something. It's saying "I'll pay $99/month for some hosted thing, but I'll never spend a week of engineer effort on really getting it right."

It's never the cost for me, it's the mutability, oos I can hack on, and make it do exactly what I want it to do, proprietary code I only ever seem to get somewhat close but never exactly there.

We made the same mistake with Anthill Pro a long time ago. The plugins were of various quality and caused us a lot of headache.

Now our build and test suite is just a script each and it can pretty much on on any CI server.

The biggest thing is that Jenkins is a job engine, not a CI server. Because it CI happens to be nothing more than executing jobs when triggered and then tracking/reporting the results Jenkins gets used for it as well as a number of other things.

If you're looking at Jenkins purely as a CI then absolutely, it's overkill. I've seen Jenkins used as a distributed cron replacement, CI server, pingdom replacement, deployer and even as a means to give non-programmers permission controlled access to execute certain jobs via parameterized builds. Just having granular control to decide how to report on cron failures from a central location is a huge help.

It's ability to combine job execution, detailed result tracking, scheduling, scripting, SSH and failure reporting from a central location with flexible permissions makes it an extremely useful tool.

The extensions that the Cloudbees folks have put together are pretty sharp too.

>If you have a strong developer team, I would suggest using something like python buildbot and programming your CI end-end in a single unified language, much better than shoe-horning plugins and groovy code inside your jenkins textarea fields!

This is obscenely bad advice. We run thousands of builds/day and none of the configuration is maintained inside jenkins. All of our job configurations resides in repository alongside all the code.

TeamCity is cute, but at scale its pricing is... kind of bad. $12k for 50 agents. And it's not that hard to have 50 Jenkins slaves, especially if you're testing on multiple platforms or on different versions for those platforms.

If you have 50 Jenkins slaves, you probably have someone who's job is almost exclusively to babysit Jenkins. At my two previous workplaces, we had entire release teams focused on that, and still Jenkins was a never ending source of problems. And they did everything "right", all the Jenkins scripts and configs were in git etc.

$12K/month for a much (I mean, it's like going from cvs to git) better solution in that situation should be seriously evaluated; perhaps you could give every team in that org their own dashboards and eliminate/distribute the whole release engeering org. $12K/year, which is the actual TeamCity pricing, is a no-brainer, which unfortunately most teams are way too cheap to pay for, headcount looks so much better. Fortunately at least cloud based CIs are starting to seriously take off, hopefully I never have to deal with a home-grown Jenkins deployment again.

Well, from my experience as the person who "babysits Jenkins" unless you appoint an actual owner for Jenkins, you'll get a real tragedy of the commons with it. Devops sounds cool and all but most devs want to write cool code: usually product features, not infrastructure stuff, so everyone will do CI/CD related work half-heartedly.

So in my obviously biased opinion, if your team is big enough, either get an actual build/release engineer/devops guy or appoint one of the devs as an owner for this, otherwise you won't get the best results.

Jenkins compiles jars with Apache Ant, Maven, and checks code out from all the relevant SCMs, which, if you don't need more than that, works perfectly.

Also, arbitrary shell scripts, (as usual) provide duct tape for everything else as needed.

Yes, software is harder to manage (long-term) when the primary interface is a GUI.

However, the new Jenkins 2.0 fixes many of the issues you've described. Build pipelines are a major new feature, as well as the integration of a "Jenkinsfile" which allows configuration of jobs via a proprietary DSL.

the integration of a "Jenkinsfile" which allows configuration of jobs via a proprietary DSL

This makes no sense to me. Only how my project is built should be checked into the project's repo, not the whole CI/CD deployment pipeline! If I start deploying to Azure instead of AWS I need to commit that change to my project? It's nonsense.

We use the pipeline plugin in my job and we use a seperate repo for pretty much the same reason. It's not a detail that's really part of the application (do you keep your chef cookbooks in the same repo?) and not only that but we use the same scripts across multiple apps.

The DSL can pull in other scripts from elsewhere, so you could just have a stub Jenkinsfile in your project, with the bulk of the pipeline config version-controlled in another repo.

And if I change where I keep my pipeline config, I have to commit a change to the project so it grabs it from the new place? Or pass it in as a parameter, so you need to know ahead of time what parameters the DSL will be expecting?

Come on now, we don't need to go to these contortions just to avoid saying that Jenkins is doing it wrong.

In this situation, yes.

You could indeed just use the Jenkinsfile for building a project. You can also specify the DSL config directly in a job, e.g. in a separate deployment job.

What would be the right place to define the deployment config?

In the job itself is fine... unless you want to version control it. This is a historical Jenkins weak-point. The plugins that exist to try to solve the problem are clumsy.

This Jenkinsfile is their solution to the problem. I'm saying it's not a very well thought-out solution, because now they're forcing us to put this thing that's not part of the project in the project, if we want it version-controlled.

So don't use the Jenkins file. I have mine in a separate repo.

Ugh. Not another proprietary DSL.

There is something amiss in the tech world these days. Too much esoterica and too many specialized tools, too many points of failure and dependencies, and too much duct tape. But, mostly, too much reinvention for frequently incremental "gains".

It seems normal, but it's really a mess. Environments are overly complex and it's a wonder anything works half the time.

At some point we will completely jump the shark. From there, my bet is there'll be a powerful movement towards simplicity.

/rant

But all the lisp people always talk about how incredible it is that one can have all sorts of DSLs....

Completely different. The Groovy DSL is a limited subset of Groovy. Lisp DSLs are embedded DSLs. They're libraries that help you write code in what looks like a new language, but the entirety of lisp remains at your disposal.

There are a lot DSLs writtn in Lisp, which don't use Lisp syntax and are not embedded.

being able to config jenkins via code is huge! looking forward to trying it

You can do this today with the Job DSL plugin.

You can do this today with the same Pipeline plugin they are announcing as a major feature for 2.0. It's been available for months.

If we were hypothetically "too big" for TeamCity, with scaling difficulties all over, do you have (or have heard) any recommendations? I hear the biggest companies all roll their own, but is there anything between TeamCity and Google-scale?

Jenkins + generated jobs (i.e. jenkins-job-dsl or the new Pipeline DSL), and keep as much as the build configuration and logic in your project where it belongs. The configuration that lives on the CI server should be pretty minimal.

We have a custom wrapper around the jenkins-job-dsl that encodes our conventions and templates to generate several hundred jobs, and it works very well for our medium-scale instance.

There's http://go.cd if you are looking at something focused on CD (Pipelines etc)

Yes, we've found gocd does a terrific job of allowing you cleanly model your full continuous delivery workflow and organize your servers.

We use jenkins at medium scale. > 400 jobs, 20k-30k builds/day.

Works just fine.

Jenkins works fine with dozens to hundreds of slaves per master, thousands of jobs, and tens of thousands of builds per day. Jenkins isn't perfect for everything, but it can be a very useful task runner. People have adapted it to do almost everything these days, from database maintenances to builds to restarting infrastructure. Given the almost unlimited triggering and reporting, you could use it to build many ETL apps as well (not saying that's a good idea).

As much as Jenkins sucks sometimes, it's what many big shops use for many things. There is tons of community around it. It works, it's battle-tested, and it's been that way for years. Anything else will have a hard time catching up for as many use-cases as Jenkins currently solves for people.

If you hate a part of it, fix it and open a PR :)

What is too big? Too many concurrent jobs? We found the integration with AWS to autolaunch workers to be quite easy to setup.

I tried buildbot, and found that the documentation didn't match observed behaviour, and their insistence on not providing any examples (combined with some in-house jargon) because "everyone is different" made the whole process particularly painful. Buildbot required a lot of attention and time to get working for us, and after all that, wasn't particularly end-user friendly. You're going to get exactly the same '2 people with tribal knowledge' if you use buildbot.