Ask HN: Why do some companies build their own CICD orchestrators? | Hacker News Reader

Ask HN: Why do some companies build their own CICD orchestrators?

14 pointsdev_016 comments

I saw Slack has something called Checkpoint

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/

That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD system out there except Jenkins and Buildkite, and both of them a pretty horrible at it unless you spend lots of time with custom scripts. At some point it will become easier to rewrite entire system rather than try to fight existing one.

I would think running something within a Docker container is running it natively?

How do you build a container from inside a container? Answering this gives you one example of a difference

Test build is dozens of gigabytes and millions of files (yeah, we have some suboptimality in how we set up bazel...), so even simple "rm -rf" can take 30+ minutes. Docker really has too high overhead for this, especially in the cleanup stage.

This is possible with github actions. Have one job create a matrix of other jobs for each of your runners.

Use self hosted runners if you are worried about costs or have special requirements.

There are limits on build queue sizes unless you buy the enterprise plan. Also doesn't work if you are not hosting code on GitHub

Test build is dozens of gigabytes and millions of files and takes 20+ minutes.

When setting up github actions matrix, we have a choice of either (1) build before every test (+20 minutes per test) or (2) making a huge docker image and upload it, and download it for each worker, which also is 20+ minutes or more.

You can use github actions if you just use github to allocate nodes, and do all the work with your custom test scheduler, but then you are starting on the road of rewriting the CI/CD for yourself.

Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful.

I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.

There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself.

Also noteworthy: Slack is going to explain this in more depth next month:

https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...

Or you can read the article: https://slack.engineering/remote-development-at-slack/

Maybe the question is really "why do some companies build a new in-house technology rather than use an existing one"?

1. Some companies are so slow and some teams so agile that creating a new working project takes less time than submitting a ticket for permission to install and demo software and get the required permission to purchase it. 2. Developers hate to deal with salespeople, so unless the CICD platform is OpenSource / free the developer might choose to build something simple. 3. Developers might be unaware that a solution exists to solve their specific problem. 4. The commercial solution to a small developer's problem might be too expensive.

The developer estimates of time and money for building something in-house often neglect the "hidden costs" ... (including the long tail). The hidden costs often include : 1. opportunity costs - a developer can't do two things at once, so if they are writing a tool, they are not releasing new features or fixing defects. 2. Maintenance costs - keeping the thing running (when the host OS is upgraded, the network changes, the repository credentials expire, etc) 3. support costs - if this touches any other outside teams or customers, someone will have issues and need support. Maybe a new team has to be created to handle the Zendesk support tickets, provide Slack help, etc.

CICD can get out of hand pretty quickly. If an off-the-shelf solution works for you with minimal changes to your code and workflow then that's fantastic. Ofttimes they're simultaneously clunky and full of unnecessary (to you) features while not supporting the few things you care about. Writing a general purpose CICD probably isn't a good idea for most organizations IMO, but writing a CICD that handles just your use cases is super easy and can take less time than writing the workarounds necessary otherwise. Plus, when things inevitably go wrong the code is right there for you to fix.

The second you notice your engineers are writing anything like custom batching logic to get around CICD limits, custom change detection because the general-purpose solution can't appropriately recognize transitive dependencies, ways out of the CICD sandbox to interact with some raw hardware device with vendored blobs that only work on XP, or anything else fancy then that's a strong indication that you would have been better off doing it yourself from the start.

I built one for a client. It was a rocketchat bot taking commands for making a release of a specific tag or latest master branch, a revert, showing what is currently deployed and printing a commit log vs previously deployed commit. Worked like a charm. Was a set of ansible scripts called by node. No need for anything fancy.

Oh and it always expected a confirmation.

There's an article here explaining one reason why they put Checkpoint in front of Jenkins: https://slack.engineering/circuit-breakers/

It has been tried, for example making diagrams in Rational Rose and generating C++ code from that. Then I assume filling in the code "details" manually. Probably someone else can comment on how that has worked.

That was before agile or TDD were such big things.

I was around when UML took off - but the only real success I have seen from automatically generating code (from diagrams or otherwise) has been in the API space recently since those contracts are easy to describe and generate with tools like Swagger

Reply on news.ycombinator.com