Hacker News Reader

Comment by theamk | Hacker News Reader

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/

That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD system out there except Jenkins and Buildkite, and both of them a pretty horrible at it unless you spend lots of time with custom scripts. At some point it will become easier to rewrite entire system rather than try to fight existing one.

View on HN

There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself.

Also noteworthy: Slack is going to explain this in more depth next month:

https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...

Or you can read the article: https://slack.engineering/remote-development-at-slack/

This is possible with github actions. Have one job create a matrix of other jobs for each of your runners.

Use self hosted runners if you are worried about costs or have special requirements.

There are limits on build queue sizes unless you buy the enterprise plan. Also doesn't work if you are not hosting code on GitHub

Test build is dozens of gigabytes and millions of files and takes 20+ minutes.

When setting up github actions matrix, we have a choice of either (1) build before every test (+20 minutes per test) or (2) making a huge docker image and upload it, and download it for each worker, which also is 20+ minutes or more.

You can use github actions if you just use github to allocate nodes, and do all the work with your custom test scheduler, but then you are starting on the road of rewriting the CI/CD for yourself.

Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful.

I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.

I would think running something within a Docker container is running it natively?

How do you build a container from inside a container? Answering this gives you one example of a difference

Test build is dozens of gigabytes and millions of files (yeah, we have some suboptimality in how we set up bazel...), so even simple "rm -rf" can take 30+ minutes. Docker really has too high overhead for this, especially in the cleanup stage.

Reply on news.ycombinator.com