Hacker News Reader

Comment by jpillora | Hacker News Reader

The problem with git ops only manifests after it’s become a standard within an org:

With 1 git ops pipeline, it’s fine, it’s the human merge gate, it’s doing its job protecting downstream

With multiple git ops pipelines however, they start to get in the way of progress - especially when they need to be joined in series

The better approach is to build API-first then optionally, add an API client into your git pipeline

View on HN

For me, the key trait that makes gitops valuable is that it is declarative rather than imperative. The state of the git repo is the desired state of the system.

The key point there is that we don't have to worry about the existing state of the system. We don't have to worry that the api call made in your git pipeline actually failed, or that something else changed the system before your api call, and your state has drifted.

You can't replicate that by adding an API client to your git pipeline.

I am not sure how you end up with multiple git ops pipelines; ideally, you shouldn't be having multiple gitops pipelines. You should have a git repo that is defining the state, and then some reconciliation system that is checking the state of the system and the state of the repo, and taking any corrective actions needed to make the system state match the git state.

> With multiple git ops pipelines however, they start to get in the way of progress - especially when they need to be joined in series

Definitely, that's why systems like Zuul exist.

They're esoteric and require a lot of engineering discipline in patience- but in my experience most people who reach for gitops aren't doing it for a sense of "everything as code" (for the audibility and theoretical reproducibility of it) it's because they think it will allow them to go faster; and a tool like Zuul is hard to learn and will intentionally slow you down.

Because slow is smooth, and smooth is fast.

Would you mind elaborating on this more, describing the differences and how tools like Zuul introduce degrees of friction that result in smooth operation and pipelines?

I know my phrasing may come off wrong, I apologize for that. But I'm asking genuinely; I've only ever seen Zuul in the wild in the Red Hat and OpenStack ecosystems.

Right, so Zuul is properly interesting if you're dealing with multi-repo setups and want to test changes across them before they merge; that's the key bit that something like GitLab CI doesn't really do.

The main thing with Zuul is speculative execution. Say you've got a queue of patches waiting to merge across different repos. Zuul will optimistically test each patch as if all the patches ahead of it in the queue have already merged.

So if you've got patches A, B, and C queued up, Zuul tests:

* A on its own

* B with A already applied

* C with both A and B applied

If something fails, Zuul rewinds and retests without the failing patch. This means you're not waiting for A to fully merge before you can even start testing B - massive time saver when you've got lots of changes flowing through. With GitLab CI, you're basically testing each MR in isolation against the current state of the target branch. If you've got interdependent changes across repos, you end up with this annoying pattern:

* Merge change in repo A

* Wait for it to land

* Now test change in repo B that depends on it

* Merge that

* Now test change in repo C...

It's serial and slow, and you find out about problems late. If change C reveals an issue with change A, you've already merged A ages ago.

Zuul also has this concept of cross-repo dependencies built in. You can explicitly say "this patch in repo A depends on that patch in repo B" and Zuul will test them together. GitLab CI can sort of hack this together with trigger pipelines and artifacts, but it's not the same thing... you're still not getting that speculative testing across the dependency tree.

The trade-off is that Zuul is significantly more complex to set up and run. It's designed for the OpenStack-style workflow where you've got dozens of repos and hundreds of patches in flight. For a single repo or even a handful of loosely-coupled repos, GitLab CI (and it's ilk) is probably fine and much simpler. But once you hit that multi-repo, high-velocity scenario, Zuul starts to make proper sense. Yet nobodies using it except hardcore foundational infrastructure providers.

> Right, so Zuul is properly interesting if you're dealing with multi-repo setups and want to test changes across them before they merge; that's the key bit that something like GitLab CI doesn't really do.

I'm not sure about that. Even when we ignore plain old commits pushed by pipeline jobs, GitLab does support multi-project pipelines.

https://docs.gitlab.com/ci/pipelines/downstream_pipelines/#m...

I'm aware of that functionality.

GitLab's multi-project pipelines trigger downstream jobs, but you're still testing each MR against the current merged state of dependencies.

Zuul's whole thing is testing unmerged changes together.

You've got MR A in repo 1, MR B in repo 2 that needs A, and MR C in repo 3 that needs B... all unmerged. Zuul lets you declare these dependencies and tests A+B+C as a unit before anything merges. Plus it speculatively applies queued changes so you're not serialising the whole lot.

GitLab has the mechanism to connect repos, but not the workflow for testing a DAG of unmerged interdependent changes. You'd need to manually coordinate checking out specific MR branches together, which is exactly the faff Zuul sorts out.

Reply on news.ycombinator.com