Uber Halved Go Monorepo CI Build Time

eng.uber.com

19 points·lox··2 comments

An interesting problem. I work at a much smaller tech company with perhaps 50 people commuting to a monorepo where we also use merge trains. I always thought such an approach wouldn’t have been possible on Uber’s scale.

We have a similar process where each merge to main runs the relevant CI pipeline again via GitLab merge trains, which effectively takes the head of main, rebases it onto your branch, then runs CI on it, and only merges the change if CI still passes on the current head. It means that merges are queued up, but it can be frustrating if you’re stuck behind commits to projects with slow pipelines (someone in our frontend team went wild with their CI setup).

We did this because we found that the CI could succeed on the branch but fail once merged into main if someone had made changes to similar areas of code, which led to a loss of productivity for everyone when they rebased main and their pipeline failed for something irrelevant to their changes.

It’s an interesting trade off, I’m sure a lot of people wouldn’t mind occasionally breaking main in exchange for more agility (if you rebase from main before committing you’re unlikely to have breaking changes, saving you from a queue, but there’s still some possibility of it happening), but I’m sure many people would be adamant that we should do our best to avoid breakages in main.

The problem of rebasing off a broken main can be kinda avoided by keeping a separate "last-built/main" branch around that is automatically updated. People have to be quick about fixing main when it breaks though, else they get out of sync.

Reply on news.ycombinator.com