A one-line change decreased our build times by 99%
medium.com
medium.com
But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.
Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there are a lot of stale undeleted branches.
What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanation I can come up with is that the whole build takes hours anyway, otherwise there is no way you wouldn't notice this sooner.
On a single core, Chromium surely takes hours to build.
Though I agree that 40min for the repository in question is highly suspect.
Earlier this year Bruce Dawson had a post indicating that it took about a CPU-day, though coalescing files (“jumbo builds”) significantly reduces build time (we’re talking down to 5h), however that’s at the expense of incremental building, and it constrains the code as you can get symbol collisions between the coalesced files.
On day, someone forgot to brief the foreign intern about the necessity of breaks, intern fixes the issue, pointy-hair-boss gets wind of the news, old crew gets fired, new intern gets promoted and fixes also the Pinterest spam on google images.
A man can dream.
That tends to be the beginners during the onboarding weeks.
It can see things that were there all along, but everyone who has been there has developed a blindness to.
An example would be a company I worked for who ran a pretty standard LAMP setup but had never heard of memcached. Simply adding that reduced the database load by like 90%.
Until you or some other person looks what the code is doing.
It could also be that it was a new hire. I shallow clone a huge monorepo similar in commits/branches and it takes seconds. My experience would instantly tell me that something is worth looking into.
Another reason (which is why we went for ephemeral runners in the first place...) is that if you have stuff that mounts a directory from the repository directory as a volume in a Docker container (e.g. for processing data), you may end up with the Docker container frying permissions in the repo folder (e.g. 0:0 owned files). Now, you can put a cleanup step as part of the CI (=docker run --rm -v $(pwd):/mnt sh -c 'chown -R $runner_uid:$runner_gid)... but unfortunately, Gitlab does not allow a "finally" step that always gets run, so in case the processing fails, the build gets aborted, the server hosting the runner crashes, ... anything happens, the permissions will be fried, and a sysadmin will need to manually intervene.
An ephemeral runner using docker:dind however? It simply gets removed.
It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.
Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step is to delete the node_modules folder, as otherwise it can't be trusted to update correctly.
I think it's partly this, and partly that everything is shared infrastructure now. I don't want to pay to keep a machine up 24/7 just to use it to run a build for 10 minutes half a dozen times per day.
So instead I lease time on shared hardware with ephemeral "containers" or "virtual machines" or whatever.
At last job, the default was letting broken changes pass the build, they break some step of the setup/run process that's not run on a partial build. New joiners came in and they couldn't build because the build was broken.
Had to fix it by setting up two jobs, one running from scratch (30 minutes) and one incremental (10 minutes). The build from scratch was catching a broken change or two every week.
It's a bit misleading to use "build time" to describe this improvement, as it makes people think about build systems, compilers, header files, or cache. On the other hand, the alternative title is descriptive and helpful to all developers, not only just builders - people who simply need to clone a branch from a large repository can benefit from this tip as well.
Where in the article does it say that? The article says this:
> This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result. Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds.
Both of those sentences say the clone time was reduced by 99%. There are percentage numbers given for how much the build time was reduced, nor any numbers about the total build time.
Presumably the build does more than just clone
If the title is a lie (which it probably is), then nevermind that number, but it's clear where it came from.
The only way build time could have been reduced by 99% is if every part of the build other than cloning is negligible. It is far more plausible to assume that the title is simply wrong.
"This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result."
So the title is just completely wrong.
"We found that setting the refspec option during git fetch reduced our build times by 99%."
So, the article contains contradictions.
Eventually they stopped asking him to come because he couldn't get anything done, and so I had a look at it. In the Movie Clip library of the project I found he had an empty text field somewhere that was configured to include a copy of almost the entire Unicode range, including thousands of CJK characters, so each time you built the SWF it would collect and compress numerous different scripts from different fonts as vectors for use by the program. And it wasn't even being used by anything.
Once I removed that one empty text field, builds went down to about ~3 seconds.
Edit: I misremembered, it was a social network thing from Google+ https://www.techspot.com/news/43064-google-adds-1-button-to-...
Praise Microsoft for turning the corner, Dislike Google for ads and snooping, Praise Apple for privacy, Dislike Zoom for privacy, Dislike Pinterest for middlewaring Google Image, and so on.
Sometimes the negative impact of a company is just more interesting to people than what the article brings to the table.
Look, I get it, sometimes you want to rant about a company that you think is doing something you don't like: my point is that we have specific threads for them where such a comment could at least be on-topic. When you come to an article that about Pinterest doing some git thing to make their builds faster and your comment is "they're ruining my search!" you're commenting at the level of someone who hasn't read the blog post.
FWIW, in the all years I have been on this site, I have seen this happen regularly and I have yet to see any reduction in such links or these kinds of discussions. Seeing as you've been here longer, I'd be curious to hear about why you might feel differently.
I opt for the "teach non-tech people how to dork" route instead: https://soatok.blog/2020/07/21/dorking-your-way-to-search-re...
And shame on you, Google, for playing along and indexing their shit, when it's not visible when I click through.
Project lead called it the biggest productivity improvement in the project; now we could build over lunch instead of over the weekend.
If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our build pipeline is the Cypress tests. (They're also the most unreliable part.)
Deploys were painful, as any missing dependency had to be searched in our notebooks over 3G, then copied to an external storage, then plugged into a Windows machine, uploaded to the production server through SCP and then deployed manually over SSH. Sometimes we spent hours doing this again and again until all dependencies were finally resolved.
I worked there for almost a year, did many cool gigs and learned a lot. But my most valuable contribution came when at some point, tired of that unpredictable torture that were the deploys, started researching into solutions. I set up a pypi proxy into one of our spare office machines and routed all my daily package installs through that. Then I copied that entire proxy content into the production machine before every deploy, and voila, no more surprises.
I left this job a few weeks later, but have heard that this solution was very useful for many devs that joined the team afterwards.
Would you say the (slow and unreliable) Cypress tests are worth it still?
Personally I've recently started using Playwright and I'm quite happy with it. There was occasional misunderstanding of their API, but 95% of time it's great. Microsoft is kicking butt these days.
And I get the reason for it; you don't want to accidentally merge breaking changes. But it does make our build pipelines very slow and unreliable.
So are they worth it? I don't know. If I had my way, we'd only run them on master, and not make it a requirement for feature branches to pass them. Because if you fix one tiny thing, you now have to wait 15 minutes again for the Cypress tests to run. I think they'd be better in a different setup than what we're doing.
So far it's been great and has saved a couple of releases in a month or so of use!
I have no idea if it is any better now, but we dropped it about 6 months ago in favor of pure Selenium C# for our UI tests.
edit: a word
Oh man, I feel your pain.
I've not yet had the opportunity of having a large Cypress suite (working on it as we speak), but is it still more stable than e.g. Selenium is? Honestly 80% of issues we had with that were 'unstable' tests.
I used to write automation, and I can say that Selenium tests can be written to be very stable. Just depends on how they are written.
A better rule might be that at least one unit or e2e test was added or updated to reflect the change in the code, and that that particular test succeeds. But run all the others on master.
One advantage (or occasional disadvantage) of Cypress test before merging, is that there is someone clearly responsible for fixing it if a test fails. Problem is, sometimes the failing test has nothing to do with anything the creator of the pull request did. It's still a mystery how that's possible, but it happens. Hence my feeling that Cypress tests aren't very reliable. At least some of ours aren't.
As a personal anecdote, clones of the Rust repository in CI used to be pretty slow, and on investigating we found out that one key problem was cloning the LLVM submodule (which Rust has a fork of).
In the end we put in place a hack to download the tar.gz of our LLVM repo from github and just copy it in place of the submodule, rather than cloning it. [0]
Also, as a counterpoint to some other comments in this thread - it's really easy to just shrug off CI getting slower. A few minutes here and there adds up. It was only because our CI would hard-fail after 3 hours that the infra team really started digging in (on this and other things) - had we left it, I suspect we might be at around 5 hours by now! Contributors want to do their work, not investigate "what does a git clone really do".
p.s. our first take on this was to have the submodules cloned and stored in the CI cache, then use the rather neat `--reference` flag [1] to grab objects from this local cache when initialising the submodule - incrementally updating the CI cache was way cheaper than recloning each time. Sadly the CI provider wasn't great at handling multi-GB caches, so we went with the approach outlined above.
[0] https://github.com/rust-lang/rust/blob/1.47.0/src/ci/init_re...
[1] https://github.com/rust-lang/rust/commit/0347ff58230af512c95...
Exactly this. Especially if the repo and CI pipeline are complicated, it is incredibly easy to just assume “it’s slow” is a fact of life.
And from the point of view of the dev-productivity team, well, they have tons of possible issues to deal with at any given time. Not just CI but the repos themselves, the build system, maybe IDEs, debuggers, ... Sure the fix ends up being easy but you have to know to go looking for it.
Frankly, I’d rather spend extra time trying to address problems/bugs/potential security holes in the actual shipped code than in fixing a poorly working CI pipeline...and I’m the kind of dev who gets really irritated by these problems. But you have to prioritize.
Basically, barring “external” forces like cost overflow, customer unhappiness, or similar...stuff like that gets fixed at an equilibrium point between how much the problem hurts the dev, how adjacent to the codebase the devs current work is, and how interesting/irritating the dev finds the problem.
That said, I generally shy away from shallow clones and probably wouldn't use it here:
- it's a trap for people who ever want to work in that repo normally (we use the trick for more than just LLVM) - I believe shallow clones, over time (e.g. for contributors), are less nefficient than deep clones - I would expect shallow cloning to reuse fewer objects and benefit less from git's design. [0] describes a historic issue on this topic
[0] https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...
What is the reason for cloning 50 commits? Whenever I clone a repo off GitHub for a quick build and don't care about sending patches back, I always use --depth=1 to avoid any history or stale assets. Is there a reason to get more commits if you don't care about having a local copy of the history? Do automated build pipelines need more info?
All that being said, I don’t know why you would need you build agents to clone the whole damn repo for every build. Why not keep a copy around? That’s what TFS does.
One other thing I've seen to reduce the Git clone bottleneck is to clone from Git once, create a Git bundle from the clone, upload the bundle to cloud storage, and then have the subsequent steps use the bundle instead of cloning directly. See these two files for the .NET Runtime repo[2][3]. I assume they do this because the clone step is slow or unreliable and then the subsequent moving around of the bundle is faster and more reliable. It also makes every node get the exact same clone (they build on macOS, Windows, and Linux).
Lastly, be careful with the depth option when cloning. It causes a higher CPU burden on the remote. You can see this in the console output when the remote says it is compressing objects. And if you subsequently do a normaly fetch after a shallow clone, you can cause the server to do ever more work[4].
1: https://github.com/dotnet/runtime/pull/35109
2: https://github.com/dotnet/runtime/blob/693c1f05188330e270b01...
3: https://github.com/dotnet/runtime/blob/693c1f05188330e270b01...
4: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...
edit: looks like git doesn't implement fetching thin packs when populating a shallow clone. It will still avoid fetching unnecessary packs, so the efficiency is still high for most software repositories.
> There are some unfinished ends of the whole shallow business:
> - maybe we have to force non-thin packs when fetching into a shallow repo (ATM they are forced non-thin).
> Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds.
This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay.
I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institutional complacency with things like this.
I'm sure everyone was happy when the clone took 1 second.
I doubt anyone noticed when the clone took 1 minute.
Someone probably started to notice when the clone took 5 minutes but didn't look.
Someone probably tried to fix it when the clone was taking 10 minutes and failed.
I wonder what 'institutional complacencies' we have. Problems we assume are unsolvable but are actually very trivial to solve.
I'm sure many engineering projects have similar improvements that just get a ticket/issue opened and never revisited due to the mountain of other seemingly pressing issues. From IPO to the start of the year Pinterest stock price had been trending downwards - I'm sure there was more external pressure to increase profitability than to fix CI build times. The stock has completely turned around since COVID, so I'm sure that changes things
Even when most of the rest of the engineering environment is fine, the build scripts and configuration often aren't under version control themselves, or are manually deployed - meaning any changes require access to carefully guarded server credentials. This may even be by design as a "security measure" - as if I didn't already have the ability to run arbitrary code on the build servers in question through unit tests etc. The gatekeepers in question are often an underfunded IT department that has too much on their plate already, and are underwhelmed by the idea of reviewing a bunch of changes to "legacy" code that they've somehow convinced themselves they'll rewrite "soon" that they don't directly benefit from anyways.
And I find I can rarely run the scripts locally. They're also often hideously locked in to a specific CI solution that I can't locally install without a ton of work on my part to figure out the mess of undocumented dependencies, and rife with edge cases that I can't easily imitate on my dev machines.
My preferred CI setups involve a single configuration file, checked into the same repository it's configuring CI for, that simply forwards to a low-dependencies script that works on dev machines. Getting there from an existing CI setup, however, can be quite the challenge.
Any simple wrappers for git that can do this transparently?
GoCD for example maintains a single copy of the repo on the server for every pipeline that refers to it and the agents have the repos that they work on checked out. Any local changes or untracked files are by default cleaned. There are settings to force reclone etc, but it's not the default.
Did someone really think "well it takes 40min, what can you do about it?" and just left it as such?
I knew people who would have that mentality in companies that are not around anymore. Take it as you want.
Yes, git is hard, but you know, maybe someone else has a better idea, or you can check SO, etc. (I don't even know why they were adding the refspecs there)
I spend a lot of time optimizing builds, because the effect is a multiplicator for everything else in development.
But it is not an easy task. One issue with performance-monitoring is that you have to carefully plan your work, or you will sit around and wait for results a lot:
Try the build: 40 minutes. Maybe add profiling statements, because you forgot them: another 40 minutes. Change something and try it out: no change, 40 minutes. Find another optimization which decreases time locally and try it out: 39.5 minutes, because on the build-server that optimization does not work that well. etc.
You just spent 160 minutes and shaved 0.5 minutes off the build.
I'm not saying it's not worth it, but that line of work is not often rewarding.
On the flip-side I once took two hours to write a java-agent which caches File.exists for class-loading and managed to decrease local startup time by 500% because the corporate virus-scanner got active less often.
Think of all these complex systems developers and SysAdmins need to maintain at a company. Then think of how well each person knows each technology. Most of them will be "T" shaped, ie know one tech well but surface-level on all the others.
If I know several tools really well (or better than the company's sysadmins / devs) I can probably find some security issues with them.
Versus the simple thing you would author yourself: if you know the engineering tradeoffs made at the per-line level you have a decent grasp of the performance and flexibility, but you are implementing it and debugging it.
I will add that I think software complexity is only going to continue increasing over the long term; it reduces in some domains, but expands in others as we develop more advanced systems. Some kind of analogy to entropy.
On the other hand TS seems to be more and more popular, which requires a compilation step.
In many domains git has replaced other version control systems.
I would love to see a new approach to version control. Things like subversion or mercurial have exposed too many drawbacks for them to win back industry.
Among startups, I'm sure git holds a near monopoly, but if you move into other parts of the industry, that monopoly loosens.
git fetch --prune --no-tags --depth 1 origin+{ref}:remotes/origin/{ref}
The pipeline I manage is many repos rather than a monorepo, and maintaining long-licheckouts in this context is not really realistic, but what does work and is very fast is just grabbing tarballs— GitLab and Github both cache them, so they don't don't cost additional compute after the first time, and downloading them is strictly less transfer and fewer round trips than the git protocol.
The only real cost is that anything at build time which needs VCS info (eg, to embed it in the binary) will need an alternate path, for example having it be able to be passed in via an envvar.
Ok, I'll ask: why does a single repository have over 2,500 branches? Why not delete the ones you no longer use?
That is if we assume they copy google's philosophy of a single monolith repository.
Pinterest has about 2000 employees, assuming 20% are active developers thats about 400 people, that gives you roughly 6 branches per developer which wouldn't be outrageous.
The example in the article is a bit of a special case. It is a huge, and old, monorepo. In the typical case, fetching everything and fetching master is equivalent because all commits in all branches make their way into master anyway. If you have a weird branching strategy where you maintain multiple, significantly diverged branches at once, but only care about one of those branches at build time, then this optimization would save you time.
Squash merges leave a ton of commits just floating in your old branch. If you delete the branch (the 40B file), all those commits are still there. Doing lots of squash merges brings you into this case I mentioned:
> If you have a weird branching strategy where you maintain multiple, significantly diverged branches at once, but only care about one of those branches at build time, then this optimization would save you time.
Based on the article linked here, they do.
Its not the fact that they had lots of branches itself, its the fact that they had lots of commits hanging out in the middle of nowhere.
If your repos weigh in at 20GB in size, with 350k commits, subject to 60k pulls in a single day, having someone with half a devops clue take a look at what your Jenkinsfile is doing with git is not exactly rocket science or a needle in a haystack. (Here's hoping they discover branch pruning too; how many of those 2500 branches are active?)
As a consultant I've seen plenty of apallingly poor workflows and practices, so this isn't all that remarkable... but for me the post seems kind of pointless.
I don't care for casting stones. It's clearly a big win, and you don't get numbers like that every day. But I feel like someone should've twigged to this much sooner.
It sounds to me like you don't have a monorepo at all and instead have six repositories for six project areas.
I do not know whether that's what the blog author meant by that though.
Their iOS and Android repos may contain the code for multiple apps. Though, I'm not aware of which other apps Pintrest (the company) creates besides the obvious one.
git clone --single-branch
achieve the same thing (i.e, check out only the branch you want to build) ?
Also, why would you not only check out one branch when doing CI ?
Implies --single-branch
The most effective organizations I’ve worked at built mechanisms and processes to disseminate these kinds of learnings and have regular brown bags on how a particular problem was solved or how others can apply their lessons.
Keep it up Pinterest engineering folks.
He says it went from 40 minutes to 30 seconds. Does this mean they found a way to grab the whole 20GB repo in 30 seconds? seems pretty darn fast to grab 20GB, but maybe on fast internal networks?
Or maybe they meant that it was 20GB if you grabbed all of the many thousands of garbage branches, when Jenkins really only needed to test "master", and finding a solution that allowed them to only grab what they needed made things faster.
I'm also curious about the incremental vs "cloning fully" aspect of it. Does each run of Jenkins clone the repo from scratch or does it incrementally pull into a directory where it has been cloned before? I could see how in a cloning-from-scratch situation the burden of cloning every branch that ever existed would be large, whereas incrementally I would think it wouldn't matter that much.
It probably means including all commits.
It looks like they were successfully only pulling the last 50 commits, but they were doing that for each of 2500 branches. Now they are pulling only the most recent 50 commits for one branch.
So if you had a large, blue second generation widget with a foo accessory and option buzz, you were value 30202, and if was the same one except red, it was 26420...
Every time the CRM software started up, it cycled through the options, generated a new XML file with all the results, this took about a minute and created like a 60MB file.
The fix was to basically version the XML file and the options definition file. If someone had already generated that file, just load the XML file instead of parsing and looping through the options file. Started up in 5 seconds!
What was the excuse that it took so long in the first place? "The CRM software is written in Java, so it's slow."
But back on topic: does anyone have any insight into when git fetches things, and what it chooses to grab? It is just "when we were writing git we chose these things as being useful to have a 'please update things before running this command' implicitly run before them"? For example, git pull seems to run a fetch for you, etc.
EDIT
https://www.jenkins.io/doc/pipeline/steps/workflow-scm-step/ makes it sounds like the branches option specifies which branches to monitor for changes, after which all branches are fetched. This still seems like a counter-intuitive design that doesn't fit the most common cases.
> The following example makes the runner shallow clone to fetch only a given branch; it does not fetch any other branches nor tags.
https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cl...
GIT_FETCH_EXTRA_FLAGS accepts all options of the git fetch command
> The default flags are:
GIT_DEPTH.
The list of refspecs.
A remote called origin.Can anyone explain this? Seems ripe for another 99% improvement even with hundreds of devs.
> Cloning monorepos that have a lot of code and history is time consuming, and we need to do it frequently throughout the day in our continuous integration pipelines.
No you don't!
If removing per-build clones was the only way to speed things up, I'm absolutely sure you could figure out how with medium difficulty at most.
Building a product with poor visibility and ridiculing users for not knowing internals is the worst practice in Computer Science.
Hadoop did the same, and has set a record of fastest software to become legacy.
Super nice to see great comments here and the nice article.
Having said that, happy to be proven wrong, and learn about it.
Not that any of this is important for the article to be interesting. In a previous job we had to fight long pull times and we quickly created a git repo for CI that would sit on a machine next to the CI server and would periodically pull from GitHub to avoid the CI to do pulls over Internet.
> This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result.
Sounds like it didn't reduce build times quite by 99%.
Then bcrypt work factor! It was originally 12, reduced it to 1 (don’t worry, production is still 12)
It should be 'git init'
Before: 1.39 seconds
After: 0.62 seconds
I have this default line in config/initializers/devise.rb:
config.stretches = Rails.env.test? ? 1 : 11
It's a trope at this point how the modern slow hashing algorithms are utterly misconfigured. Stopped counting how many times I've seen it.
Take a whole second to compute a hash on the production machine because "hashing is supposed to be slow", noting the production server is a low frequency Xeon that has many core but they're half as slow as your development with a 4GHz i7-9999.
Hashing is supposed to take milliseconds, not seconds. If it's taking longer than 100 ms you need to make it faster.
edit: found the problem, this bad stackoverflow answer that's been spreading bad recommendations for years https://security.stackexchange.com/questions/17207/recommend...
What is an “iOS monorepo” supposed to be like?
This did NOT slash build times 99%, but rather time to do a git pull.
> We found that setting the refspec option during git fetch reduced our build times by 99%.
Seems pretty clear to me that build times were reduced by 99% as a result of cutting the git fetch times significantly (but exacyt number is not give). The headline looks correct to me.
Unless their build is 100% git pull time, this did not reduce build time by 99%.