Hacker News Reader

Comment by ufo | Hacker News Reader

One thing that I still don't understand about github actions is how much computing power gets wasted downloading and reinstalling the same dependencies over and over. A build that takes a couple of seconds in my local machine can easily take several minutes in github actions.

View on HN

I prefer GA to basically every other build system especially because of the plugins but yeah, and I think caching can be turned on... but it has tons of inconsistencies related to packages and deps. This video really highlighted just how brittle the whole thing is even behind the scenes:

https://youtu.be/9qljpi5jiMQ (13:58 time stamp for the part that is especially relevant here)

After that video, I think I finally get Github Actions.

In Azure DevOps pipelines there's a fun issue where if you're pulling a dependency from nuget.org it gets pulled very quickly, but pulling from your own artifact repositories hosted in the very same Azure DevOps instance take 40-80 seconds. I'm not the one paying the extra bills when my builds take 5 to 6 times longer than they should but it's still frustrating.

And Microsoft has effectively zero incentive to fix this because it still technically works and they get to charge my employer for all that extra build agent time. Here's a few examples of people reporting similar issues and getting zero help:

https://developercommunity.visualstudio.com/t/devops-build-p...

https://developercommunity.visualstudio.com/t/nuget-restore-...

https://developercommunity.visualstudio.com/t/slow-performan...

Microsoft hosted agents aren’t charged by the minute on ADO, IIRC. You pay for a certain number of parallel agents per month.

Unless you’re booting build servers on demand on your infra, you aren’t paying more for more minutes. Just waiting :(

(Unlike GitHub, where they do charge per minute.)

do you know if ADO has any caching mechanisms for pipelines doing builds?

The issue in this case isn't caching, even if I'm not actually pulling any packages from our private nuget repo simply having that repo listed in the project's nuget.config file is enough to cause the issue.

GitHub charges per minute. Some incentive to keep actions on the most under specced machines available.

Yeah. We have a similar thing going on with self-hosted GitLab at my work. (And there, there isn’t even any incentive why it should be slow.)

When I make an edit and rebuild on my computer, it takes a few seconds to rebuild because most things are cached.

In CI it takes 20 minutes on the MR branches of our repo. 30 minutes on master. Because a whole bunch of crap is downloaded from scratch and rebuilt from scratch.

Our infrastructure guys did set up something that is able to cache stuff that some of the repos use. But with the limited access I have I have not been able to figure out how to make use of it for the repo I work on. And they don’t have time to look into it for me either.

That’s my life lol

You can define a cache key in your .gitlab-ci.yml

https://docs.gitlab.com/ee/ci/yaml/index.html#cache

Considering gitlab is a very k8s native app it's way of running a build and moving caches into it, vs moving the container over to where the cache's are is very annoying. I've had issues open for years to update the cache-key with a CSI mount and let k8s move the pods to where the mount is fast. As it is now it pulls everything from S3.

Yeah gitlab has pretty good caching support and it is basically a must if you do bigger jobs, especially on stuff like c++ apps. It is pretty brittle sometimes, and in my experience it won't be able to be as fast as actual local dev but it's still good. You can even make it use a remote cache IIRC.

On free runners I'd regularly hit the 1 hour limit. Took some time to set up self-hosted through EC2 and really optimize caching, its now down to 10-20 mins, depending on the repo. Most of that is because of the self-hosted runners, the free ones are painfully slow.

Still, on my local machine these jobs would take 5 mins so its not perfect. And as the build gets more complicated and more stages are added, the problem compounds since the initial download is the slowest part.

Do other CI services do it differently? I worked with Jenkins and Circleci and it was exactly the same.

For github actions caching was fairly easy to add.

Last time I tried to add caching to my actions, I spent several hours and still couldn't get it to work properly in the end. I really wonder how many computing hours have been wasted because the default and most likely to work is to not cache anything.

On CircleCI it's also easy to add using save_cache and restore_cache, the docs have examples for different popular frameworks. There are a couple of advanced features and I have been able to cache most build artifacts.

They also cache the Docker images without you having to do anything but it really only works if you use their base images. With custom images the cache rate doesn't look great in my experience but I have no stats to back that up.

Use a persistent runner and you can take advantage of on disk caches...at the cost of repeatability you get from a clean image on each run.

repeatability must mean different things to us, a cache hit is repeatability to me, a network call can easily fail for so many reasons

It sure can fail but running on a clean agent each time reduces the "Well, it works on my machine" scenarios you can run into.

For example:

The last run didn't clean up, or changed something global.

New runs don't work because some expected state is missing that had been set up on a previous run.

Blowing the agent away each time reduces the chance that something previously run can impact your current run.

This is at the cost of pulling from networks each time. You can reduce those issues by adding network caches for modules/dependencies.

I use persistent runners, but we had to add additional steps (docker prune, mvn -U, docker image ls, etc) to keep the runner healthy and be better able to debug issues.

Given GitHub also owns npm these days, at least some of the constantly reinstalled dependencies are just getting copied from the equivalent of "next door".

And/or they can fix GitHub’s caching of NPM if it saves NPM money.

Reply on news.ycombinator.com