Hacker News Reader

Comment by jacooper | Hacker News Reader

Honestly for me the bigger issue is the upcoming bandwidth limit.

There is no clear date to when its going to be enforced.

But its definition is so wide its crazy, its basically any egress data except the web interface and shared runners.

AFAIK it will also include git clones!, so if your project suddenly gets popular the users clones will cost you many too.

Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit.

And since it should apply to GitLab pages, it becomes useless for anything the you want to get few visits.

Since GitLab is behind cloudflare, you might as well just use Cloudflare Pages at this point.

View on HN

> Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit.

You make it sounds like it's horrible. I'd offer a different take in that sure, it can be a huge negative if done recklessly on GitLab's part. But if done correctly, it can actually be a good thing. I think CI pipelines doing a full clone from scratch on every build (or npm installs or other bootstraps) is extremely wasteful in terms of resources, so I'd be glad if that reduces it significantly.

GitLab CI default clone configuration already AFAIK only includes the last 20 commits, so its already slimmed down.

If you start from scratch, regardless of history, you're fetching at least the size of the repo. In many cases, that's hundreds of MBs for every build and I'm a huge fan of building for every commit. I find that a colossal waste.

I have a very old fashioned and "not recommended" setup with Jenkins that only pulls new commits, because it works out of a persistent working directory. Works wonders for a Django/ES6 project and sips bandwidth. I wish more of the modern containerized, start-from-scratch and so on setups would work in a similar way wrt the bandwidth they use.

GitLab's runner agent software doesn't support a local git cache.

GitLab.com Saas agents have no sticky-ness, so basically always clone.

Yeah, that does sound wasteful, just like what parent said. They should add some sort of cache support if that's the case.

Thanks for the suggestion of adding a Git cache for a GitLab Runner. A similar mechanism already exists that allows control of the Git strategy with fetch on a local working copy which is faster, and clone if not yet existing. All benefits and limitations are documented [0].

To help prevent unneeded traffic when a new pipeline job is executed, GitLab Runner uses a shallow Git clone by default on GitLab.com SaaS that only pulls a limited set of Git commits from the current head instead of a full clone. [1]

There is a feature proposal [2] to add support for partial clone and sparse-checkout strategies that have been added in more recent Git versions. Recommend commenting/subscribing.

Please note that for GitLab.com SaaS Shared Runners, the traffic limits do not apply with mostly internal cloud traffic. [3]

[0] https://docs.gitlab.com/ee/ci/runners/configure_runners.html...

[1] https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cl...

[2] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26631

[3] https://about.gitlab.com/pricing/#what-counts-towards-my-tra...

GitLab team member here. Thanks for the comment.

We understand people don’t want their quotas being filled by something outside of their control. For Open Source projects, we have GitLab for Open Source which contains higher limits from the Ultimate tier (250GB Storage, 500GB transfer/month). In addition, we intend to look into other ways to address your concern such as counting only your own traffic or allowing you to limit external traffic.

I love the fact that they haven't thought out a lot of how this needs to work. You can't even decide if it's worth sticking with GitLab because they don't even know how they are going to handle half of this stuff.

I appreciate their open discussion and iterative approach. There’s no need to get your feathers ruffled. These measures mainly impact users freeloading their services, so maybe have some empathy for the company providing them?

> These measures mainly impact users freeloading their services

Does it? We use GitLab at work, and we have to use local runners (for regulatory reasons + GitLab runners don't support what we need anyway).

Unlike other CI/CD providers (e.g. teamcity), GitLab Runners don't have a local git cache. So if you need to do a clean clone for an important build (gitlab runners don't clean up well after themselves), you need to re-clone the repo from GitLab.com.

With a 1:2 ratio for storage vs bandwidth (10GB storage, 20GB bandwidth per month), assuming using 1GB in latest commits, and 2GB total repo size (e.g. a vendored dependency that doesn't change frequently):

- 2 runners - Clean once a week - 5% of bandwidth per shallow clone - 40% of bandwidth per month on CI/CD alone.

Leaving space for 6 clones by developers (hope you don't upgrade your dev machines often).

If you're using a submodule in multiple projects you're going to tear through your bandwidth.

Like I said, it /mainly/ impacts freeloaders. Having a 1+2 GB git clone is not a common usecase. That being said, it appears that you would not be affected:

> Transfer is the amount of data egress leaving GitLab.com, except for:

> Paid plans only: self-managed runner transfer and deployments. This is determined by transfer authenticated by either a CI_JOB_TOKEN or DEPLOY_TOKEN

(https://about.gitlab.com/pricing/#what-counts-towards-my-tra...)

Freeloaders... You mean the people they encouraged to use their service and then pull the rug out from under them?

Thanks for sharing for your feedback. I have added more insights into caching and checkout strategies to reduce traffic and speed up job execution in GitLab Runners in this comment: https://news.ycombinator.com/item?id=32408960

Reply on news.ycombinator.com