Hacker News Reader

Comment by pjot | Hacker News Reader

Considering Gitlab employees use Gitlab to make changes to Gitlab, how will they fix Gitlab?

At GitHub, we (eventually) ran a completely separate GitHub instance and would deploy there first. We had tooling which allowed us to deploy manually to production from that backup instance in the event we made a breaking change.

It's git, so everyone has a copy of the repo. You can continue collaborating with send-email or some random SSH server. That's the beauty of a DSCM.

Yeah, but they also use the Pipeline features to build and deploy changes.

But they are clever, you can have the runner local.

(They start to ruin that, but lets say, the idea is still visible.)

Only for `script: "echo hello world"` style jobs; anything more real makes `gitlab-runner exec` a total farce

If by local you mean local to a cloud instance that runs automated builds on every merge for specific branches, then yes.

Granted, those build instance pull from the central git repo, and are triggered from GitLab... so also, no.

local like in tree.

Yeah you can't really. It doesn't support a ton of syntax and features.

And they also said they were going to remove that ability too. Tbf to them after the inevitable response they did say they will come up with a proper solution, though that was some time ago.

Obviously the real solution is to use Bazel and not to use Gitlab CI as a crap build system.

Bazel seems like a good idea but it's far too immature to actually work in the FOSS world, almost none of the external _rules are google quality, and it damn near requires a PhD to set up properly.

I spent a good few months learning it and it's not the tool I would reach for in almost any circumstance unfortunately.

Docs are also lacking, which is certainly not a problem with GitlabCI

Yeah it's definitely not simple, but I have repeatedly worked for companies that get stuck on all the problems that Bazel solves. I'm like "we should use Bazel, it solves this issue properly", and they're just like "nah... so anyway this is a big problem, how do we solve it??!".

It would be nice if it had better integration with existing package managers like pip, cargo, npm and so on though. Vendoring your dependencies is fine for a big project like Chrome or Android or your company monorepo; it's a bit annoying for a small CLI tool or whatever.

Another option is something like Landlock Make, but I haven't tried it.

Yeah, why not? But often GNU Make is fine for incremental builds.

This. Merge to master -> deploy to staging -> test runs -> push button enables for go-to-prod.

You could, but in all honesty, most programmers (myself included) aren't as brilliant as Linus Torvalds, and, while I could stand up a git mirror in my sleep (because it's trivial - you should be able to as well), the finer points of coordinating work via a git mirror, when https://xkcd.com/1597/ is the level of understand most people have with git, seems to say it's not going to work. Add on the fact that the git repository is the least bit of functionality (eg PRs and pipelines), and being down is a serious barrier to progress.

http://Xkcd.com/303 but for GitHub is down.

Agree - also sending a patch via email is definitely possible, but not at all practical. I’m not sure if I’ve ever come across anyone who’s gitops workflows include email

you don't need to send patches via email, you can push them over ssh pretty easily

For example on my team we have a few devs who push patches from laptop directly to their work desktop, as a part of day-to-day synchronization. No central ssh server involved.

You've never come across the Linux kernel? The Linux kernels process isn't practical? Okay, that's an interesting point of view.

Yeah, I didn't think of testing and deployment pipelines. Where I work we use GitHub actions for testing, but not for deployment. And our projects are small enough that I can easily run the tests in docker on my development PC.

GitLab team member here.

If you plan on a GitLab mirror, including the Git repository and groups/projects, the direct transfer migration can be helpful https://about.gitlab.com/blog/2023/01/18/try-out-new-way-to-...

GitLab Geo can be an alternative for larger, high-availability setup requirements. https://docs.gitlab.com/ee/administration/reference_architec...

I think it gets deployed from https://ops.gitlab.net/

GitLab team member here. Thanks for asking.

Incidents can have different types, i.e. when an application bug or performance regression is discovered, this can involve reverting MRs and rolling back releases. The Platform, Delivery group has a top-level responsibility for ensuring continuous delivery of the GitLab application software to GitLab SaaS, https://about.gitlab.com/handbook/engineering/infrastructure...

Other incidents may involve hardware or infrastructure failures, or a combination of both, infrastructure failure that renders GitLab application services unavailable. This requires cross-functional collaboration from infrastructure, product, engineering, etc. teams in the incident.

To get a better understanding here, it is helpful to review the incident management handbook https://about.gitlab.com/handbook/engineering/infrastructure...

Additional helpful information:

- The GitLab.com SaaS production architecture is documented in https://about.gitlab.com/handbook/engineering/infrastructure...

- The Monitoring of GitLab.com handbook provides insights into monitoring workflows, incident management, SLAs, etc. https://about.gitlab.com/handbook/engineering/monitoring/

- Runbooks https://about.gitlab.com/handbook/engineering/infrastructure...

For the current incident discussed in this HN thread, the review issue can be followed in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... to learn more.

Reply on news.ycombinator.com