Hacker News Reader

Comment by freedomben | Hacker News Reader

The status page links to https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... for details, but that 503s because it's on gitlab.com

View on HN

> Components: Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - GitLab SaaS Shared Runners, CI/CD - GitLab SaaS Private Runners, CI/CD - Windows Shared Runners (Beta), SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary

> Locations: Google Compute Engine, Digital Ocean, Zendesk, AWS

that's some _impressive_ blast radius and I actually struggle to think what would cause that much damage

Might be a domain issue or a routing misconfiguration. But people are mentioning 5xx errors so it could also be FELB/NLB misconfiguration.

> that's some _impressive_ blast radius

Most of the entries look like batch job processing, either by gitlab or managing jobs in runners.

Perhaps authentication, which is mandatory when managing batch jobs including in third party nodes.

If auth goes down on any service, you'll see similar blast radius.

Well, consider the fact that "most of the entries" comprise 10% of GitLab and things like "Git Operations" comprise the remaining 90%. This is a total system failure.

DNS, obviously.

According to the title of the issue (2023-07-07: Blackbox probes for https://cdn.artifacts.gitlab-static.net/cdn-test are failing) you win the game:

  $ host cdn.artifacts.gitlab-static.net
  Host cdn.artifacts.gitlab-static.net not found: 3(NXDOMAIN)

They must have used that same healthcheck for every single load-balancer or something, though, for it to nuke AWS, DO, and GCP

Thanks for your feedback. GitLab team member here.

The incident review is updated in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... I have added your feedback into https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1...

Reply on news.ycombinator.com