GitHub Actions Outage

We run Bitbucket pipelines, but a similar notion applies. Everything we do is in a container. If we wanted to run our CI manually, we can just run our steps as a script on anyone’s workstation (given keys/permission). Staying away from the proprietary run environments and being easily to run them ourselves has saved us a number of times. Especially when running on Atlassian where things go down often.

Of course there are plenty of self-hosting options if you want that responsibility, but a very rare outage of CI/CD doesn't seem like something that's worth spending resources planning around. On the scale of possible disruptions, it's pretty minor.

We looked into it, very briefly, and decided that a few hours or even a whole day of downtime per year wasn't really worth the time and cost it would take to mitigate this in our case. If our pipelines happen to fall over, and it also happens to occur during our working hours then the solution for us is to either work on something else or head down to the pub.

Kinda - what we're seeing at Depot (https://status.depot.dev/clyrvud6i57402igofm6jtb7id) is that the APIs for managing Actions runners or retrieving work are receiving rate limit errors, interestingly missing the regular rate limit status headers[1]. We do have some jobs processing, but overall the GitHub API and Actions control plane seem very overloaded by the outage.

[1] - https://docs.github.com/en/rest/using-the-rest-api/rate-limi...

Can confirm that self hosted runners are seeing issues at WarpBuild.

Just another Thursday :shrug:

https://warpbuild.instatus.com/