GitHub actions is down | Hacker News Reader

GitHub actions is down

github.community

127 pointsfagnerbrack72 comments

GitHubs status page went from industry leading with uptime graphs and instant updates to a static “everything’s ok” page (as the whole service 500s) page overnight with the Microsoft acquisition. They are burning good will like it’s a fuel.

I find this super frustrating. This weak sauce on the burrito of disappointment that is downtime adds insult to injury.

That's what happens when you join a trillion-dollar company. Somebody lets it slip that the service might not be reliable and suddenly you lose a billion dollars on the stock market.

Source? Any graphs to look at, for example. That covers years before and after the acquisition?

I've been thinking that GitHub has pretty good uptime.

(I wouldn't personally want to rely on a specific 3rd party like GitHub or Circle CI or anyone, for deploying my software, though. Wow, 560 people have followed the githubstatus.com link (I see in the Discourse forum) that's many people affected I think!)

IMHO GitHub should be spending less resources on adding achievement badges and more resources on preventing frequent service disruptions...

In a way it is good that they are focusing on the wrong things. It makes it easier for competitors to catch up and compete.

As far as actions go, I still think gitlab has a better CI/CD story.

I wished the gitlab.com runners had the same specs as the Github ones :(

More likely the team that worked on the badges (UI and data engineers) have no impact or ownership over disruptions (SREs and infra)

They're still a misallocation of Github's resources that could go into other teams.

Exactly, they could be spending more resources on hiring more SREs and infra and less UI and data engineers.

Stuffing twice the amount of engineers into a team is not exactly a good way to reduce complexity and reduce downtime, though.

Investing into a department is not the same as stuffing more engineers into a team

Almost as simple as Quality > Quantity.

But maybe hiring few more senior folks at higher salaries could help more than hiring twice as many juniors?

Or just doing less..

Investing could mean hiring more experienced candidates rather than more candidates. Investing could mean having a fraction of the devs and far more resources for each one of them. Whatever the problem this is absolutely a GitHub management issue they gotta fix.

Common rebuttal and one I sympathise with.

But that assumes that the people assigned to achievements could never help the platform to be reliable. If that’s the case, why hire them at all?

Either way you slice it, money is being spent here.

They could put together some badges and achievements for the SRE team to help motivate them to keep the f-ing thing running.

Are you saying the "It has been 001 days since the last outage" sign isn't working?

Even more likely that most SRE and infra engineers have no ownership over that either, since it's all Microsoft-authored code (in case you didn't know: GitHub Actions is Azure DevOps Pipelines is VS TFS Pipelines).

Yes but management are responsible for allocating resources. Pointing to another team that has no bearing on the issue at hand doesn't nothing to absolve management from this.

Are you making a "10 women should be able to have a baby in a month" type argument?

No -- are you presuming 'allocating resources' can only mean hiring more devs? It doesn't.

Yes and it's a poor allocation of resources. Instead of hiring n designers they could have hired m SREs and it stands to reason we'd have a less-gamified GitHub and a more stable GitHub. So plusses all around, I think.

Actions goes down so much more frequently than they disclose on their status page, every few days there's timeouts, caching problem etc...

Looks like an area ripe for disruption! (Only half-joking.)

It seems like there's enough disruption in this area already.

I think a hosted mulit-tenant CI platform appears fairly simple on the surface, but there's actually a lot of complexity under the hood when your product is arbitrary code execution as-a-service - and you want to offer a smoother user experience than something as general-purpose as a container runtime.

Especially when you throw in a large and generous free tier.

None of the recent examples of unbundled multi-tenant CI/CD appear to have wildly successful exit stories, or even continuing to run with great growth and profitability stories. I’m thinking of companies like Travis with that comment (which was, IMO, delightful to use for its era). It’s probably pretty hard to sell this to the average enterprise when you’re competing against product suites which do 100x more? Definitely feels like a situation where those suites are “Jack of all trades, and master of none” though, and collectively mediocre vs. individually excellent.

I'd like GitHub to stop disrupting itself please.

We are _besieged_ by caching issues. A good 1/4 to 1/3 of a particular AMI build action we run will fail because of a package mirror at GH being out of date.

Lots of complaining here. I'd be ticked if I was a paying customer.

But I just started using Actions yesterday. On my free open source project. And I've been using it all morning (no problems). And the whole time I was thinking: it's bloody amazing that I'm getting this for free.

(I used to work on a bit of a similar system at Google [internal remove build execution system for non-google3 projects] and ... there's a lot involved.)

> And the whole time I was thinking: it's bloody amazing that I'm getting this for free.

Me too

Now that everyone and their dog is moving to "serverless" hosting, how does one institute redundancies in case something like this happens again? It would suck if half the Internet and its open source projects go down because of a bad server configuration. Does anyone know of a "load balancer" for Github projects or source management in general? It looks like a solution many could benefit from.

Also use gitlab?

Or just some flexibility?

We don’t rely on GH for our production systems. We have a clone of our Git repo hosted in-house for anything that needs a Git repo.

We don’t rely on GH for our Primary CI/CD. We can make changes and push them through CI/CD without GH.

We have an understanding of the business impact of GH being down. It’s an enabler that we love using, but we can operate without it for a known period of time, albeit without certain workflows and value added tools.

If our tolerance for it being down we’re much greater, then it would be worth the money and overhead to host it on our own, which they support.

The simple answer is "you don't," because a couple hours downtime on your CI/CD platform shouldn't matter. You should be able to deploy by hand in an emergency.

The less simple answer is "by doing Disaster Recovery planning". Go outside and throw a rock and you'll hit a company doing hosted CI/CD. Set up two of them; when one fails, use the other.

GitHub could mitigate this by making its services region and AZ specific, and then only one region should go down at a time, letting you switch regions. Not sure why this hasn't happened, other than GHA is more of a "consumer product" and they may assume customers are too dumb to figure out how to switch regions. (This may sound harsh, but this is the same company that wouldn't update documentation about how to generate ssh keys to make them more secure because it might confuse their users)

For example, if you are using AWS, you could use Code Pipeline for your CI/CD alongside GitHub actions.

Use GHA for “PR open” triggered tasks, but Code Pipeline for tag or “PR merge” triggered tasks.

This way your deployment is decoupled from GH availability, but you get the benefits of GHAs first-party integration with your core software development loops.

maintaining bi modal behavior introduces a new set of problems. I'd take "should be able to deploy by hand in an emergency" a step further: deploy by hand, frequently. make sure that your production outage isn't the first time you are exercising this runbook.

Avoid the temptation to code up all the details of your build and deploy process in a proprietary single vendor CI tool.

Code your build using a build tool, script you deploy using tools suitable for that.

Now you can run the same process on multiple different CI services, or locally.

I recently delved into the topic of moving much of the logic from CI pipelines into Makefiles. This made me build https://github.com/mkincl/mkincl

Git is already decentralized, and you can push your code to any of the dozens of competitors easily. I often keep important projects on both Bitbucket and GitHub.

This is hitting workflows that use caching [1][2].

- [1] https://github.com/actions/setup-node/issues/516

- [2] https://github.com/actions/cache/issues/820

As of now (11:28 UTC) the status page has been updated.

https://www.githubstatus.com

Have had this for a few hours. Pretty annoying that the status page shows all green.

The Action to change the status page doesn't work, what do you expect?

They're using a 3rd party status page, it's a fancy CMS at best.

You mean the liability announcement page.

When the movement toward status pages started, it was great. Early adopters were eager to communicate their status, to the extent they technically could.

Nowadays it feels like most status pages show green even when things are decidedly not green.

Conspiracy theorists abound, but in most cases it's probably just ordinary human tendency to not broadcast one's foul-ups eagerly and broadly at the first opportunity.

Or just as likely: that only some customers are affected. So it's not a full outage.

So set it Yellow. "Some customers are/may be affected."

Customers who are affected? Appreciate the acknowledgment.

Customers who aren't? Appreciate that if they are next time, even if it's only a small subset, they'll be made aware.

Again? Last time that happened was 12 days ago. [0] Worse part is, it hasn't reflected on the status page. Typical of status pages.

Looks like relying on and going all in on Github and GitHub Actions really doesn't make any sense and I now expect this to go down once or even twice a month.

Oh dear. Perhaps self-hosting like how ReactOS, ARM, RedoxOS, GNOME, etc have done seems like a good idea.

[0] https://news.ycombinator.com/item?id=31579932

For now you can just host GitHub Actions locally. https://github.com/nektos/act

Unfortunately it's not fully-fledged enough to work properly with all CI configs. I had trouble with some conditions but also I note that it doesn't seem to implement reusable workflows yet. It's also a bit fiddly to set up so you don't run into API rate limits.

I think that people who are interested in having resilience should write their GHA CI config for act and then be happily surprised when it works on GitHub too. Or write them as something more portable (Earthfiles are interesting, but a bit clunky when I tried) that can also be executed locally — having the GitHub-specific stuff just be a very light wrapper to 'call into' the portable scripts.

I started to bring awareness to the Earthfiles goofy license, but it seems they've switched to MPL! https://github.com/earthly/earthly/releases/tag/v0.6.15

The (unfortunately named) Dagger is also an entry into that space: https://github.com/dagger/dagger#readme (Apache 2)

The title seems to be a bit misleading as this seems to be - at least right now - not a widespread issue.

We currently don't have any issues and the status page also doesn't list anything.

Status page updated not long after this post.. makes you wonder who is misleading who..

It looks like it's an artefact cache issue rather than a more widespread issue?

It seems like we could just sticky this post at this point.

they should be rewarded with "down every weeks" achievement

Why is GitHub Actions so frequently down since the rewrite to C# and moving the Azure DevOps engineers to GitHub?

I feel like you're answering your own question.

I'd be interested in reading a source for both of these points, if you have one. By "moving the Azure DevOps engineers to GitHub" do you mean they switched internal tooling on their own team? or that engineers from the ADO team switched orgs and now work for GitHub?

I can't find any info about anything in Github, much less actions specifically, being rewritten in c# either.

Bad integration work, and/or bad architecture on one or both of Github and Azure's sides.

```diff

- GitHub Actions Is Down (github.community)

+ GitHub Actions Is Down Again (github.community)

```

It happens once a quarter, sometimes more!

Github has outages nearly ever week now, sometimes multiple times a week not quarter

At what point would these outages impacting productivity be considered hostile and anti competitive?

When they impede a competitor (and only a competitor).

As it stands CI/CD is a bit of a commodity, so you can go with any of the solutions and they're largely similar.

As an individual sure, but as an organization? No way a team of 100 can just pivot to another provider.

CICD providers are not interoperable, once one is picked, it's a complete lock in.

If I was evil, I would offer a cheap service to employees of any potential contributors, one MBA types don't understand, then once they've made themselves dependent on it, make it unreliable. Boom. There goes your productivity and your credibility as competition.

Azure team != GitHub team

Reply on news.ycombinator.com