GitHub appears to be struggling with measly three nines availability

This is definitely true.

At the same time, none of the individual services has hit 3x9 uptime in the last 90 days [0], which is their Enterprise SLA [1] ...

> "Uptime" is the percentage of total possible minutes the applicable GitHub service was available in a given calendar quarter. GitHub commits to maintain at least 99.9% Uptime for the applicable GitHub service.

[0]: https://mrshu.github.io/github-statuses/

[1]: https://github.com/customer-terms/github-online-services-sla

(may have edited to add links and stuff, can't remember, one of those days)

They're not even struggling to get their average to three 9s, they're struggling to get ANY service to three 9s. They're struggling to get many services to two 9s.

Copilot may be the least stable at one 9, but the services I would consider most critical (Git & Actions) are also at one 9.

Why defend a company that clearly doesn't care about its customers and see them as a money spigot to suck dry?

The five nines tech people usually are talking about is a fiction; the only place where the measure is really real is in networking, specifically service provider networking, otherwise it's often just various ways of cleverly slicing the data to keep the status screen green. A dead giveaway is a gander at the SLAs and all the ways the SLAs are basically worthless for almost everyone in the space.

See also all of the "1 hour response time" SLAs from open source wrapper companies. Yes, in one hour they will create a case and give you case ID. But that's not how they describe it.

That's only a valid sentiment if you only use the big players. Both of those have medium/smaller competitors that have shown (for decades) that they are extremely boring, therefore stable.

It's just that everybody is using 100 tools and dependencies which themselves depend on 50 others to be working.

I think this is a really important point that is getting overlooked in most conversations about GitHub's reliability lately.

GitHub was not designed or architected for a world where millions of AI coding agents can trivially generate huge volumes of commits and PRs. This alone is such a huge spike and change in user behavior that it wouldn't be unreasonable to expect even a very well-architected site to struggle with reliability. For GitHub, N 9s of availability pre-AI simply does not mean the same thing as N 9s of availability post-AI. Those are two completely different levels of difficulty, even when N is the same.

1-4 incidents per month compared to about 1 daily.

You can pin actions versions to their hash. Some might say this is a best practice for now. It looks like this, where the comment says where the hash is supposed to point.

      Old -->   uses: actions/checkout@v4
      New -->   uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

Perhaps mixing the CI with the CD made that worse because usually deployment and delivery has complexities of its own. Back in the day you'd probably use Jenkins for the delivery piece, and the E2E nightlies, and use something more lightweight for running your tests and linters.

For that part I feel like all you need, really, is to be able to run a suite of well structured shell scripts. Maybe if you're in git you follow its hooks convention to execute scripts in a directory named after the repo event or something. Forget about creating reusable 'actions' which depend on running untrusted code.

Provide some baked in utilities to help with reporting status, caching, saving junit files and what have you.

The only thing that remains is setting up a base image with all your tooling in it. Docker does that, and is probably the only bit where you'd have to accept relying on untrusted third parties, unless you can scan them and store your own cached version of it.

I make it sound simpler than it is but for some reason we accepted distributed YAML-based balls of mud for the system that is critical to deploying our code, that has unsupervised access to almost everything. And people are now hooking AI agents into it.

That's a high bar though. Few things are better than Swiss cheese.

That’s… one 9 of reliability. You could argue the title understates the problem.

> You don't need every single service to be online in order to use GitHub.

Well that’s how they want you to use it, so it’s an epic failure in their intended use story. Another way to put this is ”if you use more GitHub features, your overall reliability goes down significantly and unpredictably”.

Look, I have never been obsessed with nines for most types of services. But the cloud service providers certainly were using it as major selling/bragging points until it got boring and old because of LLMs. Same with security. And GitHub is so upstream that downstream effects can propagate and cascade quite seriously.

These days it is very common that something like opening the diff view of a trivial PR takes 15-30 seconds to load. Sure, it will eventually load after a long wait or an F5, but it is still negatively impacting my productivity.

It seems that the same metric is about a magnitude worse than before.

https://github.com/customer-terms/github-online-services-sla

> GitHub commits to maintain at least 99.9% Uptime for the applicable GitHub service.

... and none of the individual services have hit 99.9% uptime in the last 90 days according to this site. 0_o

It is a complicated project. Thankfully their durable funding story has improved in recent years and they are staffing GHES up at levels they haven’t for at least 7 years. Hopefully it improves. I’m not there anymore, I was laid off last year.

Sure they can. Perhaps a useful example of something like this would be to consider cryptography. Crypto is ridiculously complex and difficult to do correctly. Most individual developers have no hope of producing good cryptographic code on the same scale and dependability of the big crypto libraries and organizations. At the same time these central libraries and organizations have bugs, mistakes and weaknesses that can and do cause big problems for people. None of that changes the fact that for most developers “rolling your own crypto” is a bad idea.

Once we got the email that they were going to charge for self-hosted runners that was the final nail in the coffin for us. They walked it back but we've lost faith entirely in the platform and vision.

We wouldn’t couple so much if we knew reliability would be this low. It will influence future decisions.

Then Azure Dev Ops (formerly known as Visual Studio Team System) dead o n the ocean floor.

Although given how badly GitHub seems to be doing, perhaps it's better to be ignored.

When I saw his interview: https://thenewstack.io/github-ceo-on-why-well-still-need-hum... i thought "oh, there is some semblance of sanity at Microsoft".

This was after seeing those ridiculous PRs where microsoft engineers patiently deconstructed AI slop PRs they were forced to deal with on the open source repos they maintained.

When he was gone a few months later and github was folded into microsoft's org chart the writing was firmly on the wall.

https://mrshu.github.io/github-statuses/

Overall uptime is 90% (single nine), and the best individual service at 99.8% (two nines).

The core git functionality itself is at 98.98%, which works out to being down roughly a full business day each month.

That's the reason you hear the complaints: they're from people who no longer want to be using this product but have no choice.

Because Microsoft doesn't need to innovate or even provide good service to keep the flies glued, they do what they've been doing: focus all their resources on making the glue stickier rather than focusing on making people want to stay even if they had an option to leave.

Codespaces specifically is quite good for agent heavy teams. Launch a full stack runtime for PRs that are agent owned.

    >  keep hearing that Github is terrible

GH is many things to many teams and my sense is that some parts of it are currently less stable than others. But the overall package is still quite good and delivers a lot of value, IMO.

There is a bit of an echo chamber effect with GH to some degree.