Red Squares – GitHub outages as contributions
red-squares.cian.lol
red-squares.cian.lol
Just compare the GitHub status page for public GitHub vs the enterprise cloud pages.
Enterprise has much better numbers and I’ve personally can’t remember the last time there was an outage that prevented me from doing work.
If the problems didn’t revolve around load, I’d expect to see the same uptime problems reflected on the enterprise offering.
Meaning that the load is crazy high but also that the tech stack and everything else is bad?
> Check GitHub Enterprise Cloud status by region:
> - Australia: au.githubstatus.com
> - EU: eu.githubstatus.com
> - Japan: jp.githubstatus.com
> - US: us.githubstatus.com
It'd make sense if there was a "you get what you pay for" attitude at MS re: public GitHub. It's not a good position for the free users, but, what else are you gonna do? Stand up your own? Retrain yourself on your SDLC in a new platform?
They need a competitor.
If you have a Data Residency tenant, you access it through an endpoint like octocorp.ghe.com
GitHub seems to try to use specific language here to avoid this confusion, because they are quite different products, but it seem to me they were named confusingly in the first place...
1) Criticism of being unable to achieve service is not a fault of the individual; it simply is a fault of the system. You can criticise the system, it's permissible. Especially if they have more resources than many countries and some of the best tech talent in the world on staff.
2) Their tech stack is shit, and they've gone on record for years defending it, quite arrogantly in some cases, as if nobody can possibly know anything unless they've done github (even if you've done things which scale, or someone comes in with an even larger scale, the people on HN will happily say "but it's not github" which is valid but not intellectually curious or open).
Azure is terrible and it's being foisted on the team: even if they found some technical people to put at the top who are saying it'll be ok: it is a pretty cruel platform to use.
I've personally had a few conversations about their choice of relational database which were handled pretty defensively, and I think we're all somewhat cognisant of their frontend rewrite.
It's a waste of time to rewrite the UI and push AI tools when you can't even keep the site lit.
I have nothing against the engineers- I don't know why people keep chiming in as if we're punching down at "lowly engineers" when the reality is that it's a management failure of the highest order.
They're a billion-dollar company owned by a trillion-dollar one... it's very hard to "punch down" at this system: nobody is going after the engineer, we're punching the fact that the system that is a defacto monopoly due to network effects is putting new features or pleasing their owners over the core offering.. How is that an engineering failure? That's an active choice by management.
This checks out. I once was at a conference where they (Azure) had a giant booth. A fairly well known person in the community brings me over to talk to his manager who is working the booth. "We should hire him, he's really smart." Within a minute of talking to this manager he says "You're a Linux guy? We do Windows." and physically turns away from me, conversation over. You know, fair enough, was an easy way to find that it wasn't a good fit. But the lack of curiosity about "what do you bring to the table" was pretty stunning.
Be curious.
edit: Clarifying "they"
If the context was an Azure booth, I’m still mildly surprised (they’ve long been invested in beyond-Windows) but not shocked.
(Edit: I forgot about the Actions stack. Some of that was on Windows. I was pretty far removed from that world and much closer to the classic Ruby monolith side.)
Seeing this happen in real time is helping me understand how authoritarian regimes/institutions/movements rise to power.
Very little discussion of any merit happens on these posts. It’s mostly bandwagoning and repeating the same comments they read on the last iteration.
Yet here we are.
I just don't feel comfortable with you defending the trillion dollar company as if we owe them something, or as if they're somehow the victim in all of this.
I can buy that there's more demand for service, but;
A) They are the ones pushing the AI hype (microsoft especially but github too)
B) These issues existed before the AI hype anyway
and, obviously:
C) We're not saying they're bad engineers, we're saying it's become a bad service... THAT is everyones problem, managements especially. We're not attacking the developers specifically, we're attacking the state of a core service that is failing.
I’m just saying that scaling is very likely the issue, no reason not to believe their own statements on that. And yes they are to blame for their own success here.
Or so I imagine office politics to be there. I've never worked ah Microsoft specifically though I have worked in corporate America at other large companies.
This is a flawed argument. There are many designers and frontend engineers there who have zero role in improving site reliability. They might as well keep doing their jobs, instead of having the CSS wizards and art school grads team up and try to crack Azure.
We've reframed this argument from the original "stop punching down" to, now "well, managements allocation of resources is fine because they have staff that would otherwise do nothing".
Thing is, I agree with the base of your argument, over the course of a quarter (or 3, or even 5..) the release of a feature does not mean that resources have been taken from the core.
However... it's been a really long time, and now we're hitting a critical point where the added load of AI, the rot that has been allowed to set in at the core, and the fact that they haven't been allocating staff to improving those pieces is hitting an inflection point.
I can't say for sure, as I don't work there, but I think if the trend is going lower for literally years: management could have changed course.
Those frontend designers didn't hire themselves and normal turnover is something like 5% for a healthy org: there was a conscious effort there. And those feature designers on AI can definitely have done work on reliability.
Managing and coordinating a bloated organization always has a cost and an overhead, from communication issues to technical inefficiencies.
And I also doubt the frontend/backend divide is so clear. I would bet quite a few developers are working on both.
You rarely see "outages" even if that what happens in reality, in marketing speak it's referred to as 'degraded performance' i.e. the cheque is in the post, your data is in the tubes on it's way, it's just slow! A business oriented lie.
Far more useful are the 'independent status pages' maintained by enthusiasts that are unaffiliated with whomever they are measuring.
unless, like this one, they:
- treat "some copilot chat models are failing" and "teams notifications app down" as a major outage, the same as git operations or actions failing... those are very obviously nowhere near the same operational impact and its dishonest to group them as the same
- aggregate downtime so that there is greater than 1 day of possible downtime in a 24 hour period. if 3 services are down for the same 1pm-2pm time period, that is being counted as 3 hours of downtime despite a developer only being impacted for 1 hour.
it would be cool to have an accurate status page. the only two options seem to be company-owned status pages (incentivized to under-state impact) and karma-hunter/meme status pages (incentivized to make as much red as possible for retweeting or whatever).
'GitHub Enterprise Server' is hosted on your own resources, not their cloud. It makes sense that it wouldn't have the same downtime as their cloud, but that's hardly relevant.
'GitHub Enterprise Cloud' is their offering hosted on their own resources and what I suspect most enterprise customers use. It's what I at $extremelylargecompany use. It follows the same uptime/downtime as their public non-enterprise offering.
You can tell if you are on the github enterprise with data residency because you will access github at a GHE.com domain rather than github.com. It definitely has better uptime than the public cloud but is not without its own issues.
It's a collection of many things. Some of us use a few things, some of us use lots of things.
I, for one, am mostly happy with GitHub and have been for the last 18 years I've been using it.
That said, GitHub Actions and Container Reg have been a bit... unreliable. This isn't to say all of GitHub is unreliable... just that these relatively new additions in GitHub's nearly 20 year history are a bit s** when it comes to uptime. I hope they can figure it out. :)
GitHub is not a mom&pop operation.
I expect the $3T company to handle the load, or at least place a prominent "only for hobby use" warning on top.
Also it's not a fair comparison because it's not necessarily the same code between the public and Enterprise...
You and I are in different domains. It's not daily, but I can't remember the last time I (in my company) went a week WITHOUT having to workaround some outage. Perhaps semantics, but I can "do work" through most of them, but that work isn't getting built or deployed in the same time frame it would have been had the outages not occurred. So "affected" is at least weekly for me.
> Just compare the GitHub status page for public GitHub vs the enterprise cloud pages.
I am not sure why that would be an explanation either - it could be that enterprise gets more time and money, whereas the non-paying free-riders naturally get less. This would also make sense from a business point of view (to some extent, though if only enterprise would use github, it would lose its status as main source code hosting website on the planet quickly; others are already waiting to nibble away at GitHub, the more Microslop fails here).
I can reply to an existing thread, just not create a new one.
How does something like this even slip by?...and why has it been like this for an hour?
EDIT: Oh, good, the issue should be solved in the next 3 or 4 hours. How lovely of them.
> Disruption with Grok Code Fast 1 in Copilot
> Incident with Copilot Grok Code Fast 1
> Claude Opus 4 is experiencing degraded performance
It doesn't seem fair to blame Github for this? There's nothing they can do about it?
Erase the severity and then present them all as “GitHub outages” or reduce it to an uptime graph.
I’m not happy with GitHub’s recent major outages either, but there is an ugly side of the pile-on where we’re getting these vibecoded attention seeking websites and social media posts to collect upvotes, likes, karma, and attention that blur the lines between small service degradations and total site outages to be more dramatic.
https://www.aakash.io/tech-chase/github-and-claude-are-down-...
I'm curious if most of the big players including eg Google do this thing of nerfing models or it's limited to more "smart" (read: black box models like ChatGPT.
> Across 170 days with at least one incident · worst day Thu, Nov 20, 2025 (1.1 days)
1.1 days total how is that possible? Scrolling over that day doesn't indicate the math behind the scenes - 1.3 hours single bullet point.
Also Nov 19 has a bullet point 1.3 day outage but total is 8.1 hours
I'm guessing that this site is taking the downtime in a given day across all services and adding it up, which would mean the worst possible day has 10 days of downtime (a day of downtime for each major category).
I haven't actually checked any sources to confirm there really was downtime on those days, but if we assume those numbers are true 7.8 hours + 1 day is about 1.3 days.
[0] https://www.githubstatus.com/ [1] https://mrshu.github.io/github-statuses/
Because the SLA likely doesn't consider some features of GitHub under the SLA, whereas an outage/issues for a single model is seen as problem on the Third Party page.
At a recent gig we experienced many, many GitHub outages that were not tracked on their status page, and we kept a log (i.e. just search in slack). After our business people argued with our account executives at GitHub we got hundreds of dollars of credits.
Then the business peopled complained because hundreds of dollars of credits is not worth their time. And so GitHub continues to have terrible uptime and nothing is done about it.
They expected us to log any faults and as you say the process wasn't worth it - even with massive outages - just for a few beans in credits.
GitHub has low availability simply because it doesn't cost them and they wear no legal or contractual damage from it.
If a competitor came to me and said, we will _pay_ you damages for the time your developers are offline not able to use our product to do their jobs, we would sign up immediately.
I made this one in January to help slice and dice uptime by incident category.
IMO, Claude is not fairing any better than Github.
It is a platform for CFOs to avoid having another vendor relationship.
It could very well be, that GitHub is not running on Azure yet.
Hosting forgejo is really easy as well. It being a single binary makes it really easy to handle with almost zero maintenance.
YOU NEED TO USE MOAR AI!
"survive underwater" what a joke. Yes, there will be good engineers there who will be sad to see it go this way, but they choose to be there, get paid better than 99% of humanity for that.
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
Then the load during the working days makes those ripples larger and into outages.
Fun fact: it used to be the case that GitHub was actually _less_ reliable if nobody deployed to it… there used to be various resource leaks that we didn’t see when people were deploying all day, since then the app wasn’t getting restarted constantly. After GitHub went down during a holiday break we had volunteers to deploy GitHub once a day during holiday breaks, until the underlying issues were eventually fixed.
or just a multifactor of both.
It does look like Friday outages were a bit rarer, which could be due to having a "no deployments on Friday" rule.
Add in new "productivity" tools that help you move even faster, with even less regards for how much you screw up (even though the tool could be used for you to move at the same speed, but with less screw ups), and an engineering culture which boils down to "Why not?", and you get platforms run by Microsoft that are unable to achieve two nines of reliability.
This is not limited to just pushing code but all the bells and whistles that github added as features under the assumption of some predictable growth are now exceeding the original plans.
I suspect a lot of their existing systems have to be re-architected for unanticipated scale, and it won't happen overnight for sure.
I don’t like this whole casting of doubt upon sources without providing superior or even alternate sources.
It makes it hard to discuss when one person presents a source and another says “I’m not sure that is accurate.” In a vague way.
What am I supposed to do with that? Research more sources that may or may not align with how you feel?
None of which excuses this. Can you imagine someone's reaction in 2017 if you told them that github would be below 90% uptime in 2026? It would be unimaginable.
Also go back and look at the unofficial status page from 3 years ago. It’s regularly above 99% and has been dropping steadily since then. Then in the last 3 months has dropped to below 85%.
I’ve been using github pretty much daily since 2010 and I never had a push fail or a repo be unavailable until recently.
https://www.githubstatus.com/history
And immediately find that there are numerous incidents that would show up on the modern status board as an issue but are reported with 100.00000% uptime on that graph.
One example:
https://www.githubstatus.com/incidents/bzj1hc2cnfkc
2018-07-16 17:32:53 - We are investigating reports of elevated error rates.
2018-07-16 17:34:27 - We are investigating reports of service unavailability.
2018-07-16 18:04:38 - We've discovered the issue causing connectivity failures and are remediating.
2018-07-16 18:26:48 - We're monitoring the site as systems recover. Some delays are expected as we process backlogged data.
2018-07-16 18:37:26 - We're continuing to monitor and work on further remediation efforts as the site recovers.
2018-07-16 18:54:21 - The site is stable. We are continuing to monitor and work through follow-up remediation efforts.
And there are other incidents with connection failures or elevated error rates during July 2018, but the linked graph shows "average uptime of all components 100.00000%" during July 2018.
Another from October (that also shows 100.0000% uptime)
2018-10-21 23:09:19 - We are investigating reports of elevated error rates.
2018-10-21 23:13:31 - We are investigating reports of service unavailability.
2018-10-21 23:43:55 - We're investigating problems accessing GitHub.com.
2018-10-22 00:05:37 - We're failing over a data storage system in order to restore access to GitHub.com.
2018-10-22 00:23:54 - We're continuing to work on migrating a data storage system in order to restore access to GitHub.com.
2018-10-22 00:43:12 - We continue to work on migrating a data storage system in order to restore access to GitHub.com.
2018-10-22 01:02:49 - We continue to migrate a data storage system in order to restore full access to GitHub.com.
2018-10-22 01:22:22 - We continue to work to migrate a data storage system in order to restore access to GitHub.com.
2018-10-22 01:41:19 - We are continuing to work to migrate a data storage system in order to restore access to GitHub.com.
>I’ve been using github pretty much daily since 2010 and I never had a push fail or a repo be unavailable until recently.
Looking back at their downtime history. Unless recently is within the last 3 years, it seems like you got really lucky.
They’re making political decisions based on what they sell vs what’s actually useful for their use case.
It’s kind of impossible to find out if this is true though.
This one is including external LLM services as apart of GitHub being “down”.
Personally my favorite is probably drone-ci.
I'd suggest not buying in too hard on any one of these CI systems and just writing shell scripts. Shell scripts are portable, and you can use whatever to trigger them.
Similarly, i see google releasing advancement after advancement in LLM yet i see antigravity sub where people are crying all time.
OMG it's a secret message!
If you're working on an oss project (ie distributing source anyways), probably better a good idea to mirror to GitHub as a backup anyways, let Microsoft pay for backup+bandwidth for you.
This website has no overused ai-generated animations and... I quite enjoy it. The original website[1] has a fade-in animation, big round cards, shadows, all the jazz you can think of, it's there.
This site is very readable, very honest and sober. I don't need to sift through buzzwords to figure out tiny details.
Thank you, OP!