GitHub Outage Tracker: Is GitHub Cooked?
isgithubcooked.com
isgithubcooked.com
I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figured out, but a little sympathy goes a long way. I personally wish them the best and hope their on-call people can go back to getting normal amounts of sleep soon.
I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.
I don't know the architecture or any of that, but I feel like there could be (and it's not like they would've really known this until the last year or two with the massive spike) separate infrastructure for paid users/orgs vs free the same way they make the distinction with enterprise.
I get the massive load changes that they are under over the last two years, but why does a bunch of vibe coded slop take down the same resources that my company pays for every single month and has for years? I imagine properly splitting that out would be an absolute headache and not worthwhile for them vs stabilizing the rest of the service, but damn it sucks when I get blocked at work because GH is down.
Of course it is not free of all management but for our use case it is working.
There are hiccups with the CI runners from time time but nothing major and we have another machine in another rack that serves as a backup which can be brought up ~< 20 minutes.
I know companies have long since tossed their expertise for hosting their own stuff in favour of SaaS but at some point its hard to beat the up time of a single machine.
Expertise isn't just "Do we have a sysadmin?", it's also "Does the org enable them to get their job done?".
It's a lot more expensive and has a good bit more limitations to the regular SaaS product.
I can have sympathy for the humans caught in the crossfire but only managing one nine of availability on a commercial service is not acceptable.
https://this.weekinsecurity.com/microsoft-wins-lamest-vendor...
I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.
And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for.
I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and without any knowledge of the “problems they’re solving”
I’m saying that GH is operating at a huge scale with (probably) lots of technical debt and a forced migration to new infrastructure.
I would be (and am) highly critical of leadership. I’m not going to make strong assertions about ICs without knowing their context. I’ve worked at a company with a sterling reputation for engineering excellence where brilliant ICs were kneecapped by poor leadership.
I think a lot of us, at one point or another in our careers, have worked with potato leadership that can be short-sighted or political. It isn’t a comment on the engineers.
Now the scale problems they're facing is because of those 2 actions of theirs. They didn't have to encourage such growth to turn a profit, and they still don't have to provide gratis services to slopware bots providing 0 value for society.
An outage is more like a shipping issue with the supplier, if it's owned wholly by them.
If any single company reached 1%, they would instantly be out of business, because that is not remotely acceptable, at a global scale.
Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.
Personally, all AI has done is add another layer of abstraction where “pseudo code” can be the source input and out comes the compiled program. You still need to think about the design and maintenance.
The final step is challenging; likely the most difficult part is changing all the code references and imports. Shadowing changes would be straightforward. Training your likely 25-50 engineers to use the new code review UX would likely not take that long.
Considering the wasted engineering velocity during Github outages, it's worthwhile to do even a partial migration. Github's action runners have in my experience, been the most fragile part of the platform. Given the ease of moving build and merge queue runners to alternates, it's a no-brainer.
Of course you may not want to manage the hardware yourself in which case you'd go for a dedicated server from a reputable provider (i.e. not microsoft/google) like OVH's RISE-XL, Hetzner's EX131, Scaleway's Core-9-L (or an equivalent solution from a smaller provider) for about 4000€/year.
Now let's say you add 8k€/year donations to your distro of choice, forge of choice, and other FLOSS projects and 8k€/year for backups. You got the whole thing running sustainably with >99.9% uptime for 20k€/year or 20% of the original bill.
Core parts of the product, like navigating to individual files in a code review, are broken
I think this is a good argument to underline "It's not _just_ the scale". Adding to this, the Github Code Review experience is kind-of broken, the way comments/threads are stacked in the PR overview has not improved, pagination isn't really a thing, and these issues are age old. Hopefully, one day, Github will mature.
Can you comment on that?
Enterprises are benefitting from the same AI and are increasing their volumes as well
unless the firm was prepared to see it through.
But we pay enterprise license and GitHub is a big dependency in our software flow.
If this continues to be a problem as an enterprise product they need to do something. Otherwise theyre are going to to start losing business
I used to be on-call in a high-traffic environment where single customers pushed more bits than entire nations. I chose the role. I didn't want people's sympathy, if anything, I wanted them to complain to management.
If it gets too bad they can quit. Maybe that would be for the best, just wear the thing down until it outright fails and no one wants to touch it. One less bullshit service sucking all of the oxygen out.
They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft.
And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.
I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them.
I have no sympathy.
Ding ding ding, we have a winner. I like AI. I work for an AI company. Still, Microsoft aggressively pushed GitHub users toward Copilot. They don't get to do that and complain about increased volume from AI-generated changes.
No Copilot + reasonable operation: the way things were
Copilot + reasonable operation: Well done!
No Copilot + being overwhelmed by AI commits: Sympathy.
Copilot + being overwhelmed by AI commits: "Where did that petard come from that's hoisting us?"
1. Github has enterprise users who paid for the service, their day job requires Github to be available and working
2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code.
It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.
But I think they’re in a tough spot. GitHub has historically been a huge supporter, proponent, and provider for open source projects. Engineers are difficult customers, to say the least, and the community would likely freak tf out of the segmented traffic.
The logical, pragmatic, and justifiable answer doesn’t always align with your market.
They are in a tough spot cause they want to continuously ingest all the data to train their LLMs. Any fork or decentralization at large scale of git is going to impact the training pipeline.
A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas.
When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire?
Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.
No.
Their leadership went all in on AI and in the last blog post essentially admitted some missing test coverage for a critical path. Time to learn lessons and fix your vibe coded shitslop and stop using "user graph go up" as some kind of excuse.
Yes, that is scale. And yes, that's not actual requests per second. But it's the sort of scale that big (and even mid-sized) tech has known how to deal with for decades. Microsoft doesn't have an excuse.
Bit of a Clownstrike this has become!
As I said before (and it was not a popular comment), it's easy to be an "armchair quarterback," with these services, as I think the brittleness was already baked in, and just waiting for the right time to crack. The only true way to have a robust platform, is to design something that will scale, from the start, and many startups don't do that, because they are feverishly trying to get out an MVP; even if it is a mess of bubblegum and baling wire.
They always say "We'll get it done right, once we get funding," but that never happens.
But my sympathy is limited by the fact that MS paid a lot of money for this, quite a while ago, and that is one company that knows all about issues of scale. They should have seen this coming.
We certainly do not.
This is a multi billion dollar corporation. History of robbing and stealing from others of their labor or IP. History of enshittifying once great services.
That is an active choice they are making, over and instead of any reliability for the rest of their users.
It is a problem they are embracing, and actively encouraging, for themselves.
I don't have sympathy for them. They have disrupted my work so much in the past month that it's ridiculous, and my company likely pays them millions of dollars. And you can see from this site's plotting that there is a trend towards more frequent and more critical outages. It has been extremely bad the past month.
However, YouTube did well where Google Videos did not. Who knows what it would be like if big G bought GH.
It's probably not fair to the people who are left but alas, this is apparently how the world works.
They can simply throttle access for everyone to a push/merge frequency which was normal for human-maintained projects.
They should look at how MMOs battle cheating/botting and apply similar countermeasures (it's actually quite absurd that MMOs which are actively attacked every minute have a better uptimes than Github, and on Github the problems are even self-inflicted, not caused by external attacks).
But of course throttling bots to human-level push/merge frequency wouldn't be compatible with the Github leadership's 'AI bullishness'.
What we are seeing here is a trillion-dollar company screwing over its existing well-established customer base in a desperate attempt to get some crumbs of the current hype cycle. GitHub isn't "dealing with" scaling issues, they are intentionally causing it by shoveling giant piles of coal onto the AI fire.
GitHub's issues would be over tomorrow if they were to put reasonable rate limits on their services. Stop the bots and GitHub is no longer permanently overloaded. It is an intentional business decision not to do this, and instead welcome bots with open arms - to the detriment of their existing paying customers.
I do have some sympathy for the low-level GitHub staff who have to deal with this - especially those who aren't being compensated with insane Bay Area salaries.
How about no? They are owned by a company that is neck deep in foisting AI into everything, AI that a lot of us don't want in everything, they are reaping what they seem determined to sow.
So it goes.
Also known as the AI chickens coming home to roost. Microsoft was the first to use code uploaded to GitHub to train AI models.
that's the mentality i was taught at faang
What if they figured out what their platform can actually handle, provided the contractual level of service to their paying customers first, and put some god damn limits on usage.
Like they should just stop signups if they can’t handle the capacity yet. It’s a business decision, not a technical decision, to decide to screw over customers instead of adding limits based on your actual capacity.
Does nobody use GitLab or Bitbucket or whatever? Why aren't those full of high profile outages as much?
> I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers.
I'd kind of expect that for all non-paid members will get rate limited in some ways, sooner or later. Same as what happened with DockerHub, they're probably just trying to hold out due to the reputational hit such a change would carry.
I was basically given the same response that blizzard gave that question; "you think you want that but you don't".
Imagine YouTube Classic...
I've started suggesting to my CTO that we move to something else because we do pay the Github bills and yet we are also affected by other people's activity. Avoiding that is not a hard engineering problem for an alternative.
A world where a record amount of code is pushed to GH but a large amount of paying customers leave would be "cooked"
1125 incidents / 126 months ≈ 8.9 incidents per month, not 24
still terrible, but why such an obvious error in the first sentence...
GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month
That worst day is likely an overlapping incidents accounting issue; I tried to account for overlapping incidents in another view but probably failed to port it over there.
Should be fixed soon!
Looks like they fixed it already
const incidents = e.detail.incidents;
// ...snip...
const now = new Date();
const threeMonthsAgo = new Date(now);
threeMonthsAgo.setMonth(threeMonthsAgo.getMonth() - 3);
// ...snip...
var currentFreq = recentIncidents.length / 3; // <- We out here, smoking these guns with our homeboy Claude
// ...snip...
var earliest = null;
for (var j = 0; j < incidents.length; j++) {
var dd = new Date(incidents[j].started_at); // <- eventually incidents[j].started_at is "2016-03-01T07:07:37.000Z"
if (!earliest || dd < earliest) earliest = dd;
}
// ...snip...
document.getElementById('n-since').textContent = earliest
? earliest.toLocaleDateString('en-US', { month: 'long', year: 'numeric' })
: '?';
document.getElementById('n-rate').textContent = Math.round(currentFreq * 10) / 10;
#n-rate has nothing to do with the #n-since month, it's just the last trailing 3 months. And even then, it's sort of underbaked? It's moving the date back by 3 calendar months not taking into account differing numbers of days, so it'll under-report short months.
I wouldn't trust the stats here.
Edit: whoops, author updated the template while I was writing this! It now says "Over the last 3 months", though that's still calendar months.
Github was bought by Microsoft, whether they wanted to acknowledge that internally or not. Microsoft went deep on LLMs, and specifically on LLMs for coding use cases. They must have recognized that LLM generated code and PRs would effectively DDoS GitHub.
I can only assume they simply didn't care, likely driven by greed.
For security reasons, I would love a low feature, carefully engineered, offline github-like system. Something tuned for agentic behaviours, with an understanding of hierarchical agent identities.
Migrating to something else has been raised as a concern during every engagement with clients and prospective clients in the past year.
"Do you use GitHub?"
"Yes though we'd like to move to something else, we just don't know what yet."
As soon as the next big thing shows up, they're done. And they know it.
1. Hates the status quo
2. Only uses a handful of the 10,000 features the status quo has
3. Uses a different handful than everyone else
Because that model kept Excel as the center of buisnesses around the world for 30 years and counting
In reality, Excel is the only reason Microsoft still exists.
Excel is how they lure you in. Excel really is the best spreadsheet, its the only one every doc you are sent works with, but also the one people know inside and out. And if you are into Pivot tables, PowerQuery, etc there is just no escaping it. You are utterly stuck.
But here is how they get you: its not much more to rent the whole office suite compared to just Excel. And as long as you are into it that far, Outlook and Teams are much cheaper than GSuite + Slack. And still...okish? No, not really. But we pretend. And it wouldn't happen without Excel.
And it may be even worse in EDT, which is currently in effect!
I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability
GitHub Actions is god-awful. Have you ever used any other CI tools?
Would be happy to be proven wrong!
They don't. Self-hosted runners are still free. They backed down from their plan to charge after receiving widespread backlash.
Are there any?
you are the problem
> you are the problem
Hey, arccy.
What a pathetic one-liner reply!
(update: since it was being downvoted anyway, removed the explanations part and just left the pissed off longer knee jerk reaction response to another shorter knee jerk reaction)
Coming back to your pathetic quip: what kind of piece of work one has to be to reply in this somewhat apple-fanboi-esque way (where someone says "how to fix X problem in Y" and pat comes the reply "why not just buy Y-latest, it's just $349") and what kind of unidimensional world you live in where you believe someone wanting some free and private git hosting is the problem? Ffs.
So again and with all due respect, please shut the fuck up! Because you don't know me, you don't know where I am coming from, you don't know the why, and you don't come across as someone who would ask or even care about knowing.
https://github.blog/news-insights/company-news/the-august-17...
Will either cause an outage next month?
I was ready to click...
MS is trying to catch two hares at the same time, which has never worked: Slop hosting for the masses and workflow provider for the relatively few. You can't have both. But MS won't make the hard decision. Those who need workflow are on their own to find alternatives.
The contribution graph as an outage calendar idea is a commonly recurring one :). I definitely saw it somewhere else as a static asset first before I made this site.
Any dev can just have their own git mirror and work there, but replacing github is not really feasable, not one of the other services come even close.
If github is down for say 2h i dont really care. I can live with that. IF i did care i would just host my own.
This is why I foresaw that centralizing everything to GitHub was just generally a bad idea 6 years ago. [0]
Now that there is no CEO of GitHub, there is no point to GitHub improving.