Incident with Actions, Pull Requests and Issues
githubstatus.com
githubstatus.com
I feel bad for the team, but will be great growth opportunities for all.
This is likely a pricing failure. Systems breaking due to over-usage is like going to the store and finding that they're sold out. You're much less likely to sell out if you charge more. Meanwhile, GitHub not only didn't raise prices, they cut them at the beginning of the year: https://github.com/resources/insights/2026-pricing-changes-f...
And this is beside them cutting the Pro tier from $7/seat/month to $4/seat/month back in 2020 and not touching the price since: https://docs.github.com/en/get-started/learning-about-github...
Bear in mind as well that the Pro tier includes 3,000 minutes/month and the Enterprise tier ($21/seat/month, and no published minimum seat floors) includes 50,000 minutes/month. $21/50,000 minutes/month for a solopreneur is practically giving away the farm.
GitHub needs to drastically increase pricing, at least for Actions; their current pricing is practically a WTO dumping violation.
So maybe them providing service at literally all costs to themselves is a smart strategy long term.
They wanted direct access to the repositories for other purposes than serving the customer. Such as training data for copilot, and who knows what else.
Pricing low helped ensure access to that data.
"Should" doesn't imply "does". GitHub is competing with all of the other Microsoft products that need Azure compute, in addition to Microsoft customers. If a customer is willing to buy the compute at a higher margin than the internal chargeback rate, it's definitely not a given that GitHub is getting that compute over the customer. It's hunger games for compute everywhere right now.
Problem is while we have a visceral feel for what the "value" of GitHub is to human users for human usage, I'm not sure we have a feel for what the value of the agentic equivalent is.
IMO it's probably very high on paper if you think through what kind of work is unlocked at agent speeds, provided you have a very capable work-execution fabric.
But since we're anchored to human pricing, whatever they propose will feel expensive.
slopreneurs are the issue
> That's a seriously insane number to keep up with at GitHub's scale.
What exactly is it about GitHub's workload that makes us feel like the specific 23k things happening in that slice of one second are tantamount to insanity? In my experience, much of GitHub activity is changing labels on issues and posting plaintext comments.
Technically speaking, this rate of things happening per second is well within the territory of what one large server can handle. I know it sounds wild to imply that we could run all of the GitHub public cloud off a single machine, but how crazy is it in reality when we start to consider the actual I/O bandwidth that is available in a leading edge server platform? All of VisaNet circa 2009 could run off a single mainframe. That's global. It is plausible they still operate this way.
Virtually 100% of the aspects of GitHub that would quickly saturate I/O on the transactional core can be offloaded to horizontal systems like blob storage and CDNs.
> What exactly is it about GitHub's workload that makes us feel like the specific 23k things happening in that slice of one second are tantamount to insanity?
>
> Technically speaking, this rate of things happening per second is well within the territory *of what one large server can handle*
Just as a mental exercise, consider what is happening when a GH action kicks off and everything involved in that.
The amount of write I/O we're talking about here is absolutely massive just on the GH side of it.
A Netflix or Hulu serving videos isn't the same thing because that is a read-heavy workload that works great in CDNs. GitHub has massive amounts of write volume.
Be careful what you wish for, because we might get a toll system for humans instead.
We all have businesses to run, this downtime costs us money already.
Just backseat gaming but a lot of those issues could probably be solved if they'd (temporarly) reduce the free tier. I'd say simply dropping Actions from it would reduce the load by a lot because then most vibe coders wouldn't bother integrating a CI/CD and either just use local pre-commit scripts or stop using those entirely. Not using a CI also drops the requirement for using PRs on solo-dev projects so they might also reduce load there.
This system worked pre-LLM and turned github into the de facto _hub_ for software development. It was so good that microsoft, a company previously known for its "developers, developers, developers" stance decided to spend a cool bill on it.
The proposed suggestion is that we all must make sacrifices for the machine god: Developers shouldn't write code, only correct the machine when it's wrong, mathematicians must read the slop and derive new science from it and regular people must refrain from talking dirty to it so then
why the fuck not give it our water and power too?
Majority of public utility consumers don’t give a crap about GitHub or AI. They’d likely tell you to kick rocks if you proposed any type of ration or limitation for your assertion that “we all must make sacrifices for the machine god”. Data center development is getting plenty of community pushback for exactly this reason.
Your train of thought is probably more average amongst GitHub user population but I fear is wildly off average when compared to a general population/the public.
Increasing the cost from zero to nonzero is a limitation, not a cost adjustment.
> Increasing the cost from zero to nonzero is a limitation, not a cost adjustment.
That is nonsensical. "Increasing the cost ... is not a cost adjustment". It's not a limitation if you're willing to pay the new cost. Thus, not a direct limitation. It's optional based on how much you value the service. Doing the same to water/electric would just be increasing the rates and letting supply/demand economics take over, you are saying people need to be forced to use less without mentioning any cost adjustment... not clear if you know what a cost adjustment means.
If Microsoft had brains they would leverage it to become the de facto file system for all ai agents before cursor or anthropic figure it out.
That being said, they are definitely even less reliable now due to demand.
The demand in question is upper leadership's demands to do more with less and do it faster and who cares if it works out worse for a lot of people at any point?
I like GitHub (maybe liked?), but their up-time isn't acceptable and I don't get the "poor little Microsoft/8B acquisition" story either.
You could say they should separate the human users from the slopsters in any tier. This could be done with some quota/rate limiting model.
But, if they are either on the fence or believing that agent-based development is the way forward, then that new reality is the one they would need to support. Cutting it off will send those customers elsewhere.
And, if you've managed to handle the new workload, it is basically in the noise to also handle artisanal human developers in the background on the same solution...
Source: enterprise cloud customer, it's broken all the time.
There is GHES onprem which indeed doesn't get affected by these outages, but it's shit.
When it became apparent that this growth was on the trajectory, instead of doing something to limit it and thus preserving the reliability of the platform, GitHub (or maybe Microsoft) decided to embrace that growth and allow their own platform to fail. They didn’t need to do that. And they shouldn’t have done that. They deserve all the criticisms coming at them for doing this.
In this case most simple explanation is inaction. Yeah, it was coming, but nobody was exactly in position or cared enough to do something about it.
Make Github Actions a paid-tier exclusive and this problem goes away.
GitHub are too scared to do that after the backlash around charging people for their own runners, but this would be a very different thing, simply stop giving free actions to people who have never paid them a penny.
I had no idea they charged you to run your own runner. I am immediately of the opinion that is wrong.
Conversely, I would have no issues if GHA become a premium product. Or maybe "your actions only run when we have spare compute". I've personally given Github money (back when private repos cost $$) and wouldn't be concerned about doing it again.
That is to say, I think you've nailed it.
This is a real problem for acquiring customers. CI/CD / Actions in general has a low initial perceived value and a high perceived value after use while being primarily a commodity product (serverless compute). To acquire customers they need to offer a free tier, to make money they need to charge their heavy users more, but because it is a commodity their heavy users will self-host runners or jump ship.
Action minutes is a really good proxy for value gain from cloud git services, where repo count, user count, file count/size, etc isn't.
Its a fun pricing puzzle.
I think removing the free tier altogether would be a bad move, especially given that some important open source projects rely on it. But they could possibly lower some of the limits, especially for new accounts, with a way for open source projects to apply for more, and maybe have an automatic scale up for projects that are more popular, by some metric. And maybe limit the free tier to public repos, and maybe even only repos that use an OSI approved license or something.
My company has heavy handily moved from self-hosted version control system to github and now we’re suffering more outages. Though the self hosted system had less features, and we needed to maintain our own jenkins infrastructure for CI/CD, at least we really only suffered from very major cloud-provider level outages
You said it.
And it is also significantly more difficult to retrifit that design onto a running system than to design it like that from the beginning.
The solution, of course, is not to simply start charging for all actions, because then everyone would wake up to all sorts of mystery charges. They'd need to notify that forked actions would be turned off and then opt-in for paid actions.
(I know because I recently hit it)
Sounds more like a bloated, vibe coded stack that has to be unclogged every 2 weeks.
Besides being more reliable, it's just faster to run on a machine you control.
I would not underestimate how strategically important that will prove to be for Microsoft moving forward.
I have a few GitHub actions that have ran for years without issue. You add a yaml file to your repo and it’s done.
As a hobbyist, I’m happy.
I would never do any real work with actions though.
If you're a large project, companies generally get more value paying for SaaS and letting devs do dev, rather than troubleshooting Jenkins for the billionth time.
The real value self-host brings is somewhere in the middle, where you need paid-level features with 0 fiat (and can pay in time instead).
It's like everyone collectively agreed that assembly is bad, yes, but building CI/CD in assembly is actually good.
I think 2) is much more likely than 1).
So why stick with git? Is it feasible to make a new VCS that scales better if you start from scratch?
Obviously getting adoption of something else would be a challenge. But Github has enough market-share that they could plausibly pull this off. Particularly using costs or reliability to incentivize migration.
(Not saying that this would necessarily be good for the community in general but I think good for Github.)
I'm at a point where I'm going to set two permanent branches on the remote, one pertaining to each environment. The targets will poll for the latest commit at their configured branch and perform a rebuild internally if a change is detected. Assuming git core ops are still alive, it is possible to keep this running. It's also easy to migrate to a different vendor.
It's on Microsoft Azure.
I think GitHub was rolling everything themselves when Microsoft bought them. Ever since, MS has been working to strategically shift to Azure but that's a difficult and careful process that takes time.
In the meantime, the AI boom hit and usage went through the roof, crippling aging non-scaled infrastructure that hasn't had a chance to be modernized yet.
A lot of issues might very well be "we broke something while migrating", but I think a lot are also simply capacity issues on legacy.
For someone the size of GitHub, the cloud is not infinite or infinitely scalable, and running their own datacenters is fine and completely normal and not at all "aging" or "crippled". Like the meme says, the cloud is just someone else's computer running in someone else's datacenter.
The traffic increases they cite as the reason are absolutely the reason. Those numbers are insane.
It should not surprise people when things start collapsing when there have been little falsehoods and random bits of senseless codes written by agents all over the place.
Edit: just to be clear, i'm not saying this to be snarky, I'm saying it because the comment reads like he is a user of pushgate.dev, as opposed to the owner
GitHub Actions is only OK for relatively simple CI needs, or if your project is open-source and you enjoy having free CI. No major professional/enterprise CI need should depend on it.
Now, if you're not a paying customer, maybe you just ignore this and chalk it up to "I'm getting what I paid for". And if you're a paying customer, you migrate away. What happens to GH then? Well, you (and other paying customers) are probably only a small part of the first of the identified causes, and not at all a contributor to the second. So it's not like you moving away will make GH more reliable, but it does lessen MS's motivations for fixing things: GH as a business unit works by having a free tier and a premium tier and that's fine as long as the profit from the premium tier exceeds the cost of the free tier.
THat's long winded way of saying: I wonder if you/other paying customers leaving is the real start of a GH death spiral. Less paying customers == less MS investment == less paying customers == less MS investment, etc.
For us it's not just about reliability, but risk exposure. The last couple of years really changed what our security model looks like. And it's not in public clouds, on SaaS products. LLMs are killing SaaS for sure, not by replacing them but making them a concentrated risk.
There are no more excuses to self host. You can now ask Claude to do it for you, given that “coding is solved”.
Or even vibe code your own GitHub at this point.
Can’t tell if the irony is intentional or not
The entire US economy is tottering due to a lack of anti-trust enforcement.
Even The Economist, an oligarch-friendly magazine, has been hitting the alarm bell on this for over a decade: "America needs a giant dose of competition" https://www.economist.com/briefing/2016/03/26/too-much-of-a-...