Incident with Issues and Webhooks – Resolved
githubstatus.com
githubstatus.com
Very curious to see what the future holds for Github.
In particular Github, with it's copilot-next initiative, has probably so much AI generated code inside today that fixing all this new performance problems will need lots of human developer brains.
In the same way you need to be 10x better for someone to consider switching to your product, if you get 10x worse your competitors get a free 10x by just standing still.
Something creative like separating their free and paid tiers may help them. I suspect the fact that all of this is happening to them along with their migration to Azure is probably complicating their ability to adapt their infrastructure.
[0] https://github.blog/engineering/architecture-optimization/bu...
I don't have a clear idea how that value can be captured, since it's going to be 90% AI generated code that anyone can scrape (public projects) or can't be used (private projects), so perhaps you're right.
Even if that is true, unless the value of the data corresponds to near-term revenue, then eventually the cost may simply not be possible to meet. Or for that matter, the capital to manage the increasing load may simply not exist - it does not matter how much valuable data you have, if the supply of hardware cannot keep up with your demand.
Also, I suspect that most of the "data" obtained by the incessant hammering on GitHub is not very valuable. Most business code is routine, and getting Copilot to help out with generating enormous amounts of it may not contribute much in return.
And it isn't even clear yet if the AI generated code is even particularly valuable since it's legally ambiguous as to whether or not any human ownership can be attributed to it.
The USPTO has declined copyrightability for genai artwork, it's only a matter of time before the same question comes up about code.
However, works with significant human input are covered by copyright, and most code does have such input. Human review, and correction is very common. There is a lot of AI generated code out there, and there are no cases challenging the copyright on it.
You also need to look beyond US law. Software is a global business and most software businesses do not want to write software they can only sell in certain countries.
Legislation and court decisions still pending. There are numerous lawsuits about copyrigtability of output, and right of use of copyrighted work by LLMs, and both could have ramifications for code. I don't see how it's materially different to tell Claude Code to write you a function fetching an entry from a database, and telling ChatGPT to generate you a picture of a unicorn riding a bicycle. Both have the same level of input (desired end goal), both might go through review and updates (no, pink unicorn; no, cache the database connection).
Legal challenges over code copyright are relatively rare nowadays, so I wouldn't take lack of high profile lawsuits as proof of legality / copyrightability.
And yes, this will also depend on jurisdiction. Court decisions or laws can change that. Litigation over copyright infringement via training and reproduction is ongoing in multiple jurisdiction, and it wouldn't be shocking to me if at least some decide that it is indeed copyright infringement to pirate content to train LLMs that can reproduce it.
Pages 27 and 28 of this are relevant to this: https://www.copyright.gov/ai/Copyright-and-Artificial-Intell...
How about if I write 100 lines myself, turn the AI features on, vibe code 100 lines, and repeat this for five cycles? Half the functions are AI coded and half the functions I wrote myself. How about if I just tell Claude to write the program?
And what if I tell Claude to write the program, and then spend six months tweaking most of the lines of code?
I struggle to see a specific and obvious point where a line should be drawn. It seems intuitive to me that if I spend at least a few days worth of effort on a code base (whether tweaking, correcting, or directing AI to do targeted refactors), that is meaningful human authorship even if it has thousands of lines of generated code.
I can, however, acknowledge the fairness that something which is simply one-shot output probably shouldn’t merit protection. But really, in any of these cases, it’s going to be pretty hard to prove after the fact exactly what the proportion of generated code to human authorship is, so idk how a court will really tell whether a repo with 20,000 LOC is one-shot or actually had a person spend a few weeks tweaking it.
Why should this be any different than when telling/paying a human to write the program?
You're free to enter an agreement assigning all rights to the employer or the worker, to license the work ir/revokably and/or non/transferably. There is no need to wait for a court decision to understand what the results will be.
As if they cared about that
The value is probably in knowing which AI-generated code ends up being pushed or discarded, which can't be derived from public projects. This information can then be used to finetune the next big model so it only generates the "good" code.
Do Github terms entirely prevent them from making use of data in private projects.
Usage numbers is the PR reason. Vibecoding insanity in Microsoft is the more plausible actual culprit.
Not necessarily, a few years ago they had some crucial information stuck in a single MySQL cluster (so write constrained) and were working on sharding it, but struggling: https://github.blog/engineering/infrastructure/partitioning-...
GitHub, for example, seems to implement the main repository /pulls page as a search query, which is hinted at by the prefilled search bar and was mostly confirmed last week when the search backend failed and pull requests didn’t load. But it could have been implemented as a plain API call that just loads open pull requests, and that API exists and did not go down.
If GitHub focused a bit on identifying their top 95% of high level operations (page loads including resulting API calls, for example) and making them efficient, I bet they could get a 5x or better reduction in backend load by simplifying them.
(Don’t even get me started on the diff viewer. I realize that much of its awfulness is the horribly inefficient front end, which does not directly load the back end, but I expect there is plenty of room for improvement. The plain git command line features are very fast.)
Microsoft bought Github and migrated to Azure, which is explains the findings. The query performance was fine before they started serving from Azure.
I mean honestly, as though there isn't one single person competent enough to read some logs and horizontally scale a few read only dbs to meet demand? That's not it
This is the opposite of my recollection, actually. I distinctly remember having conversations about Github struggling to scale well before MS was involved, and people claiming that MS had somehow saved Github because it had stabilized and begun adding features again.
> The query performance was fine before they started serving from Azure.
This may be correct though. The Azure migration seems more aligned with the timeline of struggling to scale.
Do you have any sources to back your claim up? At what point did Github fail to scale their search endpoints?
> This may be correct
It is.
The page also notes it obtains its data from the official status page, but big tech companies have been known to under-report outages. My general sense is they've gotten better about this in recent years; if so, that means historical data will give an erroneously rosy picture of uptime.
You said "I can't really remember any significant downtime before the Microsoft acquisition and the data supports my memories", but my memories differ (as do other commenters), and the accuracy of the supporting data seems questionable.
In other words, a centralized version control system built from the ground up to operate at scale would do far more for scalability than anything GitHub could possibly do to optimize their Git operations. Every major tech company (Amazon, Meta, Google, etc) is already doing something like this internally.
Though this would require people to start using a github-specific client rather than the traditional git+ssh. (Though the github client could still maintain a git repo locally, for compat.)
Considering all the ci/cd pipelines, PR & issue discussions, social media tracking, rich data and else that github hosts if their true issue is the actual meat and potatoes of running git I would be gobsmacked.
It is bizarre to me that so much of their tooling defaults to acting across the whole of github data points without guiding the user towards (or even making available as far as I can tell) a way to easily scope requests down outside of a complex search filter.
These are in the top left hamburger menu from the Home dashboard (edit: actually on all pages).
There's probably a fair argument about how discoverable these are (especially given their labeling as "All Issues" and "All Pull Requests") but that tip is quite helpful to me personally. Thanks for sharing it, I really appreciate it!
I'd still love something a bit more obvious and intuitive but if it's just a UX failure that makes me feel a lot better.
I would have believed this until last week when they had a little banner informing me that I might not see all the PRs but that they really were there and that I could use such-and-such API to find them myself.
If the direct API existed and was working, but the web UI wasn’t seeing the PRs, then presumably it meant that the web UI was not optimizing the default trivial search query to use the direct API.
Platform activity is surging. There were 1 billion commits in 2025.
Now, it's 275 million per week, on pace for 14 billion this year if
growth remains linear (spoiler: it won't.)
GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week
in 2025, and now 2.1B minutes so far this week.
So we're pushing incredibly hard on more CPUs, scaling services, and
strengthening GitHub’s core features.
They also had a recent blog post about availability: https://github.blog/news-insights/company-news/an-update-on-...
I don't envy the scaling issues the GitHub engineers are facing! #HugOps
Not sure if something similar exists for NPM which is big for all things JS.
I so wish this wasn't the case.
More likely pulls and pushes, and, naturally, the ci minutes they identify as the main issue.
Their current path to resolution is to migrate their codebase to a new language[2], continue to drop their inhouse ops for Azure resources and get off MySQL. Maybe one or two of those steps are legitimately a good idea - I don't have an inside scope - but technology migrations are always fraught with issues. It's quite possible these changes are just a result of them vibe-coding a mature codebase into a new language.
1. https://github.blog/news-insights/company-news/an-update-on-...
2. I'll grant that Ruby isn't the best language to use as scale but I think we're all old enough to realize that language choice is far less impactful on performance than code quality.
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
https://www.kunalganglani.com/blog/microsoft-fedramp-failure...
The recent blog post you're linking to mentioned moving data only for webhooks off MySQL, not all relational data used by the entire site; and moving "performance or scale sensitive code out of Ruby", again not the entire codebase.
Do you have an official source suggesting these migrations are more comprehensive than that?
"bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL)"
"Similarly, we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go" (in a paragraph specifically about "critical services like git and GitHub Actions")
Both of those sound highly targeted to me!
That paragraph read, to me at least, that the initial targeted changes were just the tip of the iceberg and that much heavier lifting than initially budgeted were now in scope.
I'm sorry but I really don't see how you're drawing conclusions about this meaning a move off of Ruby and MySQL entirely. That's a huuuge logical leap away from what is written in this post, and you originally stated it in a way that indicated this was a fact.
1. Models have got way better, which means you are far more likely to get something working. I know I used to have little 'tool'/'weekend projects' all the time that wouldn't get off the starting blocks before, now it takes a few minutes often to build them, and once I've built them I tend to want to have them saved on github. Quite how useful they turn out to be is another question though...
2. Related, because the models are a lot better I can generate far more code per unit time. On Sonnet last year I'd have to babysit the model and constantly 'steer' it, which meant a lot of the CC time was actually me reviewing it. Now with Opus4.7 it can often just churn away for 10-30minutes and get something reasonable.
3. Most importantly, just the volume of new users to coding agents - loads of new developers shipping far more far frequently.
4. Many users who were not on github, now signing up and pushing code to it. "Vibe coders" basically who don't have SWE experience and their agent tells them git would be a good idea.
Each of these would be a big increase in scale, but combined it is vvv high
December 2025 is considered by many people to be a major step function in agentic coding (both due to improvements in harnesses and LLMs themselves). I know my coding has forever changed since then.
Before I was basically always hands on the keyboard while working with AI. Now I'm running experiments with multiple agents over the weekend, only periodically checking in if they have any questions or need further instruction.
The last quarter is where I personally first started to see how this was all going to change things (despite having worked on both the research and product side of AI for the last few years).
> I have a feeling they've tuned the models recently to commit more often, which gives the illusion of more work being done.
Agents certainly are committing more often, but I know, at least for these projects, there really is work being done. An example: I had an agent auto-researching a forecast I was working on. This is something I've done manually for over a decade now. The iteration process is tedious and time consuming, and would often take weeks of setting up and ultimately poorly documenting many, many experiments to see what works. Now I can "set it and forget it", and get the same results I would have in hours (with much more surface area covered and much better documentation). Each experiment is a branch (or work-tree) so yes there are a lot of commits happening, but the results are measurably real.
I often think the big divide related to the success with agents is whether or not the quality of ones work can be objectively measured. For those of us doing work that can be measured, the impact of agents is still hard to comprehend.
If you are correct , and GitHub is scaling its compute mostly as a reaction to this externality (agents churning through code that will mostly be discarded), then you can look forward to getting billed for your usage. After all, it is hard to build a scalable system without back-pressure.
But if it ends up costing extra for GH, especially for work usage, then it's just a simple calculation of "is this worth it?" which I suspect for most cases will be 'yes'.
Once the landgrab-stage flat-pricing goes away, it will become a case-by-case calculation because unsupervised agents can (and will) run up your billing with zero understanding of the business value of what they're instructed to solve.
What kind of products/services are you building where you aren't able to tie your eval suite to business value? If you can't, then why are you building whatever is it you are in the first place?
By far one of the biggest changes I think we'll see in things being built by agents is reducing the gap between code and value. The first stage is to start making it possible to measure quality (evals) and the second stage is to more closely align measurable equality with value. The business value of the tokens spent on my team was discussed my first day.
> Once the landgrab-stage flat-pricing goes away
Aside from the above point, I'm already running local LLMs on my homelab that, while not quite what I want for truly production work, have been able to iterate on and solve real, non-trivial research tasks for effectively zero cost (energy cost was roughly on par with running an old light bulb).
The way open, local models have been developing there will be many cases where if proprietary providers over-charge it won't be a deal breaker to just switch to local models. Not to mention that there are plenty of open, but non-local models that are already 5x cheaper and roughly on par with the mainstream model providers.
There are no evals in my org that can quantify the value of a proposed feature, rank it against ongoing support issues that pop up, or know when to stop expending effort when no solution has been found or too many unknowns crop up. We still rely on natural intelligence for that, and haven't YOLO'd (ha) on Independent agents. I'd rather quit than spend my day herding agents and have my job reduced to just a code-review monkey.
Benchmark evals are at least 3 degrees removed from actual business value - maybe less of your tasks are repetitive. None of the harnesses I've used have a sense of a compute budget - outside of Boolean think/no-thinking modes.
The start of basically any project involves building up and documenting context around the project itself (and for a new company, the organization itself), this is kept at multiple levels of granularity (cross project, project specific, task specific, and human readable documentation). All experiments are planned out and documented as they go.
This becomes extremely important because after a weekend of running experiments stakeholders (and myself) often have questions, with everything in memory or some other stored context it's trivial to get answers to all sorts of questions.
Maybe it's because of this, but in both Claude Code and Codex I haven't run into any issues with models getting "dumb as a rock", even after compaction (or occasional full terminal crashes) they seem to have no trouble marching on.
There's a portion of this that is agentic driven and there's a portion of this that's just github making their own bed.
1. Arguably anticompetitive pricing like MSFT is used to doing with the office suite.
Github is a giant complicated stateful mess with a lot of reads and writes. It also has a lot of features at this point. Hard to scale and hard to optimize.
That being said, you are correct. It is absolutely no surprise to me that Actions has the worst uptime.
Sure, AI will undoubtedly have increased their workload, but how much of the shown figures is real, and how much is the PR department trying to make it look like Copilot & friends is a massive success?
1. https://github.blog/news-insights/company-news/an-update-on-...
What you described above will piss off and alienate even more people. Eventually there will be a critical threshold crossed. Microslop will be the first victim of Skynet 11.0 (I lost track of its current version but you can see how much damage is caused by AI in general now - this was the beginning of skynet. Except that it sucks).
GitHub has become a dumping ground for broken code and it has more bots than ever. As much as I hate ID verification it might be a necessarily evil at this point because clearly their anti-bot measures aren't working.
I don't know how this is even remotely close to acceptable.
GH is not a social network
It's a lot easier to get bug reports and fixes when everyone is on the same auth system.
That's why there is also a call for federated forges
There is also the "gamification" aspect that GitHub have. Doesn't motivate me personally, but could have effect on some others.
Projects on GitHub gets a lot more visibility. To the point that many projects that do not use GitHub as their main forge are still often mirroring their repository there, and have to deal with double source of bug reports or pr.
I'm confused with auth has to do with it?
We've had OAuth 2.0 since 2012
Actually, forget about the site...
Buy ad space on Github's outage page?
It treats any service being down as the entire platform being down which is nonsense.
It's just lying with statistics.
People love to clown on the fediverse because of having to choose a server. Which is no different from email. I guess the difference is that their ISP used to give them email.
It might be hard to create such systems using ruby and microslop AI management though
It used to be that GitHub could rely on a finite number of people interacting with their platform in real human ways in real observable patterns. So I'm assuming that they scale for those patterns, and optimize for the UI and UX hotspots.
But now everyone's got a moltbot running 24/7, sometimes many, and it's completely overloading a lot of services. Especially services like GitHub which are very much agent-centric nowadays.
Why is GH the only service provider seeing such consistently bad availability then? Everyone has had to scale massively all the time, if GH is choosing moltbots capacity over basic availability for the rest of the humans, they have made the wrong choice.
I follow some of the accounts that run 24/7 agent sesssion. Their projects are not even that novel for the number of commits that appear on the profile. Many of the commits have the log of beads, claude session etc (no change to the actual code). Some of them are ports of some projects to another language. AI surely will increase the productivity, but the waste and noise that some people are willing to commit ....
Microsoft forces AI usage down everyone's throats.
AI bot usage takes down github.
I have to assume that there are some serious fights going on between the poor SRE teams wanting to throttle bots, and MS not wanting to do anything to dissuade AI usage.
I lost track which Monday morning PST in a row this is.
For my personal project, I've been considering moving everything over to Codeberg. Stability of GH being one reason, but I also like the idea of an alternative that is not strictly tied to a big tech company.
About an hour ago, clicking "Resolve Conversation" in a Pull Request failed a few times with an error message that appeared lower on the page (outside the viewport), and which I did not see the first few times. I had to reload the page after every few actions to get the server to register new ones.
I told a colleague, and added "Github might be having an issue with some other service, and it's just bleeding over to PR comments? Maybe it will snowball into a larger outage?"
I’ve made 4000 commits in the last 2.5 months. That’s just to main. And I push up tons of artifacts daily for regression testing.
For $0.
For about 5 minutes, Google had a pay-as-you-go service on GCP for git. I used to use that, because I wanted to own my stuff. But (I guess) because everyone used "free" github -- and like a lot of other Google services -- they sunsetted it.
So now I'm using github for free. But would rather pay for my storage an usage with a (big) cloud provider.
Though i plan to use my own server later.
How am I expected to comment "LGTM" if I can't even get to the PR?
And I still only have local git repository.
I do not like why git doesn't have a built in bug or issue tracker and a kanban board or something.
I wished Git and Fossil hybrid should have won.
I thought about using fossil many times but it seems codex and claude have deeper integration with git.
I don't like installing software which keeps growing into infinite feature Monster. Maybe I'll install gitea or forgjo idk.
That's the last piece of puzzle remaining for me I've already mastered deployment and HA on bare metal from OVH and Hertzner, already have scaled to tons of users
Radicle supports issues directly as part of the git object database.
What's worse, GitHub is widely used as a CI/CD solution, it runs massive amount of build pipelines and test suites. There is a ton of players in this space, too.
GitHub's main value proposition was having all these things in one place, as a convenient web app, for free or for moderate money. So they're crushed by the success of their model.
Don't let "agentic" "coding" be the reason to avoid fossil.
Fossil and other VCS are much easier for humans to use than Git is; there's no reason to have an LLM burning up tokens and the environment to do tasks you'd do yourself quickly and correctly.
If there is so much extra code, where is it going? Is everyone just creating giant piles of throwaway slop?
yes, like the crypto or productivity 'industry', where every project exists to manage your other crypto and productivity projects, the dynamic of 'AI programming' seems to be to make software to manage your other two dozen AI things
mysteriously enough, big open source projects that people actually use in the real world aren't seeing their issues closed at any higher rates, almost as if that requires actual maintenance and programming (which Fred Brooks told us this half a century ago)
I've done a lot of "plain compute" work [0] with the Big Three Cloud Compute vendors. Azure is by far the worst. Mysterious resource creation failures, mysterious resource deletion failures, mysterious "incompatible schema" failures when talking to Azure provisioning and status infrastructure, mysterious and inexplicable performance problems, etc, etc, etc. Unless I was being paid a lot of money to use Azure, I'd take Google's legendarily nonexistent support over Azure's unreliability any day.
[0] That is, "create a VM with persistent disks, Internet access, and maybe a load balancer in front and ignore all of the other features provided by the vendor"
pre-edit: They just removed the incident and set the status page to all green as of 10:01 PT???
This is not sustainable and hurts actual customers.
Although I think I still ironically end up finding out about the outages on HN first.
A useful website to show why GitHub is feeling the burn so much with higher usage!
https://www.dayswithoutgithubincident.com/ (from an above HN item)
Eventually it will be!
"Ultimately, if more companies treated the framework as an extension of the application, it would result in higher resilience and stability. Investment in Rails ensures your foundation will not crumble under the weight of your application. Treating it as an unimportant part of your application is a mistake and many, many leaders make this mistake." https://github.blog/engineering/architecture-optimization/bu...
The people who warned about building dependencies on third party services are enjoying some schadenfreude now.
Ironic that a source code control system that was designed to be distributed is now presenting problems due to the failure of centralized services.
Microslop, 'xcuse me, Microsoft is killing off GitHub. Not necessarily by design or on purpose, but by slop-neglect effect. AI is eating the corporate brain that Microsoft used to have (well, a long time ago).
Personally (but who am I) I don’t need to check whether my changes work on Windows 7 and SunOS in order to continue on a project most of the time.