I’ve noticed most posts and replies are just people coming to their own conclusions based on whatever published data.
I think it would be helpful to get some actual, non-corporate/marketing information on the goings-on by those that actually see what’s happening on the inside.
Thanks
Platform activity is surging. There were 1 billion commits in 2025. As of three months ago, it was 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
20% of all GitHub accounts were created in the past 6 months https://x.com/kdaigle/status/2082604368399159542
Might be worth considering a /GB/MO fee. Let's face it, VCS is a (glorified) storage service.
I just checked no 1 vibe coder https://github.com/steipete and he has like 279,440. Compared this to Taylor who works on huge project like Laravel and is only around 8k commits.
Around the time when Windows 95 was released, Microsoft did predict that the internet will become popular also among private users. That is why
- they included Winsock (Windows' TCP/UDP/IP stack) as native part of Windows 95 (in opposite to Windows 3.x where you had to install a third-party implementation of the Winsock API such as Trumpet Winsock), though it was typically not installed by default,
- support to use MSN (the dial-up online service and internet servicen provider [1]) was pre-installed with Windows 95.
What was not clear at that time was that the world wide web would become the killer application for the internet for consumer users (at that time, the more likely killer application of the internet for consumers was seen in proprietary content providers such as AOL, Compuserve or MSN), so Microsoft provided no web browser with the original version of Windows 95.
But even at that time, it was conceivable, though not clear that the world wide web could become important. So, already at the release time of Windows 95, Microsoft provided Internet Explorer 1.0 as part of the Plus! add-on pack for Windows 95. With Windows 95 OEM Service Release 1, the Internet Explorer 2.0 got part of Windows 95, and Windows 98 introduced the (optional) Active Desktop [2], in was centered around the Internet Explorer.
So, in my opinion Microsoft did everything "correct" here: they provided Internet Explorer from beginning on as an optional separate component (Plus! add-on pack), and once it was obvious that the world wide web would become very important, they reacted fast and included the Internet Explorer in the next service release of Windows 95 (Windows 95 OEM Service Release 1).
--
[1] https://en.wikipedia.org/wiki/MSN_Dial-Up_Internet_Access
Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it.
Instead you have to realize that you are entering into a high growth phase and design around that.
It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly.
At some point you have to admit that Github ready for the moment.
Edit: Found the thread: https://news.ycombinator.com/item?id=47616242
Don't they sign NDAs? Do they not care? Are they not enforceable?
Very curious to know.
Some fuck-ups have such long-lasting effects that they outlast the NDAs that would cover them
For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.
GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.
And they have been working on this semi-shared state design for over 10 years now.
Like, I get there are other features they have, and things like history, but I don't think it's quite the same.
Isn't most (not all) of the LLM's resources stateless compute to a large degree?
(Not defending MS at all though, I currently can't create a PR on the required branch for my work (won't load the branch list) and their availability is really bad recently).
Google “RLHF”.
Also Github's feature offering has been incredibly stable for the last 6 years. There isn't an excuse for it to be unstable like this.
That's due to the need for ultra-fast memory and inference compute, it's not the same thing.
You think GitHub ought to match that?
So has Github's owner Microsoft. It's build out is on par with OpenAI and Anthropic.
> You think GitHub ought to match that?
Microsoft is.
The issue is that the software stack is not stable. That is clearly the issue here.
It isn't a hardware build out issue. It is related to the architecture of Github.
By spending tens of billions in investor cash. It is not clear what a sustainable business model looks like for these companies at the moment.
> They are handling it a lot better than Github.
Are they? Anthropic has fairly frequent issues.
There's high growth and then there's demand increasing 10x in a year. For a mature business like github that's a massive, and I can't think of many big businesses who would be able to scale smoothly to accommodate that kind of demand.
No, they are not.
Charge $1/month per repo. If your project isn't worth that, run your own git or use another free forge.
Free Users use GitHub => when a company needs source control => all their coders use GitHub => company buys expensive enterprise contract.
If there’s less of an advantage to using free GitHub then that funnel works less effectively.
Seems like github is too popular for itself.
Could do similar for GH actions usage.
Encouraging big commits seems antithetical to the idea of git in general.
OK so then use the additional revenue to add more resources and employees? What the f*k are they doing over there?
If they're really that cash-strapped at GitHub, then are they planning to just continue allowing this to happen and drown, or fix the issues? This has been ongoing for over a year.
What revenue? I'd assume the people pushing thousands of AI-generated commits are not the people paying for GitHub Enterprise.
I'd assume at least a portion of them are paying for GitHub Pro, and by now at least a small % would have converted to Enterprise.
If not, then they need to fix their operational model - I don't understand people acting like this is some insurmountable challenge given Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion in 2026.
Clearly, something needs to change. But if they're not careful they might provoke a reaction from their existing userbase that's even more negative that the response to these outages.
Of course it can - everything is tough at that scale, but their parent company has billions in profit. They've had a year to come up with a plan/fix and have failed miserably.
There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc.
When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience.
There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills.
Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today.
But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.
Ironically the core application is written in rails, which may be part of the issue
Fixing small errors and keeping going on maybe is a good business strategy.
The last result is probably what you said.
The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is.
My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.
kinda reminds me of it. not my most valuable contribution to forensics, admittedly.
If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.
Something more serious is at play. And I want to hear from the actual underlings.
100% agree.
> Remember these folks hire people with plenty of system design experience.
Not my overall experience 30 years in the industry.
> They likely also have the money for hardware resources.
You'll always be asked to do more with less, at least, that's my own experience.
So even if these cloud/SAAS solutions perform poorly it is not like customers have wherewithal to create alternate solutions.
it's not impossible. they don't, right up until they do
Amazing you can say this with a straight face in these DC building times.
Have you tried getting Azure east instances anytime lately?
Hey man, in case you hadn't heard, the hardware industry is having a lot of trouble keeping up. "Expand our capacity" isn't as easy as it was 2 years ago.
Additional load reveals limits and breaklines that were never known before - your temporary integer IDs are suddenly too small, you reach the maximum file descriptors in a Linux distro, your hash function suddenly has repeating conflicts, your DB index is suddenly oversaturated.
Having massive exponential growth of your traffic would probably require constant refactoring to meet the ever changing system limits.
It's hard to design something (large and complex) for much more than 10X scale with only HW capacity increases.
Now however, it's competing for resources directly with the corporations core strategic plans.
GitHub having issues is embarrassing, but the shareholders don't really care.
Something about these two factors seems to have conspired to wreak absolute havoc on their uptime in the ~7yr hence. The AI slopocalypse hasn't helped matters, but the foundations were laid far earlier.
Would love to hear from an insider, either for or against this conspiracy theory.
(talking about leadership, not my lovely msft engineers reading this)
That is how Microsoft markets Azure. :-)
Hardware probably isn't going to do a lot with demand rising 10x or more at peak times, if they had scaled and tested systems with a view to supporting an order of magnitude more requests than expected then all that contingency resource is already gone. Adding 10% more hardware is going to be very expensive and will likely not have that much effect anyway without design changes to make better use of that hardware, scaling by orders of magnitude often reveals bottlenecks that were not apparent at the previous scales, or were apparent but dismissed (I imagine somewhere in a comment or design doc someone having written “we will need to rethink this if the system is to be asked to process 10ˣ⁺¹ things/period rather than 10ˣ”).
And once you've hired those people, even if they get up to speed pretty much instantly, a system change isn't going to be designed/made/rolled-out in short order so patchwork fixes are still needed in the meantime, and any roll-out probably needs to be gradual which adds further complication.
Scaling isn't easy. Scaling fast is a royal PITA.
The AI boom caught them napping though, and they're slipping well into the too unreliable category
Microsoft, who invested eyewatering sums into AI, even near to betting the whole company on an AI boom, is surprised about the AI boom?
I bet it's mostly due to Azure being brittle and the AI boom is just a nice cover story (not saying it's not challenging to handle all that new load) to save face and sell more Azure services.
Microsoft management knows this. It's not like there haven't been attempts to solve these core problems. The issue is GitHub was built by exceptional engineers, and none of those people are still around.
I maintain all these problems began when they migrated off Erlang and egitd. :)
The cloud does scale that easily, the software doesn't. :-)
What speaks against the volume activity is that GitHub works perfectly after an outage. People don't just stop AI coding after an outage. If anything, the volume should be higher for people to catch up.
It is more like rebooting a vibe coded WIN-95 because of a vibe coded integer overflow (joke).
Had these DATA and message been tossed around more people wouldn't be so frustrated and would be more understanding.
Additionally, there is a strategic reason to be the repository for all code, free, or not which is likely at conflict with reliability goals.
(e.g. they could get rid of their free tier which would improve reliability at the cost of allowing a new competitor to be the repository for all new AI driven code)
The long term strategic benefit of being the home to all code (or most of it) is enormous. They are likely balancing that strategic goal with what is seen as short-term internally as reliability.
Further, it is clear that Microsoft and GitHub have been strongly encouraging, if not requiring, AI tool use in their own, internal development. I have seen several automated PRs within GitHub repos that were 100% done by AI. Big surprise where that's leading.
Eventually, though, they decided to migrate the whole thing to Azure. And they were far enough through that to be basically committed... when AI coding started hitting them with much higher workloads.
I personally think the reliability problems are more to do with the reliability of the Azure migration. But both factors are likely relevant.
Ruby is single-threaded, slow with poor GC and memory leaks. As Twitter's Alex Payne said publicly in 2008: "there shouldn't be doubt in anybody's mind that Ruby itself is slow" and that scaling Rails meant "just throw more instances at it"
It's just pointless hacks on hacks. GitHub didn't need React on the frontend and any potential resource savings of client side rendering were lost when they realized they need to do SSR on that stuff too.
I've encountered so much frontend jank as they expanded that portion of the stack whereas it was always excellent when it was just Ruby SSR and minimal JS on the frontend.
The downtime also starts happening in 2020, which is a while after the acquisition. 2020 was the year many people started learning to code while being stuck at home.
After Microsoft acquired them, they started rolling out new features at an incredible pace. And if you look at the breakdown of the downtime, most of the downtime is in features that didn't exist before the acquisition.
Though not all of the downtime is in new features. The core is also less stable than ever.
My (not-so-actuarial 1990's-experience-colored) feeling is that this is Microsoft doing what Microsoft does best, namely including a lot of seemingly compelling features in a product to get people hooked and to keep competitors in a position of trying to keep up.
--------
[1] Possibly none, are you paying for a pro or enterprise account?
[2] There will be a little indirect income perhaps, via deriving value from tracking users and being able to use the combined content for various analytics (including lately AI training).
[3] Potentially reducing the cost/complexity of integrating disparate systems⁴, themselves.
[4] Sometimes in-house developed systems that might distract key devs from the company's products/services that they can sell.
Having been through a couple of acquisitions (as a cog in the machine being acquired, not a founder or purchaser or such), the last one at the end of 2025, and seen others as an unofficial close observer via friends/family, I think this is quite normal. No matter how thorough the due diligence and other investigations are pre-purchase, there will usually be an information gathering time before any significant changes are made. Even without that there are a couple of months being very careful not to rock the boat too much, so as to not put the willies up shareholders and key human resources. The first significant changes often won't happen for 3-to-6 months, unless part of the reason for the acquisition is that the company was on the verge of failing or was growing too fast to cope. During that time, unless key people jump ship early forcing change to an extent, things below certain management levels are much more business-as-usual than you might expect.
(Github did probably want a native CI implementation but I'm not really sure Actions was the way to do it. It seems so needlessly complex to me, not to mention the massive security issues. The rest of those features are just far from something I'd ever use so I don't even really know how to comment on them)
https://x.com/mariorod1/status/2085800861469495465
I really think something deeper is going wrong there, and they're not being honest with their paying customers (and enterprises) about it.
This behavior is what led them to the current situation and will only get worse.
A technical foundation is not infinitely strong. Put a big enough castle on a swamp, and it will sink into the swamp.
HN Search: azure capacity - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Azure Capacity Crunch Extends into 2026 Amid Data Center Constraints - https://windowsforum.com/threads/azure-capacity-crunch-exten... - October 9th, 2025
lol ... That is absolutely hilarious ... I'm not sure if it is /s, but if this is true, enterprise stupidity has reached a new level.
- GitHub attempting (and seemingly failing) to move to Azure infrastucture for its website backend
- AI generated code wrecking the site due to the volume of activities.
Don't act shocked.
Also, gritty insider information is never the same as the official word.
Are you also shocked that you can stand in the middle of the highway and get hit by a car and die? You can do it if someone asks you but you’re not forced to.
See: https://news.ycombinator.com/item?id=49353915
Disclaimer: not a lawyer
A whole lot of AI agents are presumably using Github as part of a workflow.
Would be story worthy if one of them mis interpreted their instructions and is causing havoc.
Like "Really make sure changes are saved to git" -> agent "I have to hack into github backend to really make sure the changes are saved to disk"
That was about 2 years ago. Im not kidding.
I dont know what the moral of the story is, but I found it weird at the time ( for added context - i was using AI back then about 10 hours a day, BUT I think sentiment on HN was "still" around the vibe of "you use AI to code without checking every line? I doubt your projects work" - but I had nothing better to do then wrestle with it and was surprised how I hadnt checked my code in weeks but stuff "worked". Its much better now and agents are accepted ofcourse but it sort of "snuck up" on people even in the tech community as recently as that.
I guess what Im saying is time is going quickly.
Couldn't they charge just a teeny tiny amount per PR or diff size or something and this would basically go away? I don't actually know what is the heaviest load. Probably actions? I understand it would be nice if it was free, but they have to pay for their infrastructure.
Or maybe pay extra for priority. Does everyone pushing llm generated code they'll rewrite or abandon in a week need all of the GitHub processes executed instantly? I'm probably way out of my depth here on what the actual bottlenecks are.
Could they break out reliability promises and reserve capacity for just the basics?
No employee with professional ethics would tell you that (the real internal situation).
You can observe and analyze the reasons from some things.
Use it and tolerate the bad things, or on some time point go to other place (use other tool).
Aug 17, 21:15 UTC Resolved - On August 17, 2026, from 13:28–21:15 UTC (7h 47m),
GitHub.com experienced elevated errors and latency across Issues, Pull Requests, APIs, Actions, and Copilot. At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. SAML/OIDC authentication, SCIM, and Team Sync were also affected, as well as Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on GitHub.com. Most services recovered by 16:36 UTC as our Central US datacenter recovered; Actions was degraded until approximately 18:03 UTC; and Copilot Token Service fully recovered by 21:02.
Some of the failing traffic was moved from Central US to Northern Virginia where it was served successfully until the network failure in Central US was debugged and resolved. Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery. The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR and 2) blocking inbound Copilot Token Service token requests at the load balancers with a 403, and then gradually ramping back up traffic per-site to allow callers to succeed. Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS. Reducing gateway authentication retries and blocking retry-triggering responses stabilized Copilot Token Service and completed recovery.
Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints.
To prevent recurrence, our follow-up actions include:
- Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
- Auditing Istio request, concurrency, and scaling limits across affected services.
- Reviewing retry limits and backoff behavior across gateways and clients.
- Addressing the VS Code retry behavior that amplified Copilot token traffic.
So basically bad code pushes that caused request amplification and then huge gaps in operational scaling and reliability standards. Oof.
Again, I am patiently waiting for the AI bubble to burst.
Does the AI ever look back at the shit trail it left behind?
Diffs are no longer diffs, they look like largescale delete and rewrite
Pick one!
How many non Ruby apps and all the databases and systems behind could climb their traffic 14x when they probably only perf-tested for a sustained 2-3x and, and paid for a profitable amount of rack space rather than a “lol, VC money, we’ll 10x later” amount of infra.
You load test and see how much it takes until the system falls over - basically you DDoS yourself and see what the numbers were when things broke, and what broke first second so on. Maybe in a sandbox maybe against prod.
It's very telling that issues began almost to the day of the Microsoft acquisition.
If it sounds like I'm defending GitHub a bit, yeah, I am. I can't think of a comparable situation that any other web site has been through. This isn't "oops, we didn't plan for the Black Friday sale", this is a once-in-a-generation event focused on one, important web property that no one would have handled perfectly. (Even us all-knowing commenters here on HN.)
You know that old trope in every submarine movie, where the captain tells the driver to dive lower than they've ever gone, and someone says "I don't know if the boat can take it!" and then they switch the camera to some engineering room where the boat is groaning under the pressure and a bolt comes loose and water starts spraying everywhere but someone runs up with a giant wrench and tightens it and looks around at everyone else in the room like, "whoa, that was close..."?
Or Scotty and Star Trek, "It canna take much more, Captain!"
That's what GitHub has been going through. It's had problems, but it's still here, and usually working well.
- Let me start with: GitHub is ~3,000 talented and really nice people trying to do the right thing, with a great culture. That's the most important thing I want to communicate. While I'm sure no one at GitHub is happy about their uptime, believe me when I tell you that GitHub Engineering is exceptional. They have great leadership, depth at all levels (Distinguished / Principal / Staff / Senior / earlier-in-career) and if you ever get a chance to hire someone from GitHub, you should. I hope all of you get to work with engineers as good as GitHub's.
- GitHub has been improving all aspects of its infrastructure steadily for many years now. The GitHub Engineering blog https://github.blog/engineering/ has been documenting this the entire time. Go point your favorite LLM at it and ask for a summary of all of the major system improvements since 2020. If that work hadn't already been done, GitHub would be a smoking pile of servers at this point. There's a lot I could list, I'm not sure which are already public, but improvements on the order of using thousands fewer CPU's to serve even more traffic than before have been made, and still are, I'm sure.
- GitHub was already serving billions of requests/day before agentic coding hit. Their challenge wasn't scaling a fresh new system with a few users an order of magnitude; it was taking one of the busiest and most important sites on the Internet, and getting hit with 14x traffic in a year, and having to plan for 100x. If you think your systems and infrastructure would have survived that, if you think you would have been able to politically navigate and succeed in getting projects green-lit at a large company to prepare for 100x scaling before it hit, to get those resources for "we might have scaling problems in a year or two" instead of getting them to ramp up on AI coding and other features that were crucial to growth right now, you don't understand large organizational dynamics. That's not a complaint about GitHub or Microsoft; it's an observation about capitalism and how any mature management group prioritizes things in software. I'd expect everyone in the San Francisco/Silicon Valley Reality Distortion Field to understand that.
- We talk in terms of "14x commits" to Git but that's only part of the story. GitHub Actions, webhooks, github.com itself, and other parts of GitHub, have all been under pressure. It's the totality of it, the seams that have been exposed at scale that couldn't have been exposed without that scale, that have caused the instability. That's why architecture gets overhauled.
- Yeah, GitHub has had less consistent uptime since the 2018 Microsoft acquisition. The GitHub that existed before that had much less functionality, an order of magnitude fewer users, and had received very little improvement in the few years before. Microsoft invested and enabled GitHub to grow into something much bigger than it ever could have without them.
- Be grateful that Microsoft - with a 50-year history of shipping developer tools, and more experience operating enterprise software than any other company on the planet - acquired GitHub instead of Google, which was the other major player in contention. Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there. Which amazing developer tools from Google do you use regularly? Yeah, I thought so. On behalf of Microsoft, you're welcome.
- Did you notice that GitHub swapped out one of its data centers last year for one 3x larger? No? Maybe that's because they executed it flawlessly, with no downtime. If you've ever done that on a massive web site with as much scrutiny as GitHub receives, you get a gold star.
- Did you notice that over 50% of GitHub read traffic is now being served from Azure instead of GitHub's own data centers, and growing, and that all GitHub Monolith traffic is scheduled to be served from Azure instead of GitHub's data centers by the end of CY26? This massive migration is taking place while traffic is going insane. From https://github.blog/news-insights/company-news/github-availa...: "GitHub can now serve a larger share of customer requests from independent Azure capacity, reducing reliance on any single datacenter while preserving performance. Monolith read traffic served from Azure Central US peaked at 52.75% on July 28—the first time we consistently remained above the halfway line. Git traffic in Azure reached 47%, up from 43% in June, and 29% of all repositories now have a second replica in Central US, making failover less disruptive when a region degrades."
- OpenAI and Anthropic obviously have significant scaling challenges as well, but there's a huge separation between the GPU-based inference part, and the CPU-based front-end systems. Their CPU-based systems are significantly simpler and newer than GitHub's, so I'm not surprised they have fewer outages, but it's still non-zero.
- Ruby on Rails is not the problem (and, for the record, I don't even like Ruby or Rails). Most of the performance-critical systems that were built on Ruby have been migrated to Go or some other language that takes full advantage of multithreading. The web endpoints still served by Ruby are fine.
- Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet - only AWS is larger in terms of hardware but not in terms of the number of products they ship - and Microsoft's first-party systems that run on it, all at the same time, are among the largest, busiest, most important systems on the planet. Entra ID, Azure SQL, Service Bus, Event Hubs, Office 365, Cosmos DB, OneDrive, etc. all in the tens-to-hundreds-of-billions requests/day. I'm not even counting the massive customer-owned systems that run on it, including almost the entire Fortune 500. Nothing is perfect, everyone has downtime, we all always want more and better features, I want improvements from Azure, too, but, please, grow up. I have opinions, too, I've been programming since the Apple ][+, there are popular technologies that I don't like, but I know that's subjective. "Your favorite technology sucks, mine is better" is not an objective statement about anything.
Anyway, if you haven't shipped at GitHub's scale, in this dynamic of a world of feature churn, and one-of, if not the largest, traffic spikes in the history of the web, have fun saying whatever you're going to say. I've never worked at Google or Amazon, I have opinions about their product design and culture, but there's nothing I can say about their infrastructure and systems because I have no idea about them, and what we all do in this crazy world of programming is harder than it looks.
For some, this won't be the answer to "what's going on?" because I'm not pointing fingers at any one thing. To me, the real answer is: an unprecedented scaling event on what was already one of the busiest sites on the web exposed seams in their systems, including the need to move from their own data centers to a much more scalable cloud provider - that are being addressed as quickly as possible.
I'm not telling you not to look for alternatives, I'm not telling you what to do about it, I'm not saying that all of their problems will magically be solved soon, I have no idea. I'm even launching a new version control system myself very soon to compete with them. But I do know that the thought "GitHub doesn't know what they're doing" is wrong and unhelpful.
The fact that Azure is big and important has very little to do with the quality of the implementation (and my experience with the services on top of it suggests that quality is not great).
> GitHub Actions
For me this is probably this biggest culprit along with the general pressure of AI against the hardware. Github actions became really popular. So many of those 14x commits are generating real CPU and network load.
This is a really non trivial problem. They can't just wall it off. They have real customers on that stuff.
GitHub Actions is 'simpler' to scale, it's just a bunch of runners, and Azure has a lot of capacity for VMs (mostly). Git itself is a compute intensive process and is far more interlinked.
Actions are unbounded, take minutes to resolve - sometimes hours and there are megabytes, gigabytes of network transactions.
Scaling commits is a load balancer and some more instances of a thing you’ve built over and over.
Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure. Oh, and maybe you want to make sure folks aren’t trying to hack that infrastructure as well.
It is a very, very hard problem.
> Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure.
That infra scaling is what the cloud provider can easily offer.
Meanwhile, GitHub repos/issues/PRs is all the business logic that they've written, evolved over time.
Many coupled distributed DBs, indexes, consistency concerns, etc.
That's trickier to scale.
Android
Youtube
Waymo
Chrome
Workspace
GCP
>Which amazing developer tools from Google do you use regularly?
Kubernetes
Jax
Tensorflow
gRPC
Go
Istio