Hacker News Reader

Comment by korrectional | Hacker News Reader

I don't really understand why this is happening at this scale, it's not like they just became broke and can't afford a proper server... can someone explain?

View on HN

Agents are shipping code faster all over the world and in some cases 24 hours a day. Additionally, some significant number of non-developers are now developers i.e. they are also shipping to github regularly.

This is not limited to just pushing code but all the bells and whistles that github added as features under the assumption of some predictable growth are now exceeding the original plans.

I suspect a lot of their existing systems have to be re-architected for unanticipated scale, and it won't happen overnight for sure.

They were sucking 5 years ago before agents existed. I don’t think this has anything to do with recent changes.

https://damrnelson.github.io/github-historical-uptime/

Whoa, if that is even remotely accurate then the talk about agents is a complete red herring.

If I remember correctly the status page was not precise before the acquisition - so take with a big grain of salt the 100% pre-acquisition values

I remember the status page being quite accurate before the acquisition.

I don’t like this whole casting of doubt upon sources without providing superior or even alternate sources.

It makes it hard to discuss when one person presents a source and another says “I’m not sure that is accurate.” In a vague way.

What am I supposed to do with that? Research more sources that may or may not align with how you feel?

Pretty damning. Would also be interesting to see the number of commits overlayed. The graph tells a great story about the correlation with MS's takeover, but I wonder if at the same time that uptime went to shit, MS was shifting over large numbers of enterprise contracts to github. That would be a more complete story IMO.

None of which excuses this. Can you imagine someone's reaction in 2017 if you told them that github would be below 90% uptime in 2026? It would be unimaginable.

That’s nonsense. GitHub didn’t have 100% uptime before 2020. I remember downtime back then. And Microsoft didn’t make changes that fast. The only thing that changed is the accuracy of their status page.

Also go back and look at the unofficial status page from 3 years ago. It’s regularly above 99% and has been dropping steadily since then. Then in the last 3 months has dropped to below 85%.

This is coming from github’s status page. You need to reconcile memory of downtime with github’s official record.

I’ve been using github pretty much daily since 2010 and I never had a push fail or a repo be unavailable until recently.

I can tell you that graph is full of shit because you can go back through recorded incidents here

https://www.githubstatus.com/history

And immediately find that there are numerous incidents that would show up on the modern status board as an issue but are reported with 100.00000% uptime on that graph.

One example:

https://www.githubstatus.com/incidents/bzj1hc2cnfkc

2018-07-16 17:32:53 - We are investigating reports of elevated error rates.

2018-07-16 17:34:27 - We are investigating reports of service unavailability.

2018-07-16 18:04:38 - We've discovered the issue causing connectivity failures and are remediating.

2018-07-16 18:26:48 - We're monitoring the site as systems recover. Some delays are expected as we process backlogged data.

2018-07-16 18:37:26 - We're continuing to monitor and work on further remediation efforts as the site recovers.

2018-07-16 18:54:21 - The site is stable. We are continuing to monitor and work through follow-up remediation efforts.

And there are other incidents with connection failures or elevated error rates during July 2018, but the linked graph shows "average uptime of all components 100.00000%" during July 2018.

Another from October (that also shows 100.0000% uptime)

2018-10-21 23:09:19 - We are investigating reports of elevated error rates.

2018-10-21 23:13:31 - We are investigating reports of service unavailability.

2018-10-21 23:43:55 - We're investigating problems accessing GitHub.com.

2018-10-22 00:05:37 - We're failing over a data storage system in order to restore access to GitHub.com.

2018-10-22 00:23:54 - We're continuing to work on migrating a data storage system in order to restore access to GitHub.com.

2018-10-22 00:43:12 - We continue to work on migrating a data storage system in order to restore access to GitHub.com.

2018-10-22 01:02:49 - We continue to migrate a data storage system in order to restore full access to GitHub.com.

2018-10-22 01:22:22 - We continue to work to migrate a data storage system in order to restore access to GitHub.com.

2018-10-22 01:41:19 - We are continuing to work to migrate a data storage system in order to restore access to GitHub.com.

>I’ve been using github pretty much daily since 2010 and I never had a push fail or a repo be unavailable until recently.

Looking back at their downtime history. Unless recently is within the last 3 years, it seems like you got really lucky.

See previous days articles. Agentic coding. Going from 1b annual commits to estimated 14b or more from one year to another.

They’re on track to 30x volume yoy by their own words

Most of the outages are actually the unavailability of single AI models, not the core service.

The faster you move, the more you screw up, almost no company producing software have figured out how to move fast and not screw up. It's so hard, that companies even used to boast about how much they didn't care about screwing up, as long as they moved fast.

Add in new "productivity" tools that help you move even faster, with even less regards for how much you screw up (even though the tool could be used for you to move at the same speed, but with less screw ups), and an engineering culture which boils down to "Why not?", and you get platforms run by Microsoft that are unable to achieve two nines of reliability.

I suspect it’s caused because Microsoft is using buggy Microsoft tech instead of the original stack.

They’re making political decisions based on what they sell vs what’s actually useful for their use case.

It’s kind of impossible to find out if this is true though.

That doesn’t track because GitHub Enterprise Cloud has great uptime. This is all load based, vibe coded ai slop code shipped at record numbers from users who will never convert to paid. The real question is what are they doing about that?

How is github enterprise cloud uptime tracked?

Reply on news.ycombinator.com