GitHub incident: 2022/03/24
githubstatus.com
githubstatus.com
I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide whether to tune the doublewrite buffer or redo log, or both, or neither.
If they do have the required staff it still might not be readily available due to org chart boundaries.
Unclear if GitHub has the staff or if they are able to draw from the larger Microsoft pool.
If they keep having issues I expect Microsoft to push them to move everything to MSSQL
Medium-term, the more closely aligned with the rest of Microsoft’s technology they can become, the better - not many MSFT folks understand the ins and outs of sharded Percona, but many of them do understand SQL Server and .NET.
https://www.businessinsider.com/githubs-ceo-ditches-meritocr...
Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outages continue.
Now it’s scheduled to fall over in about 6 months and everyone is freaking out again. It’s not new features that are hurting us, it’s the existing core product line. All the new stuff is built on horizontally sharded DBs from the get-go.
Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it
Not difficult, not hard to maintain. Worked fine for our ~500 projects (each with pipelines/ci|cd) and ~100 devs.
I believe people when they tell me they've managed it without major problems, but clearly they're the 10x developers of the self-hosted world, because the people I work with seem to find it hard - and that is not a slight on them, because clearly even Github can't do so much better!
Still waiting on that callback/reply. Starting to wonder if Microsoft even wants our money anymore.
I still have no idea how much the final quote for GitHub could have been.
If I dont hear back from GitHub enterprise sales before the end of the week, I am going to take this as a strong hint that we are too small for microsoft to care about, suggesting that we would be safer & happier on a different vendor's product.
* We get a non-machine message that migrations to GitHub are temporarily disabled.
* We get an exact message that the delivery of webhooks and action kickoff is delayed.
So while there are no 500s - I just have to wait a few more minutes for builds to kick off. So at least I can still review code in the meantime.
Ever since Travis CI sold its soul and I've been forced to use Github Actions, and I've come to hate every single moment I have to work with some broken or buggy action, with the inability to restart only failed jobs (they added this vital feature only a few weeks ago, 2+ years after launching actions), with slow job spin-up times, which are only slightly reduced when using the self-hosted, RAM-hogging action runners which are only capable of running one job per instance, forcing me to waste RAM setting up multiple instances of the runners, each bundling its own copy of node and the .NET runtime.
I've been using gitlab CI @ work, and I absolutely love its reliability and ease of use and configuration.
gitlab feels easier to use, easier to write and i prefer the interface.
EDIT: I see there are people down-thread that are discussing the SAAS version, ours was a on-site deployment so we could pretty much just spin up more iron if the executors were straining.
From [0]:
>> Until the next time GitHub goes down again (hopefully that won't be in another month's time).
>> That says it all really. Lets reset the counter and try this again.
Going to reset the counter once again. Hopefully GitHub won't have another outage in a week's (or even a months) time.
It turns out that my whole comment chain in [0] was right in the 'long term' of not 'centralizing everything' [1] on GitHub since 2020.
at the time of posting, webhooks and actions are still struggling :(
there have now been 23 incident outages in 90 days. if anyone from Microsoft is reading this, you need to start considering this degradation as the companies highest priority. At this point, Gitea, Gitlab and others (self hosted or otherwise) are beginning to gain traction as a more reliable and performant alternative.
Youre facing the very real possibility of spending nearly eight billion dollars to capture the developer mindshare of the internet, only to lose it in just four years.
(It was pretty nice that Skype let you do that really easily though)
Well there you go. The lesson is to not go 'all in' or 'centralize everything' on GitHub, which is what everyone was doing and why it has now become increasingly unreliable as I predicted, years ago in the long run. [0]
Perhaps it is time to self-host then?
Ideally it'd be a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency, but I think that's just unrealistic.
Having used git prior to Github et al, where I had remotes set up to literally each collaborator that they individually hosted, I would never want that user experience over what I have now. The centralized model is far too compelling. So I'm curious what the bleeding edge distributed model looks like today.
For example, in a work environment, what’s even the point of ‘forking’ in GitHub. I don’t want my own version of any repo, I don’t want to be playing weird rebase/merge upstream/push origin nonsense. I don’t even want to be creating branches or naming branches. I just want a centralised repo: I make a commit and… that’s it, get a commit hash and be able to have that commit anywhere else I check out the repo, and open a PR/diff to merge into master.
Epic probably won't merge it though unless you represent a big studio.
IMO feature wise, GitHub really has improved tenfold. I remember having to fight third party CIs and project trackers, now we're running them all on GitHub and it's a well-integrated pleasure. If it does work, which it usually does and hopefully goes back to doing.
I also really enjoy their live vscode IDE, which lets you e.g. make small changes to PRs in a web vscode by pressing . on the PR page. This is different from their hosted VM offering and free, confusingly.