~10 downtime per month. What is happening at GitHub?
It took a community drumbeat and persistence from an enterprise customer to get the status message to even show a problem last month. [1]
It does suck and I do think there must be some political infighting going on that the service is having so many disruptions.
There’s no excuse for something this important to not only have so much unplanned downtime, but no resources to connect with the community by offering post mortems or other reasonable interactions.
That said, I’m still all in on GA. It’s amazing and the coupling with repos is great. It continues to be subtly refined. So I just hope whoever is holding this product back gets out of the way.
[1] https://github.community/t/random-unknown-blob-error-when-pu...
If a large open-source organisation was to rely on say, GitHub Actions for example, well you'll probably see more and more of "GitHub down" posts and they'll be unable to push that critical patch or run that cloud CI on GitHub, and some maybe considering solutions like this [0].
Every time this happens, you'll be completely locked in and ending up contacting / complaining to the CEO of GitHub for support via Twitter.
No thanks and no deal I'm afraid.
You can either deal with the occasional non-productivity from a SAAS offering (which for GH has never lasted more than ~half a work day), or you can spin up all your own stack on-prem and generate 10 additional full-time problems in the quest to solve this one periodic issue.
The trick is to never put yourself in a position where your tools absolutely must work immediately or you lose a customer. Why make a promise on delivering a piece of software until its already in hand? Also, if github goes down and I really wanted to get an issue comment in, I can just open a text editor and keep a note around in my local repo until everything is back up. I can even do some crazy things, like share my local branches with other developers over side channels until things get back to normal in the centralized system.
Wasn't this kinda the entire point of talking developers into moving to the git model? Would be fun to rewind the clock and use these takes as an argument for sticking with TFS, et. al.
Haha, right, totally crazy, you could use git just like it was designed to be used! Who would do that!!
(Joking aside, despite Githubs frequent outages, I don't recall the git service itself ever being affected)
The most important part of their service has never failed to work for me over the last 6 years.
However, saying "just get GitLab and deploy it to your own server" glosses over the huge time sink it is, especially for small companies that are already short-staffed, to maintain something like that. I sure as heck do not want to be responsible for keeping my GitLab server up.
As you say, if you're writing an issue, put it in an editor issues.md file or something. If you're working on code, even better, just commit locally.
For us, we have ~8 people that need to use the system all at the same time. We utilize issues very heavily (we are entering 5 figures), with lots of data-heavy QA content throughout (screenshots/videos/binaries/etc). Additionally, our customer environments are actually configured to talk directly to our GitHub repository for purposes of rebuilding themselves from source at update time.
Because of the number of participants who are involved with our particular usage of GitHub, we find that a hosted solution with horizontal scalability and resilience to be an excellent fit. We have made the decision to make it Microsoft's problem to figure out how to eventually deal with 10k+ issues and 200+ employees/clients trying to hit the same host all at the same time.
If we had decided to host our own GitHub/Lab server in our cloud environment, we would be having to constantly review the capacity of the IT systems. As we add employees and customers, the load we put on our source control solution will increase linearly. Additionally, because of the deploy-time approach, having a solution that is backed by someone else's network means that we don't have to worry about our private network being slammed by outside requests. Our total checkout is nearing a gigabyte, so you can see how this might scale poorly if we operated out of our own infrastructure.
I almost feel like we are abusive of Microsoft's generosity considering the sheer amount of content we have throughout our organization's account. Every day I wonder when I am going to get some email demanding that we switch to a more expensive enterprise plan because of how we use the service. Maybe that day will never come. Even if it does, I will gladly shell out for the bigger contract.
I have to imagine that you're not exactly a small fish, but also not making them sweat too much either.
I wonder if they have a big feature underway or are just migrating more infrastructure to Azure?
EDIT: Either way, some postmortems would be appreciated before more customers have to look for a backup solution...
> There must be quite a story behind this - will you be putting up a post-mortem ? (Post mortems of business "outages" are usually more instructional)
> Yes. We will. Please stay tuned.
[1] https://news.ycombinator.com/item?id=21718171
[2] https://news.ycombinator.com/item?id=21718083
[3] https://web.archive.org/web/20191205225751/https://developer...
I think if revenue or product quality is tied to a VCS, having an active-active or active-passive setup is the way to go.
Fortunately, I'm on an on-prem product so that investment hasn't seemed worth it yet.
This doesn't mean we don't escrow our code, but rather than try to rebuild from source, I just take a short coffee break and wait for the impacted service to come back up :)
I'm consulting for a company that uses Azure DevOps and I cannot believe how much harder it is to use than Github Enterprise for getting things done. The documentation is also strictly worse, and localization is just not there at all.
-edit- I assume someone who works on Azure DevOps might be looking, so a few small specific things so you don't think I'm just a hater.
- It is hard to use markdown consistently, and it is particularly painful when doing any project management work on DevOps (which I assume is one of its strong points)
- The lack of a Japanese menu really sucks for something that is supposedly aimed at enterprises. Having to explain both native and English language vocabulary terms is double plus ungood (again, in an enterprise product put out by a company that has usually done excellent business in Japan)
- It's baffling that I can't make changes to the template when you give me a template option. I assume it is a permissions issue, but really?
Welcome to AzureHub.
They belong to Microsoft. Reliability was never a feature of Microsoft products.
Based off these reports most of their recent issues have been them hitting scale limits for their MySQL configurations and not having sufficient monitoring.
They were acquired by Microsoft a while ago, and now the chicken are coming home to roost.
It's pretty much on par with my Azure experience.
But that's really the only negative.
I'm trying to find a single term to describe all of Azure, and I'm having difficulty with it. Sophomoric? Like a place where the leaders are a bunch of B-class players, who lead all sorts of C-, D- and all the way down to Z-class characters.
And hopefully you'll never need support with Azure, especially urgent support, because it's atrocious.