GitHub was down again
GitHub is down once again!
With git's distributed nature, this would be very much possible.
Also, pushing to a mirror is not exactly the thing you want.
What makes apt easier is that pretty much everybody is just downloading from the source or from a mirror. When the source stops, mirrors don't get updated and everything still works. That's very different from the usage model that people have with Github.
Gihub/Gitlab have extended the Git usage model very much, they are not just git. You can't easily migrate away from them to another git offering (and not even very easily between both).
wow, when I read this, it is actually much cleaner than what I would use as in the past: the overall system strength is min {i in 0...m} subsystem[i] where m is the total number of subsystems
A sum of constituents is more likely to be a situation where you're fine if at least one of the options works (e.g. a cluster with redundancy).
This will actually get you a semiring (easy to check), although whether that is really useful remains to be seen. It's nice to have anyway.
If you had separate services the CI service database going down would not affect anything else. A centralised database for all the services will take down all the services when it borks
If your priority is profit or cost management then duplicated infrastructure will make no sense
Decentralizing on the other hand makes it possible to replace individual links.
If GitHub goes down you can switch to Gitlab. But you can't do that if you're using GitHub for git and issues and CI and static sites, etc.
I think the outages could be related to:
1. Github mobile just went public so they may of changed the scaling params to keep up with expected increase in traffic 2. The new notification system seems to be a lot heavier, and they could still be catching up to the changes 3. They were somewhat recently acquired by Microsoft so maybe they're migrating to Azure to reduce expenses
Whatever the cause, 11 days with outages out of 90 is pretty rough when you rely on Github for project management, a central hub for viewing CI, and all VC concerns. Feel bad for the smaller companies that wholly adopted GitOps and are blocked deploying hotfixes during these outages.
They recently posted a "post-incident analysis" [0] about several "service disruptions" in February that were all a result of database issues.
Obviously, I have no idea if today's outage is related.
---
[0]: https://github.blog/2020-03-26-february-service-disruptions-...
The status page looks like a Pez dispenser.
Usually companies put in place a release freeze or "Code Purple" when there are such demonstrated problems with releasing stable code.
Distributed systems for the win!
the fact that git is distributed is but a small part of the whole.
You can also do a "request pull" over email, which if it sounds confusing it's because GitHub wanted it to:
https://news.ycombinator.com/item?id=22955595
... which was likely because users regarded it as a quasidupe of
https://news.ycombinator.com/item?id=22935941.
By 'quasidupe' I mean it's not strictly the same story, but the difference ('down again') isn't significant enough for the thread to be substantially different.
(The submitted title on this one was "GitHub: Here We Go Again".)
Is it possible to run an experiment. Remove the flag option but highlight the hide option to users. Users who are not interested in the topic can remove it from view but do not kill the discussion for those who are interested in it. Then see if the quality of posts/discussions decline.
For those too young to remember, Microsoft bought Hotmail (forget the silly CamelCasing, we know that HoTMaiL was HTML + mail by now) which was based on FreeBSD+Apache and was champing at the bit to use it to demonstrate the scalability and stability of their then relatively new NT operating system and the IIS web server. Let's just say that... things did not go the way Microsoft would have wanted and the demonstration more or less achieved the opposite of what they intended. It took them a long time to move the frontend to Windows and an even longer time to do the same to the backend.
They won't make that mistake again but they might succumb to featuritis or wrongfooted attempts to steer Github-users further and further into the Microsoft world.
[0] https://help.github.com/en/github/site-policy/github-enterpr...
On a side note, the status.github.com page is quite delayed. Following live updates from Twitter has been a better strategy for confirming that GitHub is having issues: https://twitter.com/search?q=github&src=typed_query&f=live
Yes.
What other services are good? I loved using Phabricator at Facebook, but its cloud option is pricy.
Best I can imagine now is using GitHub as a primary and other hosted services as a mirror. Mirroring the git repos will only get you so far though, you won't have the services built around them (e.g. issues, merge requests, comments, actions) or you may not be mirroring all git repos that you need access to. But a reduced service may be sufficient for short periods during an outage.
I've also been experimenting with sircmpwn's git.sr.ht, if you are a minimalist to me like that as well.
It's a different workflow, but we've had zero unplanned outages in 2020 and are the highest performance software forge in general: https://forgeperf.org
I wrote about mirroring in another comment [2]:
> I put together a script [0] to automate the process to set the primary remote to Sourcehut (git.sr.ht) and mirror to GitLab and GitHub. And yes, that script is in a repo that's also mirrored to GitHub and GitLab (check the README). It combines well with my `git pushall` alias [1].
> [0]: https://git.sr.ht/~seirdy/dotfiles/tree/master/Executables/s...
> [1]: https://git.sr.ht/~seirdy/dotfiles/tree/master/.config/git/c...
Bonus points for having one remote being a tiny self-hosted instance, like plain git+ssh. I don't think I've had any reliability issues with plain git+ssh on an rbpi.
Furthermore, using mailing lists or something like git-bug instead of a vendor-locked-in issue-tracker keeps issues decentralized and eliminates downtime.
Yikes.
Is this possible with a self-hosted gitlab?
Trying to get that to push to a cloud repo (GCP/Azure/whatever) as backup. Managed to selfhost gitlab but at the edge of my technical git knowledge here
Years and years of stability to the point where it starts to taken for granted. About a year post Microsoft purchase, and here we are.
thank god they added back a button to group by repository. it was literally unusable otherwise - just a stream of disorganized crap.
the repo name is still uselessly duplicated in every issue line despite the common heading when grouped. you can no longer "mark all as read" in a repo without going elsewhere to see the hidden notifs, pressing "select all" and clicking "done"...so next time you refresh the page without doing that, it all floats to the surface again. etc etc.
ugh.