GitHub Outage Currently Ongoing – Issues, PRs, Notifications | Hacker News Reader

GitHub Outage Currently Ongoing – Issues, PRs, Notifications

githubstatus.com

140 pointsdazbradbury101 comments

Open articleView on HN

Alright, cue the usual:

Monoculture bad

Microsoft bad

Git is distributed but all the things around it that we really need aren't

You can set up git to push to multiple remotes automatically

Nobody is actually using git in distributed mode

Did I forget anything?

The usual fluff about migrating away to Gitlab?

"We would migrate to Gitlab, but it's currently down"

Given the context I guess he is talking about the self-hosted version.

See also https://gogs.io/

Irony of ironies. Gitlab's import-from-GitHub functionality has been broken ever since we tried to migrate away. (They're going ot fix it soon.)

I did this for the first time 2 days ago on two medium-size GH repos and it while was crazy slow, it eventually completed after about 8 hours or so. The actual bits seemed to finish in a relatively quick amount of time, but the project landing page continually showed "import in progress" for all those 8 hours.

Git is simple to import, but all other related data may be subject to API rate limits.

I've opened an issue with some improvements suggestions, like surfacing when we are rate limited and allowing gradual access to the data while the import is going on: https://gitlab.com/gitlab-org/gitlab-ce/issues/66525

I'm sorry you had a bad experience with this.

The importer can still get better but we're importing about 500 projects per hour at peak https://www.dropbox.com/s/hr3ndcmu21aehmk/Screenshot%202019-...

Some imports work. The ones that failed are those with release notes (which were also the more important ones). And they fail silently (succeeded but no releases imported), which is annoying. I could dig up the issue numbers.

I'm generally very pleased with GitLab. I'm delighted that it's open source.

Glad to hear; I'd really like to see us improve some of our import capabilities at GitLab. We've created a separate team specific to importers and have started moving much more quickly here. Thanks for your patience.

Please let me know if this issue addresses the problem you encountered, would love to make sure we're prioritizing it. https://gitlab.com/gitlab-org/gitlab-ce/issues/20780

I think these issues were implicated:

https://gitlab.com/gitlab-org/gitlab-ce/issues/65194 https://gitlab.com/gitlab-org/gitlab-ce/issues/52956

No idea when these are going to be deployed. 12.2, 12.3, 12.5... But I'll keep checking in and trying the import every few weeks.

"Just run gitea/gogs in a raspberry pi in the closet, works for me"

I hide mine in a Barnes & Noble

That’s smart. No one will ever find it.

That should do it.

cue somebody asking the difference and a paragraph explaining the drama that unfolded between them

For many companies it's not just about running Git server. It's team productivity and all the tooling around it.

The conspiracy about how much infrastructure has been going down this summer, and how it's probably interns / a state actor / people pushing to get things in production before performance reviews / people being on holiday / that we've reached peak complexity.

Seriously though this summer has been unusually bad ceteris paribus right? We need a State of the SaaS address.

Give us some credit: it's the best 2019 we've had so far!

2019 BC doesn't even have a Wikipedia page!

You forgot my favorite: senior people leaving for greener/less stressful pastures.

Conspiracy of people being on holiday?

If GitHub actually had an open source version, then I could selfhost on infrastructure that I trust.

Wait till you hear about Phabricator! It was a Facebook tool that got open sourced and then the devs spun out their own company to provide paid support and whatnot. But you can just self-host it and frankly, it's glorious.

https://www.phacility.com/

I actually have, and even better it can easily be setup with Helm on a Kubernetes cluster:

https://github.com/helm/charts/tree/master/stable/phabricato...

at that rate wouldn't i just use gitlab

If you want a complete solution that mostly works, then yes. However, I think there is a lot of benefit in tools like Phabricator, or rolling your own SCM with Gitea and implementing Drone CI.

What about "rookie mistakes by code artisans who are making it up as they go along"?

I hope to see a trend towards "you can host a Git repository yourself, why do you depend on GitHub?".

> Git is distributed but all the things around it that we really need aren't

Can anyone recommend an issue tracker that is distributed with the repo?

@TODO comments strewn around in your repo.

I was thinking of something that wouldn't be affected by the git history. There's no point in having branches for issue comments.

Something like:

  $ git issue add "foo doesn't work"
  Created issue ece5591: foo doesn't work
  $ git issue comment ece5591 "the problem might be with bar"
  Created comment 0191fa1 on issue #10.
  $ git issue comment --reply 0191fa1 "or with baz"
  Created comment f98e783 on issue #10.
  $ git issue show ece5591
  foo doesn't work -- ece5591 Your Name <your@email>
    the problem might be with bar -- 0191fa1 Your Name <your@email>
      or with baz -- f98e783 Your Name <your@email>
  $ git issue push
  Pushed issues ece5591.
  $ git issue pull
  Pulled issues 3ebdc7e, 24cdb90.

EDIT: I just gave myself a clue:

https://github.com/dspinellis/git-issue

The nice idea to having issues in your history/branches is that it easily inverts the "flow" of branch state information: most issue trackers do a lot of work to figure which changes affect which issues, and which branches those changes are in. This often requires a lot of explicit manual work like making commit notes always include issue numbers for instance and "magic" commands like "Resolves #10" to auto-close issues.

If the issues are just another source artifact alongside the code in the same git history a lot of that information flows the other direction. Instead of a magic command in a commit note to close an issue, an issue closing shows up in a diff in the commit. Figuring out which issues are still open in a given branch (what isn't in our release branch yet?) is a simple command in a branch. Figuring out which issues changed between branches (what's finished in this feature branch that hasn't been merged to release yet?) is a typical diff operation.

In terms of documenting the state and progress of an individual branch in a complex project there can be a lot of magic in having issues tracked directly inside code branches. It's a fascinating ideal for code documentation to have issue comments, changes, and workflows side-by-side the code that changed with it.

That said, issue trackers are sadly not ideal code artifacts for a lot of reasons, including that it is easy to forget that issues aren't just for coders, but also stakeholders/product owners/PMs/etc.

I'm building one: https://github.com/MichaelMure/git-bug

A commited todo.txt [1] file with project.

[1]: todotxt.org

There were a whole bunch of these being actively developed around 2009-2012, but all of them slowly died out because each one had at least one of several issues that made them difficult or annoying to use, usually either requiring forgettable steps to actually distribute, or polluting the commits/branches with their own tracking.

My personal solution is similar to sibling comments, but more structured - using vimwiki in a subdirectory inside the repo, and committing updates alongside the relevant code changes.

The knowing meta-post.

And the meta-meta follow on.

Publicly visible outages on "pro" internet properties are actually a good thing because they help correct the views so commonly held by low-information executives about 5-nines and uptime. The more ammo people have, in aggregate, that "XXX never goes down" is a lie, the better off we will be as an industry.

I kind of wish HN wouldn't show these outages unless they're going on for multiple hours or at least hide them after they're back up. Usually by the time I see them and then check myself the site is already back up. Maybe I just need to read HN more often.

Yea, there are a lot of outage posts. It does kinda illustrate how many people depend on these core services though; and how little distributeiness or elasticity is built into the system.

Linux repos and ISO download sites have TONS of mirrors, sometimes contributed by companies, but also a lot of universities, government and public institutions.

Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible?

There is a deeper question here than just the outages; but rather why they affect so many people.

> Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible?

Just speculating but I think distributed systems are just inherently more complex. Serving multiple mirrors is easy, hosting across a set of providers and keeping things in sync gets harder.

It's a problem, and besides building completely open source systems that are designed to have distributed instances, I think its a pretty hard thing to fix.

I have a theory:

Centralized infrastructure tends to be less complex and therefore cheaper. Economies of scale drive people into using a centralized monolith and then network effects do the rest.

If you want a decentralized system to sustainably compete with a centralized one, the decentralized system must provide tangible and obvious surplus value in comparison to the centralized one or otherwise it is not economical.

In reality that's often not the case. Usability is often worse or infrastructure is in favor of centralization (e.g. asymmetric broadband internet connections).

> Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible?

> There is a deeper question here than just the outages; but rather why they affect so many people.

The answer to these seems so obvious to me perhaps I'm missing something - Outside the centralized services the solutions are fragmented (ie the decentralized solutions are used by a lot of people but are not large by definition).

I use my own Linux virtual server (as it happens, not in AWS although I have no anti-AWS beef) to host my private websites and my git repos. There are a million services that do this kind of hosting and people use all kinds of solutions like this. You will never hear about my private git repo being down because only I care when that happens.

I find the post-mortems helpful. I don't want to make the same mistakes with my infrastructure, and it gives me something to reference when, for instance, the upper management decides they want to migrate to one of these services.

True, but this is not a link to a post-mortem, it's just the status page.

Maybe publicizing these outages is the main reason the post-mortems are made available.

That seems at least partially true. However, I suspect that people would continue to think highly of tech companies that posted post-mortems, even if the issues were hitherto unknown. Ie, maybe it isn't only an act to prove that 'we are still trustworthy and transparent,' but also somewhat positive in and of itself?

The link has been voted for - so there is at least a minimal amount of interest.

I like those posts. Big providers are idolised by clients. Those posts help me little bit with fighting against unreasonable expectations and faulty comparisons.

I like those, even when it's been down for one or two hours only, but the title should be edited so that we know the outage is over. I'd still prefer this thread to be replaced by a postmortem but I don't think there's one yet.

> Maybe I just need to read HN more often.

Last night I wrote a little spreadsheet for myself showing how much time I spend doing various things. In particular, I for one do NOT need to read HN more often

Events like this make me realize how much I rely on my CI/CD pipeline for working on new features, deploying etc. I am often too lazy to run E2E tests on my machine if I know that on CI it will take only a few minutes to be done.

I feel guilty because I’ve never installed GitLab’s CI process locally and have only run it for the past few years through git commits. I keep expecting someone to complain, but so far, so good.

I expect similar poor behavior testing GitHub’s actions because it’s so convenient to only run through the service.

It is important though that it’s possible to run without CI/cd. If it wasn’t available I would complain because of the risk of lock-in.

A peer-to-peer (IPFS) stack for code collaboration.

https://radicle.xyz/

A Radicle project contains a git repository, plus the associated issues and proposals.

^ Neat project

Funny thing, I didn't even notice. In fact, being heavily reliant on Git and Github, I encountered maybe one or two issues a year when I couldn't push to Github because it was unavailable. In those cases, the progress can still be made locally, my workflow was barely affected. Of course, the deployment was impossible, but the downtime was never significant enough to worry or do anything about.

Git is designed as distributed VCS, so an outage of a centralized server should not matter too much. So far, the theory. Using Github excessively is the way back to centralization, nowadays handing control over to Microsoft. Everybody who is concerned (probably not too much) should have a look into alternatives.

Git itself is decentralised, but many (most? (all?)) of the workflows people build around it are not.

Individuals and teams who already have what they need in the local repos can continue to work through an outage of the VCS part of github, but at the points where they need to collaborate (merging each other's changes, issue tracking, etc.) the workflows break down. Yes you could share changes in a more distributed manner, or workaround the outage in other ways, but in reality people will stop and wait for the central repo to be available again. Also, pulling changes from my repo to yours directly to avoid that part of the downed service doesn't solve the issue tracker or CI manager also being down. The clue is in the name: git HUB.

That said, the issue is similar to how people see the results of aviation accidents: a jumbo going down takes out a lot of people but statistically, considering all air journeys, that is a lot less than the deaths that would happen due to equivalent car journeys. I suspect that everyone being a bit inconvenienced for a while when github/gitlab/other-centralised-service has issues like today doesn't add up to as much as the many small individual inconveniences they'd experience over time with their own local instances of, for example, gitlab, especially if including the routine maintenance time required which doesn't exist as much when using github, or gitlab managed instead of self-hosted - the outage just feels bigger than all the little problems because everyone is affected at once.

There are so many ways we become dependent of github, that nobody will shame on us because we are impacted by a github outage. In this context, is it useful to protect against outages ? It may be cheaper to do nothing.

Yea because the alternative (hosting git yourself on your own servers) will have better availability and few issues (UX just came to mind!)

https://about.gitlab.com/install/

So the UX point is moot. From there, it is just a matter of how much do you want to pay for availability (multizone replication, etc)

We're self hosting Gogs on a Hetzner VM and it's been a great experience. Of course it has less features. But it's simple and fast and I do about 10 minutes sys admin per quarter.

Resolved, all systems operational, that was quick.

If you don't the the "community" features of GitHub, we have had very good results with Amazon Code Commit. I've never experienced an outage, it's easy to manage, and very inexpensive.

https://aws.amazon.com/codecommit/

I see posts like this make the front page of hacker news every so often, but I don't recall seeing posts about things like making list outages for something like the LKML (Linux Kernel Mailing List) or an IRC network like Freenode.

Perhaps because Freenode has about 0.25% the number of users of Github - ~100,000 vs 40 million?

No, it's just the US tech bubble. When my whole country and some was suddenly denied all google services for several hours, including CDNs, DNS, all cloud, gmail, android app store, gdocs, all everything, it didn't make any news here either, despite being much more disruptive, affecting 10+ million people, and being a much more intersting reminder of how much techies make other people and themselves dependent on this megacorp.

All kinds of stuff broke, including "the entire internet" for people who were made to use 8.8.8.8 by their ISPs. E-shops that load jQuery from google stopped working, for no real reason whatsoever, people getting lost because they couldn't navigate anymore in the real world, etc.

Since then I try to flag all this repetitive nonsense about slack/github going down for a few minutes, to no avail. It keeps comming back to the first page of HN. :D

> When my whole country and some was suddenly denied all google services

> didn't make any news here either

Denied by government by any chance? Makes sense why it din't hit the news.

Denied by google engineers making a mess with router misconfiguration. The internet traffic at key points went down by ~30% just by google taking itself offline.

Also what was fun was everyone being blocked from all services that use reCAPTCHA.

God... reCAPTCHA needs to die. I refuse to use any websites that use it!

People care about what impacts or relates to them. That's human nature. Even your example was "my country" which is something that relates to you.

That being said, being dependent on Google is not unique to my country, and almost everyone should relate to that, at this point in history. And the scale of dependence only shows when it's truly completely shut down.

Did you post about that incident when it happened?

Yes.

> IRC network like Freenode

When was the last time ALL freenode servers were offline? I know about occasional netsplits but those do not affect all of their servers and freenode itself is still operational.

Justyna yesterday I was complaining about GitHub monoculture in the context of Atlassian sunsetting Mercurial. Outages like this are normal, but the problem is lack of good competitors.

There are quite a decent list of competitors, all of which have similar or sometimes better offerings. There are also plenty of fully free and open source solutions, should you want to do things yourself.

The Github monopoly is driven by its users.

I tried gitlab but I hated the web interface. It was lacking in so many departments. Any other alternatives to github to try?

What was lacking? The GUI on Gitlab is phenomenal imo.

On gitlab,repository and code searching function should be improved loll

Same could be said for github. I regularly literally copy and paste a variable name from a file on Github itself - paste it into the search, and it finds 0 results.

HOW COULD YOU FIND NOTHING?! Not even the file I copied it from? A simple grep search would've had no issue and would've been faster!

Yeah, I always end up having to clone the repo to grep.

Bitbucket is pretty good

Bitbucket suffers from all the usual problems of an Atlassian-product: A badly designed user-interface, and frequent stability/downtime problems.

Problem 1 with Atlassian is they're using Java.

Problem 2 with Atlassian is that their portfolio is entirely made up out of stuff they bought up over years, with wildly different coding and UI/UX standards/history.

This. The UI is quite primitive compared to GitHub IMHO. That and Gitlab is littered with half-implemented or poorly implemented workflows. The MR workflow to support code reviews was never really thought through.

Hi, GitLab employee, is there anything else (besides the MR workflow) specifically that you think we should improve? UI-wise or whatever else you've noticed

> The MR workflow to support code reviews was never really thought through.

Uh, I've been using this for quite some time in GitLab and never found it lacking. Curious what do you find missing there?

- sr.ht (OSS, managed, self-hosting)

- Gitlab (sorta OSS, managed, self-hosting)

- Gitea (OSS, self-hosting)

- Gogs (OSS, self-hosting)

- Bitbucket (managed)

- Azure Repos (managed)

- Google Cloud Repositories (managed)

To be honest, Gitlab has come a long way. And things like search that people point out doesn't work well on Github either, where code search is super broken and often skips results or find nothing where a git grep would drown you in results.

I think you're right about a general move towards a Git monoculture (as opposed to a GitHub one), but there are dozens of great Git repo hosting/collaboration companies - GitLab, Bitbucket & SourceForge to name a few of the bigger ones

The problem is bad design and/or integration of components. VMS clusters over leased lines went years without outages. Record was 17 years at a rail yard. They also did rolling upgrades across both different OS versions and CPU architectures. The methods are public if any competitors want to match or exceed them:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na...

These modern cloud services using Linux ecosystem are doing rolling outages instead. I'm amazed people keep making them dependencies. Maybe their local systems using similar technology went down more often. Maybe this is an improvement. I'd still put mine on an OpenBSD or OpenVMS cluster if it was business-critical with lots of money on the line. I want it staying up, up, up. :)

Modern apps are designed to deal with part of them being down - it is the core upgrade strategy.

VMS clusters were (and modern equivalents are) awesome, but they have a different speed of innovation / development.

It is a trade off - perfect uptime, but that new feature could take a year, or 99.9% uptime, but it can be written and deployed tomorrow.

Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this:

https://www.youtube.com/watch?v=4fFDFbi3toc

Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation.

In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security:

https://www.microsoft.com/en-us/research/project/ironclad/

https://www.microsoft.com/en-us/research/publication/p-safe-...

Plus some of their other tools in various states of usability:

https://rise4fun.com/

Our Erlang nodes are never down. We do live code updates and have gone several years now with no downtime.

I've been having trouble getting GitHub to send me a verification email (after adding a new email address) for over a week. Anybody else having this problem? I wonder if it's connected.

Also still ongoing: They don't have IPv6.

it's time to switch to SourceForge.

I've migrated all of our repos to codeplex.

What's a, "PR". (I'm being serious)

https://help.github.com/en/articles/about-pull-requests

Thank you, of course - I was wondering if PR was the same PR as in Agile.

Reply on news.ycombinator.com