GitHub Is Down | Hacker News Reader

GitHub Is Down

githubstatus.com

213 pointsjuancampa90 comments

Open articleView on HN

PSA: I just received a github phishing request that was not detected by Gmail. I almost fell for it.

Phishing email looks like a "review your suspicious activity" alert, but the alert is the suspicious activity.

https://imgur.com/a/zdtWmuN

Just received a repository vulnerability notification as well, in the Pillow dependency but it seems legit.

Did the odd capitalization, "April 01th", and unrelated sending address not tip you off?

Yeah I gotta agree with you. How is this any different from the other 10s of thousands of variations of phishing attempts?

If it's not signed and it's not from github.com or a registered subdomain, or if the URL of the action isn't _explicitly_ github.com... it's not legit. It doesn't matter if it slipped through GMail's filter or not...

I don't mean to shame people for falling for phishing. I fell for a convincing Steam phish once when I was a young teenager after a friend's account was compromised and messaged me with a link to a perfect clone of the login page and a similar domain.

But as far as phishes go, this one just seems not too convincing.

In the case of the github phishing it does have a valid certificate and the host name is something that looks legit enough it will probably fool a lot of people

Saw many similar request on twitter as well. Hopefully the two things ( Being Down and Phishing ) are coincidence.

Apparently the earlier thread about Microsoft's cloud stability was a bit prescient.

Has GitHub moved their infrastructure to Azure? Last I recall they ran mostly on their own servers.

I'd imagine the higher volume due to remote work (e.g. people collaborating more on issues and such who may have communicated in person previously) is also a contributing factor.

From the sound of their earlier 'not postmortem but we'll do one soon', it sounded like they currently have a database tier that's creaking at its foundations, and I'd guess any time there's extra strain or someone pushes a fix to the GitHub codebase that adds a tiny bit of unoptimized code, it strains it past the limit.

I'm not sure. My post was mostly in jest. But being at Microsoft clearly hasn't helped.

Even our static assets from raw.githubusercontent.com are throwing 500s. Whatever it is, it sounds like a pretty widespread failure...

Yeah, can confirm. I'm working on some visualizations for covid-19 where I pull data from raw.githubusercontent.com and I'm getting 500s.

There are some CDNs in front of rawgit like jsDelivr that may work.

If only there was some kind of distributed version control system. /s

Of course, there is. You just need to host it. Self-hosted GitLab mirrors work great when GitHub is down.

https://docs.gitlab.com/ee/user/project/repository/repositor...

I think he meant git itself

Or Gitea, if you want to use a fraction of the resources! I use it solely as a GitHub backup — you can mirror Git repositories from anywhere and it pulls changes on a schedule. Literally set it up this week, thank goodness.

https://gitea.io/en-us/

Instead of relying on GitLab's API, it's probably a better idea to just use git remotes properly. I put together a script [0] to automate the process to set the primary remote to Sourcehut (git.sr.ht) and mirror to GitLab and GitHub. And yes, that script is in a repo that's also mirrored to GitHub and GitLab (check the README). It combines well with my `git pushall` alias [1].

[0]: https://git.sr.ht/~seirdy/dotfiles/tree/master/Executables/s...

[1]: https://git.sr.ht/~seirdy/dotfiles/tree/master/.config/git/c...

You can just sync with someone else’s repo by specifying it on the command line. Nothing to install, no script, nothing. It’s built into git and is right there in the single line summary of the man pages for git-fetch, git-pull etc: they all take a repository argument.

Managing keys for all the contributors quickly gets to be a chore.

Identity management is what Github really is all about.

I don't see the point for major open-source projects or even company projects to depend on a third-party provider for git services in the cloud that is not on-premise. It's fine if the source code is self-hosted or on premise and the mirror is hosted on some cloud VCS like Github, but not the other-way round, which for the latter you risk getting problems like this.

The case for self-hosting a VCS server for serious projects is made once again.

I can understand this sentiment, but a lot of projects and companies (especially startups) want something that "just works" and not something that they'll need to invest resources into maintaining. Something like this can happen on your self-hosted git server and you won't have an SLA or anything to fall back on.

Github (and gitlab) has a large value add with regard to discoverability, issue tracking, user management, hosting simplicity, and usually solid uptime.

GitHub addiction is not just about git itself. It's about the entire ecosystem of pull requests, issues, forking, linking to related projects, CI integrations, hosting for docs, and so on, all integrated under one sign-on umbrella. It is awfully convenient. Yes, for sure you can self-host one of the alternatives like GitLab, or assemble an equivalent set of services from smaller parts. It is absolutely reasonable to suggest that people should - I'm even inclined to agree - but comparing against git all by itself is the wrong comparison.

It is indeed convenient but the whole ecosystem now runs the risk of being a single source of failure if even one part of the service goes down and it is out of your control to do anything about it and also defeats the purpose of decentralisation of using git. It may not matter for personal or toy projects, but for company or serious ones, it does.

In the case of other risks of only using GH to host your project, Its like hosting your encrypted private key on Keybase because it is temptingly convenient but both run the risk of going down or getting compromised in a security breach.

Serious projects like the Linux kernel, WebKit and Chromium aren't primarily hosted on Github but have Github mirrors instead which makes more sense.

Having GitHub as the primary repository is OK as long as you do not depend on it for working, testing and deploying.

The issue is that people start using the CI/CD, the PRs, the issues and... yeah.

Unless you have the (ops) capacity to host it yourself reliably, chances are GitHub will be up more than your self-hosted solution.

As I'm typing this, I'm waiting for GitHub to recover so I can deploy an update to a rather important project for a big client. However, I'm happy it's their engineers to be scrambling to fix their git hosting, instead of me context switching from my work and rushing to duct-tape piece of plumbing I really don't care to think about.

and they rolled out a fix in 45 min which is less time then i took for lunch today.

> Unless you have the (ops) capacity to host it yourself reliably, chances are GitHub will be up more than your self-hosted solution.

For an individual it's absolutely trivial to self-host git over ssh more reliably than github.

It just costs a bit of money.

money is probably the main reason people don't selfhost

> self-host git over ssh more reliably than github.

Host where? That host may be down. If it's local, then you run into other problems trying to access it remotely.

Still, I find github to be down quite a lot lately which is concerning. It doesn't stop me from working, I just can't review PRs and get changes.. but I would be more worried if I were using github actions stopping me from releasing a hotfix.

We use a self-hosted GitLab ourselves and I can honestly say we have more up time than GitHub.

But 9/10 every project we use is on GitHub and our work is stalled. Centralization will be our downfall :(

You may be able to mirror those to your GitLab and then do all your work from the mirror.

https://docs.gitlab.com/ee/user/project/repository/repositor...

Congrats on the self-hosted uptime :)

The nature of our product is to follow 100s of projects. And most of the interesting stuff is actually not in the code but in the issue tracker, release notes etc (for us).

This is why it's a good idea to have multiple remotes and advertise those remotes in the project README (my examples: [0]). You'll lose centralized issues during downtime, but you'll at least be able to interact with some remotes and do basic collaboration with `git format-patch` and `git send-email`.

If something like sourcehut [1] goes down, nothing will change; users will still be able to push/pull to mirrors and work with mailing lists and git's built-in email-based pull-requests.

[0]: chattiest-channels, dotfiles, term-dmenu, and mpd-scripts at https://git.sr.ht/~seirdy

[1]: https://sourcehut.org/

UTC for status updates?? Not PST/PDT??

Amazing!

Yep. git push/pull are throwing permission denied errors. I thought I'd check ssh itself.

ssh -T git@github.com throws the same permission denied error.

That's great. 2 minutes before it went down a pipeline workflow ran in Heroku deploying a bug to my app. Now I can't deploy the hotfix since it depends on Github and I'm not able to revert to the last successful build on the Heroku interface.

Any ideas?

How about git push heroku master from local branch?

Ho! I just forgot about it. Thanks a lot. It's running ¯\_(ツ)_/¯

Remove spiderweb dependencies from your deployment pipeline?

This is a good point, but it could be worded better.

One of the things a site reliability engineer should think about is how well the site can be operated when dependencies have issue. After an incident like this, even if you were able to recover, it's worth thinking about how things could have gone better.

In the past I had a painful experience with one application I was supporting that needed to install NPM packages on deployment. We couldn't successfully deploy (or scale up) for the duration of that outage. In that case we realized it was safer to switch to server images with all assets pre-installed and an NPM cache to give the build a better chance of succeeding. The next NPM outage we only noticed after the fact :)

Not certain how this particular deployment pipeline is failing due to the GitHub outage, but a post-mortem to discuss may be helpful and protect against future issues.

To your point on NPM dependencies, CDNs for JS libraries is another thing that I recently learned is common practice, which I don't fully understand. It seems like if the CDN goes down then your application stops working, but I am convinced I'm missing something here because it seems like such poor engineering judgement. This seems to be really common in SPAs, which is precisely where (it seems to me) you shouldn't be using a CDN.

It seems the convenience of cloud based deployment pipelines is not really worth situations like this.

People do it because it is faster (limit of requests per domain) and cheaper.

Specifically for the javascript frameworks, the use of a popular CDN increases the chance that the browser will have the asset in the cache already. Browser cache is a huge win for load times.

Static file hosting via a managed CDN is a fairly reliable option, better than many companies can build on their own.

Deploy to another Heroku environment and use DNS to reroute the traffic temporarily?

Doesn't Heroku have a rollback feature in their dashboard?

I didn't realize until today that https://www.npmjs.com/package/geckodriver downloads from Github.

Yep, hitting Nginx 504s when accessing my (and others') repositories.

I was building a github app and was scratching my head about where I messed up. The website and front end were loading but github isn't sending webhooks.

I was deploying code to production as it died!

Well that is just the worst luck. Condolences as you revert locally?

Wow, it's totally dead, not even a read-only view.

Well, I guess I'm done working for the day.

Today I have a deadline. It heats up the emotions.

Paulo Coelho: “And, when you want something, all the universe conspires in helping you to achieve it.”

Murphy: [GitHub is down]

https://twitter.com/QuantumGameIO/status/1245818849246748674

have never seen the github 500 screen before! the 404 screen is delightful, this one is as well

silver linings

I'm not even getting the pretty errors :) Just the generic nginx 504 timeout.

I was trying to authorize a third-party client and got an angry pink unicorn.

Same here. Never seen that before.

You can reliably hit the 504 screen (no real styling) by trying to pull up the blamelog for a large file that has a complex history

Lol. Couldn't work on github, I thought it was just a slow connection so checked HN instead to see whether the page loads - and now I guess it wasn't just an issue on my end :D

Okay... so, when is Amazon launching their GitHub alternative?

Oh, they have CodeCommit... good luck with that one though!

There's the AWS CodeCommit service but it's for private repositories only

https://aws.amazon.com/codecommit/

I wrote some open source code to automatically mirror all my commits to Keybase. </shameless plug>

I think read replica mirrors are good practice for fault-tolerant systems

link?

it's probably on github

Hopefully does not depend on github webhooks for replication!

It does, but it would've mirrored your latest commit before it went down.

Anyway, here's how you connect Keybase to GitHub:

https://blog.codefor.cash/2019/08/30/free-automatic-github-b...

https://github.com/codeforcash/github-to-keybase-mirror

Note: this was a v1, and I'm sure there are more resilient ways to do this. Critical feedback strongly encouraged (ideally along with suggestions).

Hooks (https://git-scm.com/docs/githooks) together with templates (https://coderwall.com/p/jp7d5q/create-a-global-git-commit-ho...) sound like a good alternative.

Edit: I can see where your solution has an advantage. When one accepts contributions, all of that can be automated with web hooks via github. Still, having to run lambda/similar for this is a bit heavy weight.

Edit 2: Considering that for a release, I would opt in for release from a tag or explicit commit, that would be a non issue. Pull when needed, for redundancy, use another, self hosted bare repo. Push there in case of github down.

Wouldn't replacing the github hook with a git-hooks script incrementally improve the solution as it currently exists?

Then, combine that with a git-hook script that designate all non-release branches push to team's Keybase repo as well? Maybe including a custom prefix in the branch name to avoid need for conflict resolution?

I'm also thinking that it would be easier to use git-hooks to push to AWS CodeCommit (in a random region that GitHub is unlikely to use) paired with CloudWatch events to invoke service backup to Keybase

Of course, ideally it would be a Terraform (or similar open source stack) script so it's not dependent on AWS as a vendor... I have experienced the real pains of vendor lock-in, even with cross-region there is opportunity to improve robustness

Didn't check the status page but I figured once I had my before bedtime peak at the trending repos.

Their status page is quite pretty, does anyone know of an open-source option that looks similar?

The BSOD for a new generation...

Just managed to push something, but the web interface is still down.

EDIT: (21:00 UTC ) Just got in :D

It's OK to rely on Github clones for dependencies, they said.

Mind your own codes, they said.

I first thought I was having network saturation problems due to my gf’s kids playing games instead of doing their school online. I then checked some other sites and found they were fine but my first assumption was not that GitHub itself was having problems.

GitHub or how to re-centralize what was properly decentralized. Sad.

Seems to be back up now.

Still down for me

...its back now for me.

Yeah navigation is slow or timing out on my repos.

I too went straight to HN upon realizing this

Working now

It's working again.

ahh, angry pink unicorns.

Reminds me of the old Twitter failwhale

Hehe, somebody modified the XKCD "Compiling" comic.

https://twitter.com/PonteIneptique/status/783190337602846720

Reply on news.ycombinator.com