Major Service Outage at GitHub | Hacker News Reader

Major Service Outage at GitHub

status.github.com

155 pointsacconrad85 comments

Github is down --> check out the unicorn --> refresh hacker news --> refresh again --> shitpost in comments

The coding equivalent of accident rubbernecking.

It also has an awesome automatic mirror mode that can remain sync'd to a remkte repository. I don't care what your company does or how you run it, but please don't just rely on Github to host your code without a mirror somewhere. There are drawbacks to this incessant centralization.

Instead of Gogs it is now recommended to use Gitea, since it has more activity (~400 more commits last time I checked) and is community driven: https://gitea.io/

According to their Blog: Gitea is a community fork of the popular self-hosted Git service Gogs. We’re a growing group of former Gogs users and contributors who found the single-maintainer management model of Gogs frustrating and thus decided to make an effort to build a more open and faster development model. https://blog.gitea.io/2016/12/welcome-to-gitea/

I would love this if gitea had a docker image for arm prebuilt. gogs/gogs-rpi is several versions behind gogs/gogs, but it at least exists.

I'm capable of building my own, but it's nice when there's already an official x86 version to have an arm version too.

I'll have to check out the source code for their github replacement when github is back up.

https://try.gitea.io/gitea/gitea

Just this week I setup gogs on my VPS to mirror all of my repos on github. It's part of my new year goal to avoid my dependence on 3rd party apps. Of course I still am depending on my VPS provider so...

The nice thing about git is that it's distributed. You can place as many mirrors as you like. If you make the project and repo names the exact same as the official ones, then switching between them can be as easy as replacing a hostname in your dependency manager.

Just set up a Banana Pi or whatever at home to mirror your mirror.

The specs for a beaglebone black are worse than a raspberry pi 3 in every way except the eMMC storage. Am I missing something? (I've actually been looking for something stronger to include in my raspberry pi cluster at home, but this doesnt seem like it)

Whoops you are right, I meant the Banana Pi m3(http://www.banana-pi.org/m3.html).

Banana pi is probably overkill for a low-key git repo. Its double the price of a raspberry pi.

That said, I'll actually be looking into this to possibly run arm stuff that needs more power.

does it mirror issues, PRs, wiki pages etc?

I don't think so (we use other apps for those features) but Github's wiki is just a git repo, isn't it?

How am I supposed to do any work if I can't copy and paste code.

StackOverflow is still up.

My builds are broken because they have dependencies on github repos.

This is a teaching moment for you then. Minimize external dependencies when you've moved from "thank god it works" to "this is production ready."

One solution to that problem is to run your own local git server, with an automatic process for pulling github repositories of the dependencies you need. Then you can always know that you will have the access to our dependencies even with a third party is unavailable.

I wonder how many companies are unable to deploy new updates now because they rely on Github?

Friends don't let friends deploy on Fridays... it's always a horrible idea.

Especially in France where it's 5:27.

excellent point sir.

Meanwhile, today is Friday the 13th.

Deploying on Friday is fine if you can rollback quickly :)

I think the point was that you don't want to be messing with production when people are going to be gone for days. Even if it's easy to roll back I'd just as soon not deal with it. If your application is of such importance that rolling a change out Friday v. Monday makes a different in the business's bottom line, that business can almost certainly afford to have weekend staff that could cover it.

Are you trolling?

_Friday the 13th_, no less.

Not if you've got your CI configured properly.

mine.

Try to deploy -> bad key -> check github website -> go to hackernews

Mine :)

mine...

mine as well

And mine

Mine. Let this tree grow!

Currently 98.9212% app server availability in the last month.

That means 7h 52m 49.7s of potential downtime/unavailability [1] in the last month.

And the trend of that percentage doesn't look that rosy.

[1] https://uptime.is/98.9212

I hope everybody on HN is doing okay with this recent outage, I'm here if anybody wants to talk.

Seriously, Github Outage Syndrome (GOS) can be horrible. If you don't talk about your outage feelings, they will eat you from the inside out.

Obviously anything you tell me will be kept in the strictest confidence, all without judgement.

Stay strong HN, Github will be back soon!

Edited to Add: Why they hate guys? Didn't mean to offend, thought the post said it all and could probably do with a little levity. My bad. Maybe.

I've configured one remote with two pushurls and set this remote to all my branches. This way I always push to BitBucket and GitHub at the same time. Just wondering, how this would work if I was not the single user contributing.

You'd have to either coordinate with the other users to have them do the same, or officially declare which one is "official" and which one is "backup" and set up something so the backup tracks the official (and then you only need to push to the official as well)

…and this is how we remember how much we rely on GitHub, and how much value it provides. Every day.

It sort of becomes your Internet and computer. No computer no work. No Internet no VPN no work.

But in this case I guess investing in local mirror is a great investment.

A little bit OT, but having colored favicons based on the status is a nice touch!

https://status.github.com/images/status-icon-green.png

https://status.github.com/images/status-icon-orange.png

https://status.github.com/images/status-icon-red.png

Long live Gitlab...

Except. Pushes of new repos are still broken in the CE. No fix for over 3 weeks.

https://gitlab.com/gitlab-com/support-forum/issues/207

https://gitlab.com/gitlab-org/gitlab-ce/issues/26369

https://gitlab.com/gitlab-org/gitlab-ce/issues/21502

Only work around is to create a README on empty repos via the GUI and which then opens up the Protected Branches page. That said the rules around who is able to commit on the repos are broken.

Hey there, thanks for bringing this up. First, I want to apologize for the inconvenience you've experienced as a result of this issue. Second, I'd like to apologize for the delay in getting back to you on this as well.

You're right, 3+ weeks is a pretty long time for something like this to be affecting people. Unfortunately I don't have an ETA on when this issue will be resolved, but I'll be keeping tabs on it going forward. Please feel free to add additional comments to the issues if you haven't already, or you can reach out to me directly. Again, I'd like to apologize for any trouble this caused you and others affected by this issue.

This issue should be fixed as of today in 8.16.3

Gitlab can go down. Gitlab might have outages as well unless you are using the self-hosted version (although, in this case there is a self-hosted version of Github as well).

Just because Gitlab is not as famous as Github and hackers don't run their botnets on it, it doesn't mean it is safe or stable.

I'm envisioning some kind of caching forward proxy for git. Point your clones/fetches at it and it will proxy the connection to the source fork. If the source fork is down, it will serve its cached copy of the fork.

I found this[0] but it's prerelease and looks like it hasn't been touched in five years. Anyone know of any other solutions like this? Good idea? Bad idea?

0. https://github.com/rohanpm/ngitcached

GitHub adds value. They have a few engineering issues to solve given the negative attention such a popular code versioning system attracts. Although there are some outages due to this condition, the value added to our organization far outweighs these minor inconveniences.

Been working on a down detector type app, its like half baked at the mo, hopefully I can build up a data set that allows people to fairly compare the reliability of different SASS products such as github and bitbucket

https://young-forest-39406.herokuapp.com/sites

Has anyone heard what might be the cause?

> 09:14 MST Major service outage.

It's been 4 minutes, so I highly doubt anybody at GitHub even knows yet. Maybe one guy who's thinking a lot of "oh crap, oh crap, oh crap" right now.

The last outage they were back up in 4 minutes.

Pretty sure devops at a service like Github have slack alerts and/or something like pagerduty. They'll be drowning in alerts.

Knows the cause, that is. :-)

Rumors are that they edited some JSON files. Not a good day to touch JSON.

It just went down about 10 minutes ago, you'll have to wait a bit for an explanation...

So there was an outage yesterday which required failover to another data center. Wonder if both are related. Either way I want to read their postmortem. GitHub's postmortem is usually very educational.

I use JSPM on my projects and I have several dependencies on github repos that aren't in NPM. Anyone have any ideas on a fallback solution for when github goes out like this?

Have a local git server, with an automatic process for pulling in the repositories that are your dependencies. Then use the repos from that local git server instead of github directly.

Is there a way to be able to point JSPM to a different, non-github repo? A remote clone of the repo might work then.

Hacker News Time! :)

It's always Hacker News Time!

I count 4 threads on /new referencing GitHub down :D

but we are keeping this one :) I like it

Problems at quite a few places actually - http://downdetector.com/

Friday the 13th...

The day following a full moon...

The day github had a major outage...

GitHub's Twitter status post at the start of the outage:

https://twitter.com/githubstatus/status/819940738452615168

Currently 442 retweets, 266 likes.

⋯

OK, they say it's back up:

https://twitter.com/githubstatus/status/819947685503442945

Still getting 503 First Byte Timeouts when trying to view content on raw.githubusercontent.com so definitely not all back yet

OAuth API is still down.

Someone was faster: https://news.ycombinator.com/item?id=13391883

Again.

Affects both the website and git.

Affects push to GitHub*

I really hope that your copy of git isn't broken because GitHub went down :p

Haha got me. :P

No I just tried both the website and a `git pull`.

It's back :).

Seems back up now.

"Github is down" is the new compiling.

http://xkcd.com/303/

Site came back online, but ssh/git still down.

looks like it is back up now

It's back. [END OF THREAD ?]

Since I can't push to my repo, I'm gonna say no, it's not back.

Github has been down (or very slow) a lot the last few months. They used to have a good engineering reputation (at least with me), but it's been tough times over there.

Reply on news.ycombinator.com