Hacker News Reader

Comment by IgorPartola | Hacker News Reader

I honestly feel bad for the engineers at GitHub for having to deal with stuff like this. GitHub is large, so they are a target, and the specifics of what they do means that caching is not a straightforward task. I imagine there are a lot more vectors of attack that have not been used yet and guarding against them is always going to be on a case-by-case basis. In the meantime, when GitHub is having downtime or even badtime it impacts its users pretty significantly. The private repo's I work on are a source of income for GitHub, but if this gets common enough the people in charge might just move away from it to a smaller competitor that doesn't have these problems just so that my time is not wasted on waiting on GitHub to come back up.

View on HN

Isn't the whole point of git that you don't need to even have internet access to get work done?

You still have your local source code, but GitHub gives you issue tracking, comments, pull requests, and other things you could still use for work - none of which can be cloned locally.

Okay yes, but it probably isn't going to be down for more than a few hours at a time. I'm sure pull requests and issues can wait a few hours vs. actually fixing bugs or writing new code. Right?

No, they can't wait a few hours. Not when you have a 60 developer team on a private GitHub enterprise account, and it's the middle of the workday.

If you have a team that large and are relying on cloud services, then you need to have a plan in place for short interruptions like this. If you really have a 60-dev team, you should have knocked out all the SPOFs in the systems that support them - 60-dev teams are millions of dollars a year in salary, even at bargain basement wages. There is not a single serious provider that will guarantee you 100% uptime, because outages, however rare, do happen.

I don't disagree with you, but it's also not that simple. Anyway - I'm simply trying to point out that it is a bit deal when a large cloud service like GitHub goes down.

I'm not sure what you mean by "a private GitHub enterprise account", but GitHub has a product called "GitHub Enterprise", which is essentially GitHub in a VM that you can install in your own data center. We have a lot fewer than 60 developers, and we have a GitHub Enterprise installation, partly out of concern about issues like this (and also maybe a little IP protection paranoia).

if all 60 are working together sure, but if it's a handful of devs working with each other, you can pull directly from each others repository.

PITB, yes. But you can still get work done.

Again, not so easy when it's a massive multi-team effort - we have a continuous integration system that builds every branch (we use a GitFlow based model) on every commit that gets pushed to GitHub.

You can pull from eachother, but since you know github will be up in a few hours and it would take more than that to really coordinate any workflow change, most of the time people just goof off until github is back.

Plus you are forgetting that lot of automated jobs get triggered on github changes. Many shops kick off all kinds of tests, deployments, and other things based on changes to the github repo.

If devs want to use that as an excuse to do other things, that's cool.

My point was simply that you can still be productive when github goes down, if you want to be.

Sure, developers can still be productive. But what about QA, UX, Product Managers? They rely on automated jobs that get triggered by GitHub changes.

I don't think you're "wrong" for the developer use case, but the reality is that it can bring large teams to a screeching halt.

I find it easy to imagine that if the task you are doing right now requires looking at open issues, it could be a blocker. I depend on an internally controlled issue tracking system, and if my current task is creating or responding to defects and requirements, I'm not going to be happy if that system is down.

One other problem is that if you're using a tool like Capistrano to deploy, you usually set your remote repo to Github. When Github goes down, you can't deploy until it comes back up or you set up your own remote server.

You'd have to hope so! I wanted to use the site, but having it down didn't bother me. But if I depended on it for my day job, I'd want more than just my source code available.

For me, issues are the most important part. I might set up a script to run daily that would import my issue list into a local csv.

Sort of. When was the last time you clones a repo remotely from another user's laptop who is sitting hundreds of miles away? Do you even have an account on their laptop to be able to SSH to it? I am no Linus. I don't email patches. I want to grab my coworker's latest pushed changes and if GitHub is down, I cannot do that. That sucks.

Also, our deploy process is to do a `git pull` on the remote server, then run the build process, and finally to deploy the built stuff. When GitHub is down, we don't have a process for this.

I agree, both of these things could be avoided by having a different procedure in place, but that would obviate the need for GitHub altogether. Why use it when we can already share code and push it to our servers to be built? The point of GitHub is to provide a nice upstream that you can push to and pull from a la SVN because for most projects that model works really well. git provides all the niceties of local branching, rebasing, etc. while GitHub makes it easy to collaborate.

Having said that I'd love to hear how others handle this type of challenge.

It depends on what you mean by "work". In a lot of organizations, github downtime is a blocker for doing a release. Writing and committing code is not the only kind of work.

No, that's one of several features of git. It's not "the point"

I thought the error message was

"Commit locally github is down"

Reply on news.ycombinator.com